Modes of Convergence, LLN and CLT

Four notions of convergence

DefinitionThe four modes

Let $X_1, X_2, \dots$ and $X$ be random variables on a common probability space.

• $X_n \to X$ almost surely ($X_n \xrightarrow{a.s.} X$) if $\P\big(\lim_n X_n = X\big) = 1$.
• $X_n \to X$ in probability ($X_n \xrightarrow{\P} X$) if $\P(|X_n - X| > \eps) \to 0$ for every $\eps>0$.
• $X_n \to X$ in $L^p$ ($X_n \xrightarrow{L^p} X$) if $\E|X_n - X|^p \to 0$.
• $X_n \to X$ in distribution ($X_n \rightsquigarrow X$) if $F_{X_n}(t) \to F_X(t)$ at every $t$ where $F_X$ is continuous.

Only the last is a statement about laws alone; the other three compare the random variables themselves and require them to live on the same space.

TheoremImplications between modes

$$X_n \xrightarrow{a.s.} X \;\Longrightarrow\; X_n \xrightarrow{\P} X, \qquad X_n \xrightarrow{L^p} X \;\Longrightarrow\; X_n \xrightarrow{\P} X, \qquad X_n \xrightarrow{\P} X \;\Longrightarrow\; X_n \rightsquigarrow X .$$

None of the reverse implications holds in general, and almost-sure and $L^p$ convergence do not imply each other.

ProofReveal

The law of large numbers

TheoremWeak law of large numbers

Let $X_1, X_2, \dots$ be uncorrelated with common mean $\mu$ and variance at most $\sigma^2 < \infty$, and let $\bar X_n = \frac1n\sum_{i\le n}X_i$. Then $\bar X_n \xrightarrow{\P} \mu$; quantitatively,

$$\P\big(|\bar X_n - \mu| \ge t\big) \le \frac{\sigma^2}{nt^2}\quad\text{for every } t>0 .$$

ProofReveal

TheoremStrong law of large numbers

If $X_1, X_2, \dots$ are i.i.d. with $\E|X_1| < \infty$ and $\E X_1 = \mu$, then $\bar X_n \xrightarrow{a.s.} \mu$. Conversely, if $\E|X_1| = \infty$ then $\limsup_n |\bar X_n| = \infty$ almost surely.

Kolmogorov's strong law is quoted rather than proved: the standard proofs (via truncation and Kolmogorov's maximal inequality, or via the ergodic theorem) are longer than this page warrants, and the references give them properly. What is worth stressing is the hypothesis gap: the strong law needs only a first moment, while the elementary weak law above needed a second.

The central limit theorem

TheoremCentral limit theorem (Lindeberg–Lévy)

Let $X_1, X_2,\dots$ be i.i.d. with mean $\mu$ and variance $\sigma^2 \in (0,\infty)$. Then

$$\frac{\sqrt{n}\,(\bar X_n - \mu)}{\sigma} \rightsquigarrow \Ncal(0,1) .$$

The characteristic-function proof is standard and is not reproduced here. The structure worth remembering is: $\varphi_{\sqrt n(\bar X_n - \mu)/\sigma}(t) = \big(\varphi_{(X_1-\mu)/\sigma}(t/\sqrt n)\big)^n$, expand the inner characteristic function to second order, and recognise $(1 - t^2/2n + o(1/n))^n \to e^{-t^2/2}$.

TheoremBerry–Esseen bound

Under the hypotheses above with additionally $\rho = \E|X_1 - \mu|^3 < \infty$, there is an absolute constant $C$ (one may take $C \le 0.5$) with

$$\sup_{t\in\R}\left|\P\left(\frac{\sqrt n(\bar X_n - \mu)}{\sigma} \le t\right) - \Phi(t)\right| \le \frac{C\rho}{\sigma^3\sqrt n} .$$

This is the quantitative form of the CLT, quoted from the literature. Read the bound carefully: it controls the uniform error in the CDF by something of order $n^{-1/2}$. That is exactly the problem.

Empirical observationThe CLT does not license Gaussian tail estimates

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

It is tempting to argue: "$\bar X_n$ is approximately $\Ncal(\mu, \sigma^2/n)$ by the CLT, therefore $\P(|\bar X_n - \mu| > t) \approx 2\big(1 - \Phi(t\sqrt n/\sigma)\big) \le e^{-nt^2/2\sigma^2}$." This step is not valid as stated, and the reason is quantitative rather than philosophical.

Run it: Concentration inequalities, side by sideEmpirical tail probabilities of a sample mean against the Markov, Chebyshev, Hoeffding and Bernstein bounds.

Problems

Separating the four modes of convergenceProof · Intermediate · 30 min The weak law needs only uncorrelatednessProof · Beginner · 15 min Why the CLT cannot be used for tail boundsNumerical · Advanced · 30 min