Expectation, Variance and Conditional Expectation

Expectation as an integral, the variance identities used in every bias–variance computation, and conditional expectation as an orthogonal projection.

ProbabilityCore⏱ 3 hours10 min readexpectationvariancecovarianceconditional expectationtower rule

4 problems →

AssumesProbability Spaces and Random Variables

Expectation

DefinitionExpectation

For a random variable $X \ge 0$ , $\E X = \int_\Omega X \, d\P \in [0,\infty]$ , defined as the supremum of the integrals of simple functions below $X$ . For general $X$ , write $X = X^+ - X^-$ and set $\E X = \E X^+ - \E X^-$ whenever at least one term is finite; $X$ is integrable when $\E|X| < \infty$ .

Expectation is linear, monotone, and satisfies $|\E X| \le \E|X|$ . The following identity is the one that turns tail bounds into moment bounds and back, and it is used constantly on the sub-Gaussian page.

PropositionTail formula for the expectation

For $X \ge 0$ ,

\E X = \int_0^\infty \P(X > t)\,dt, \qquad\text{and more generally}\qquad \E[X^p] = p\int_0^\infty t^{p-1}\,\P(X>t)\,dt \quad (p > 0).

Variance, covariance and the decompositions that matter

DefinitionVariance and covariance

For square-integrable $X, Y$ ,

\Var(X) = \E\big[(X - \E X)^2\big] = \E[X^2] - (\E X)^2, \qquad \Cov(X,Y) = \E\big[(X-\E X)(Y - \E Y)\big] .

For a random vector $X \in \R^d$ the covariance matrix is $\Sigma = \E[(X - \E X)(X-\E X)^\top] \in \mathbb{S}^d$ , and $\Sigma \succeq 0$ because $v^\top \Sigma v = \Var(\ip{v}{X}) \ge 0$ .

PropositionBias–variance and variance decompositions

Let $\hat\theta$ be a square-integrable estimator of a constant $\theta$ , and let $\mathcal{G}$ be a sub- $\sigma$ -algebra. Then

\underbrace{\E\big[(\hat\theta - \theta)^2\big]}_{\text{mean squared error}} = \underbrace{\big(\E\hat\theta - \theta\big)^2}_{\text{bias}^2} + \underbrace{\Var(\hat\theta)}_{\text{variance}},

and

\Var(X) = \E\big[\Var(X \mid \mathcal{G})\big] + \Var\big(\E[X\mid\mathcal{G}]\big)

(the law of total variance).

Key idea

The bias–variance identity is exact algebra, not an approximation or a heuristic trade-off. What is heuristic is the claim that reducing one must increase the other; that claim is a description of certain model families, and the double descent page shows regimes where it plainly fails. Keep the identity and the folklore separate.

The vector version of the variance-of-a-sum computation is used in every mini-batch argument:

\Var\left(\frac1n\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i,j}\Cov(X_i,X_j) \;\overset{\text{i.i.d.}}{=}\; \frac{\sigma^2}{n} .

The middle expression is the honest one. The $\sigma^2/n$ rate is a consequence of independence, and correlated samples — adjacent tokens in a text corpus, say — do not enjoy it.

Conditional expectation

DefinitionConditional expectation

Let $X$ be integrable and $\mathcal{G} \subseteq \mathcal{A}$ a sub- $\sigma$ -algebra. A random variable $Z$ is a conditional expectation of $X$ given $\mathcal{G}$ , written $Z = \E[X\mid\mathcal{G}]$ , if

$Z$ is $\mathcal{G}$ -measurable, and
$\E[Z\one_G] = \E[X\one_G]$ for every $G \in \mathcal{G}$ .

Such a $Z$ exists and is unique up to almost-sure equality.

TheoremConditional expectation as an L2 projection

If $\E[X^2] < \infty$ , then $\E[X\mid\mathcal{G}]$ is the orthogonal projection of $X$ onto the closed subspace $L^2(\mathcal{G}) \subseteq L^2(\mathcal{A})$ of square-integrable $\mathcal{G}$ -measurable variables. Consequently

\E[X \mid \mathcal{G}] = \argmin_{Z \in L^2(\mathcal{G})} \E\big[(X - Z)^2\big] .

PropositionTower property and pulling out what is known

Let $X, Y$ be integrable and $\mathcal{H} \subseteq \mathcal{G}$ sub- $\sigma$ -algebras.

Tower / smoothing. $\E\big[\E[X\mid\mathcal{G}]\ \big|\ \mathcal{H}\big] = \E[X\mid\mathcal{H}]$ ; in particular $\E\big[\E[X\mid\mathcal{G}]\big] = \E X$ .
Taking out what is known. If $Y$ is $\mathcal{G}$ -measurable and $XY$ is integrable, then $\E[XY\mid\mathcal{G}] = Y\,\E[X\mid\mathcal{G}]$ .
Independence. If $X$ is independent of $\mathcal{G}$ then $\E[X\mid\mathcal{G}] = \E X$ .

Problems

Moments from tail bounds via the layer-cake formulaProof · Intermediate · 30 min Law of total variance and its useProof · Intermediate · 30 min Conditional expectation minimises mean squared errorProof · Intermediate · 30 min The 1/n variance rate requires uncorrelated samplesNumerical · Intermediate · 30 min

References

BookRick Durrett (2019). Probability: Theory and Examples
BookDavid Williams (1991). Probability with MartingalesChapter 9 gives the conditional-expectation construction in the form used here.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

PropositionTail formula for the expectationproof includedThe mean of a non-negative random variable is the integral of its tail probability.
PropositionBias–variance and variance decompositionsproof includedMean squared error splits into squared bias plus variance; total variance splits into explained and residual parts.
TheoremConditional expectation as an L2 projectionproof includedConditional expectation is the orthogonal projection of a square-integrable variable onto the sub-sigma-algebra's functions.
PropositionTower property and pulling out what is knownproof includedThe two computational rules that make every conditional-expectation manipulation work.

Problems (4)

All problems