Sample Covariance and the Marchenko–Pastur Law

What the spectrum of an empirical covariance matrix looks like when the dimension is comparable to the sample size, and why that breaks classical statistics.

Random MatricesGraduate⏱ 3 hours6 min readmarchenko-pastursample covariancespectrumaspect ratioconditioning

5 problems →

AssumesSpectral Norm of Random Matrices

The law

TheoremMarchenko–Pastur law

Let $X \in \R^{n\times p}$ have i.i.d. entries with mean $0$ and variance $\sigma^2$ , and set $\hat\Sigma = \frac1n X^\top X$ . Suppose $n,p\to\infty$ with $p/n \to \lambda \in (0,\infty)$ . Then the empirical spectral distribution of $\hat\Sigma$ converges weakly, almost surely, to the Marchenko–Pastur law $\mu_{MP}$ , which has

density $\displaystyle \rho_\lambda(x) = \frac{\sqrt{(b-x)(x-a)}}{2\pi\sigma^2\lambda x}$ on $[a,b]$ , where $a = \sigma^2(1-\sqrt\lambda)^2$ and $b = \sigma^2(1+\sqrt\lambda)^2$ ;
plus an atom of mass $1 - 1/\lambda$ at $0$ when $\lambda > 1$ .

The proof (via the Stieltjes transform $s(z) = \int (x-z)^{-1}d\mu(x)$ , which satisfies a quadratic self-consistent equation in the limit) is in the references and is not reproduced here.

PropositionThe density integrates to one

For $\lambda \in (0,1]$ and $\sigma = 1$ , $\int_a^b \rho_\lambda(x)\,dx = 1$ .

PropositionBulk edges and conditioning

With $\sigma = 1$ and $\lambda = p/n \le 1$ , the extreme eigenvalues of $\hat\Sigma$ satisfy, almost surely in the limit,

\lambda_{\max}(\hat\Sigma) \to (1+\sqrt\lambda)^2, \qquad \lambda_{\min}(\hat\Sigma) \to (1-\sqrt\lambda)^2,

so the condition number satisfies

\kappa(\hat\Sigma) \to \left(\frac{1+\sqrt\lambda}{1-\sqrt\lambda}\right)^2 .

Key idea

Put numbers in. At $\lambda = 0.01$ ( $n = 100p$ ), $\kappa \approx 1.5$ — the sample covariance is essentially as well conditioned as the truth. At $\lambda = 0.5$ , $\kappa \approx 34$ . At $\lambda = 0.9$ , $\kappa \approx 1500$ . At $\lambda = 1$ the smallest eigenvalue reaches $0$ and $\hat\Sigma$ is singular. Every method that inverts a sample covariance — ordinary least squares, linear discriminant analysis, Gaussian maximum likelihood, Newton's method on a quadratic — degrades along exactly this curve, and the degradation is not a numerical artefact but the true geometry of the estimate.

Run it: Marchenko–Pastur lawEigenvalues of a sample covariance matrix against the Marchenko–Pastur density for aspect ratio λ = p/n.

CorollaryRecovering the classical limit

As $\lambda \downarrow 0$ the support $[(1-\sqrt\lambda)^2,(1+\sqrt\lambda)^2]$ shrinks to $\{1\}$ , so $\mu_{MP} \Rightarrow \delta_1$ . This is the classical statement: for $p$ fixed and $n\to\infty$ , $\hat\Sigma\to I_p$ and all eigenvalues converge to $1$ .

What the law does not describe

Empirical observationReal covariance spectra are not Marchenko–Pastur

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

Plot the eigenvalue histogram of the covariance of a real dataset — word embeddings, gene expression, daily asset returns — and it will typically show a Marchenko–Pastur-like bulk plus a small number of large outlying eigenvalues. The bulk is well described by the law; the outliers are not, and they are usually the interesting part.

What this does not say

The theorem above assumes the population covariance is $\sigma^2 I_p$ — no signal at all. Real data has signal, and the relevant model is the spiked covariance model: $\Sigma = I_p + \sum_{k=1}^r h_k v_kv_k^\top$ with $r$ small. There, the correct statement is the Baik–Ben Arous–Péché phase transition: a spike of strength $h$ is detectable in the sample spectrum if and only if $h > \sqrt\lambda$ , and below that threshold the corresponding sample eigenvalue is absorbed into the bulk and the sample eigenvector is asymptotically uncorrelated with $v_k$ .

That is a sharp and somewhat brutal statement: there is a signal strength below which no amount of eigen-analysis recovers the direction, not because the estimator is bad but because the information is not in the sample. It is quoted here from the reference and is a separate theorem, not a corollary of the Marchenko–Pastur law.

Problems

Marchenko–Pastur edges and conditioningNumerical · Beginner · 15 min The Marchenko–Pastur density integrates to oneProof · Advanced · 1 hour Simulate the Marchenko–Pastur lawCoding · Intermediate · 1 hour Why ordinary least squares degrades as p approaches nProof · Advanced · 30 min

References

PaperVladimir A. Marchenko and Leonid A. Pastur (1967). Distribution of eigenvalues for some sets of random matricesThe original result.
BookZhidong Bai and Jack W. Silverstein (2010). Spectral Analysis of Large Dimensional Random MatricesContains the modern proof via Stieltjes transforms, together with the spiked-model extensions.
PaperJinho Baik, Gérard Ben Arous and Sandrine Péché (2005). Phase transition of the largest eigenvalue for nonnull complex sample covariance matricesThe BBP transition, which is what actually governs spiked covariance models.
BookRoman Vershynin (2018). High-Dimensional Probability: An Introduction with Applications in Data Science

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

TheoremMarchenko–Pastur lawThe empirical spectral distribution of a sample covariance matrix with identity population covariance converges to an explicit density supported on an interval determined by the aspect ratio.
PropositionBulk edges and the conditioning of the sample covarianceproof includedThe extreme eigenvalues sit at (1 ± sqrt(lambda))^2, so the condition number blows up as the aspect ratio approaches one.
PropositionThe Marchenko–Pastur density integrates to oneproof includedA verification that the stated density is a probability density on its bulk when the aspect ratio is at most one.
CorollaryRecovering the classical limitproof includedAs the aspect ratio tends to zero the bulk collapses to a point mass at one, which is the classical fixed-dimension result.
Empirical observationReal covariance spectra are not Marchenko–PasturEmpirical spectra of real data typically show spiked structure that the identity-population law does not describe.

Experiments

Marchenko–Pastur law
Eigenvalues of a sample covariance matrix against the Marchenko–Pastur density for aspect ratio λ = p/n.

Problems (5)

All problems