Drift Detection

Four things called drift

DefinitionFour things called drift

Write $P(x)$ for the input distribution and $P(y \mid x)$ for the relationship being modelled.

Name	What changed	Example	Remedy
Covariate shift	$P(x)$	new customer segment asks different questions	retrain or extend coverage
Prior shift	$P(y)$	fraud rate triples	recalibrate thresholds
Concept drift	$P(y \mid x)$	the refund policy changed, so the right answer changed	update the corpus, retrain
Upstream corruption	$x$ itself is wrong	a field stopped being populated	fix the pipeline

The fourth is not drift at all and is the most common in practice. It is listed here because it presents identically on every monitoring dashboard, and because treating it as drift leads to retraining a model on corrupted data — which converts a fixable bug into a permanent one.

PSI, and what it is

DefinitionThe population stability index

Bin a variable into $k$ buckets. Let $p_i$ be the fraction of the reference sample in bucket $i$ and $q_i$ the fraction of the current sample. Then

$$\mathrm{PSI}(p, q) \;=\; \sum_{i=1}^{k} (p_i - q_i)\,\ln\frac{p_i}{q_i} .$$

Buckets are built from the reference distribution — usually deciles — and held fixed thereafter.

PropositionPSI is the symmetrised KL divergence

$$\mathrm{PSI}(p,q) \;=\; D_{\mathrm{KL}}(p \,\|\, q) \;+\; D_{\mathrm{KL}}(q \,\|\, p) ,$$

the Jeffreys divergence. Consequently $\mathrm{PSI} \ge 0$, with equality if and only if $p_i = q_i$ for every $i$, and it is symmetric in its arguments.

ProofReveal

HeuristicThe conventional PSI thresholds

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

PSI	Conventional reading
$< 0.1$	no significant change
$0.1 - 0.25$	moderate change, investigate
$> 0.25$	significant change, act

Embedding distributions

For an LLM system the input is text, and the useful distributional handle is its embedding.

def embedding_drift(reference: np.ndarray, current: np.ndarray) -> dict:
    """Three complementary views of how an embedding cloud has moved."""
    mu_r, mu_c = reference.mean(0), current.mean(0)
    centroid_shift = np.linalg.norm(mu_c - mu_r) / np.linalg.norm(mu_r)

    # Spread: has the cloud changed shape as well as position?
    spread_ratio = current.std(0).mean() / reference.std(0).mean()

    # Coverage: what fraction of current queries land far from everything we have seen?
    d = pairwise_min_distance(current, reference)          # nearest reference neighbour
    tail = float(np.mean(d > np.quantile(pairwise_min_distance(reference, reference), 0.99)))

    return {"centroid_shift": centroid_shift, "spread_ratio": spread_ratio, "novel_fraction": tail}

The third number is the operationally useful one: the fraction of current queries that are unlike anything in the reference set. A rising novel fraction is the earliest signal that the retrieval corpus no longer covers what people are asking, and unlike PSI it points at specific examples you can read.

Alerting on many things at once

PropositionMonitoring many features guarantees false alarms

Run an independent level-$\alpha$ test on each of $m$ features every period. Under the null — no drift anywhere — the probability of at least one alarm in a single period is $1-(1-\alpha)^m$, and over $T$ periods

$$\Pr[\text{at least one false alarm}] \;=\; 1-(1-\alpha)^{mT} \;\xrightarrow[mT\to\infty]{}\; 1 .$$

With $m = 50$ features at $\alpha = 0.01$, the daily false-alarm probability is $1 - 0.99^{50} = 0.395$, and over a 30-day month it is $1 - 0.99^{1500} \approx 1$.

ProofReveal

When the labels are late

HeuristicMonitor proxies when labels are late

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Ground truth for a generative system arrives weeks later, incompletely, or never. These move earlier and are available immediately:

Proxy	Reads on	Typical failure it catches
refusal rate	outputs	prompt or policy change, safety filter regression
mean output length	outputs	truncation bugs, model swap, prompt regression
tool-call rate and failure rate	outputs	tool schema drift, broken integration
retrieval score distribution	retrieval	index staleness, embedding mismatch
"no relevant documents" rate	retrieval	corpus coverage gap
user retry / rephrase rate	behaviour	answer quality, broadly
escalation-to-human rate	behaviour	the closest thing to a quality label
thumbs-down rate	behaviour	quality, heavily biased in sampling

Each is weak individually. Together, and with per-segment breakdowns, they catch most regressions before any labelled data exists.