Evaluation Harnesses

Turning "is it good?" into a test suite that runs in CI — golden sets, graders, the failure modes of LLM-as-judge, and why a passing eval on a changed config proves nothing.

AI EngineeringGraduate⏱ 3 hours7 min readevaluationgolden setLLM as judgeregression testingCI

AssumesData Pipelines and Lineage · Rollout, Canaries and Online Experiments

Anatomy

DefinitionThe four parts of an eval

A dataset — inputs with, where possible, references. Versioned, immutable, in source control or an addressable store.
A runner — executes the system under an exactly pinned configuration and records raw outputs alongside that configuration.
A grader — turns each output into a score. The most consequential choice.
An aggregation and decision rule — how scores combine and what constitutes a regression, decided before looking at the result.

@dataclass(frozen=True)
class EvalRun:
    dataset: str          # "support-golden-v7"          immutable id
    config_hash: str      # model + prompt + tools + index + sampling
    results: list[Result]
    started_at: datetime
    git_sha: str

HeuristicAn eval result is meaningless without the configuration that produced it

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Behaviour is determined jointly by the model version, the system prompt, the tool schemas, the retrieval index version and the sampling parameters. An eval that records only the model version cannot answer "what changed?" when the score moves, and there are four other candidates.

def config_hash(cfg) -> str:
    return sha256(json.dumps({
        "model": cfg.model,                    # pinned, never an alias
        "system_prompt": sha256(cfg.system_prompt.encode()).hexdigest(),
        "tools": sorted((t.name, t.schema_hash) for t in cfg.tools),
        "index": (cfg.index_name, cfg.index_version),
        "sampling": {"temperature": cfg.temperature, "top_p": cfg.top_p, "seed": cfg.seed},
    }, sort_keys=True).encode()).hexdigest()

Choosing a grader

DefinitionThe grader ladder

Grader	Cost	Reliability	Use when
exact / normalised match	free	perfect	the answer is a value: a number, a label, an id
programmatic check	free	perfect within its rules	the output has structure: valid JSON, compiles, passes tests, cites a real id
reference similarity	cheap	moderate	a reference exists but wording may vary
model judge	expensive	biased, see below	open-ended quality with a written rubric
human	very expensive	the ground truth, with its own variance	calibrating everything above

Climb the ladder only when forced. A surprising fraction of what teams grade with a model judge is programmatically checkable: did it cite a document that exists? is the JSON valid against the schema? did it call the tool at all? is the refund amount in the answer equal to the one in the record?

Key idea

The best graders are assertions about verifiable properties, not judgements about quality. They are free, deterministic and never drift. Every open-ended criterion you can convert into a checkable one removes a source of noise from the suite permanently — and the conversion is usually a conversation about what "good" actually means, which is valuable independently of the eval.

Model judges

Empirical observationModel judges have systematic biases

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

Published evaluations of LLM judges report three biases in particular: a preference for longer answers independent of quality; a position effect when two candidates are compared in sequence; and self-enhancement, a preference for outputs produced by the same model family as the judge.

Assumptions

These are measurements of particular models on particular benchmarks, not universal laws, and their magnitudes vary. They are reported here because the mitigations below are cheap and worth applying regardless of the exact effect size in your setting.

Mitigations, in order of value for effort:

Randomise position and average both orders. Removes the position effect essentially entirely, at double the judge cost.
Grade against a rubric with explicit criteria, not "which is better". Rubrics reduce the verbosity effect because length is not one of the criteria.
Use a different model family to judge than the one generating, to avoid self-enhancement.
Calibrate against humans on a sample. Measure judge–human agreement on a hundred items and report it alongside every score. A judge with 70% agreement is measuring something, and it is not what you think.
Require a reason before the score, and log it. Reasons make disagreements inspectable, and an implausible reason on a passing item is a signal the rubric is wrong.

How big must the set be?

PropositionA golden set of size n cannot resolve differences below its noise floor

With binary grading, the pass rate $\hat p$ on $n$ independent items has standard error $\sqrt{p(1-p)/n} \le 1/(2\sqrt n)$ . Detecting a change of $\Delta$ in the pass rate with 80% power at the 5% level needs, by the two-sample sizing result,

n \;\ge\; \frac{2\,(z_{0.975}+z_{0.8})^2\,p(1-p)}{\Delta^2} \;\approx\; \frac{15.7\,p(1-p)}{\Delta^2} .

At $p = 0.9$ : detecting a 5-point drop needs $n \approx 565$ ; a 1-point drop needs $n \approx 14{,}130$ .

Key idea

A 50-item golden set has a standard error of about 4 percentage points. It cannot tell a 5% real regression from noise, which means most hand-curated eval sets are too small to detect the regressions they were built to catch. Either grow the set, or accept that it is a smoke test — useful for catching catastrophic breakage, useless for the incremental decisions people actually use it for.

Running it in CI

on: [pull_request]
jobs:
  eval:
    steps:
      - run: eval run --suite smoke --config-from-branch      # ~2 min, catches breakage
      - run: eval compare --against main --fail-on regression

Three tiers, by cost and cadence:

Tier	Size	Cadence	Blocks
smoke	50–100 items, programmatic graders only	every PR	yes
regression	500–5000 items, full ladder	nightly and pre-release	release
deep	everything, including human review	before a model change	model rollout

What this does not say

Offline evaluation measures the system against a fixed distribution that you chose, and the distribution is where the errors hide: the golden set contains the failures you already knew about. It cannot discover a new failure mode, and it systematically over-represents what someone thought to write down. The corrective is sampling production traffic into the eval set continuously — which is also what makes the drift-monitoring proposal in Drift Detection work, since the two need the same apparatus.

References

PaperL. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaMeasures position, verbosity and self-enhancement bias in model judges.
BookR. Kohavi, D. Tang, Y. Xu (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B TestingThe offline/online split and the guardrail-metric discipline assumed here.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionThe four parts of an evalA dataset, a runner pinned to an exact configuration, a grader, and an aggregation with a decision rule.
DefinitionThe grader ladderPrefer exact match, then programmatic checks, then reference-based similarity, then a model judge, then a human — cheaper and more reliable first.
Empirical observationModel judges have systematic biasesReported evaluations find judges favour longer answers, answers in the first position, and outputs from the model family doing the judging.
PropositionA golden set of size n cannot resolve differences below its noise floorproof includedWith binary grading on n items, the standard error of the pass rate is at most 1/(2 sqrt n), so small sets cannot distinguish small regressions.
HeuristicAn eval result is meaningless without the configuration that produced itModel, prompt, tools, retrieval index and sampling parameters jointly determine behaviour, so all of them must be pinned and recorded.