Data Pipelines and Lineage

The pipeline that feeds a model — ingestion, validation, chunking, embedding, indexing — and the lineage records that make "why did it answer that?" a question with an answer.

AI EngineeringAdvanced⏱ 3 hours6 min readMLOpsdata pipelinelineageversioningidempotency

AssumesThe Anatomy of an AI System

The stages

DefinitionThe stages, and what each may not do

source ──► ingest ──► validate ──► transform ──► embed ──► index ──► publish
             │           │            │            │         │          │
          raw copy    gate, not    chunk, clean,  batch,   upsert,   atomic
          retained     filter       enrich        cache    version    swap

Stage	Owns	Must not
ingest	fetching, deduplication, raw retention	transform anything
validate	schema and statistical checks	silently drop bad records
transform	chunking, cleaning, metadata extraction	depend on wall-clock time
embed	vectorisation, batching, caching	mix embedding model versions
index	upsert into the vector store, build	serve from a half-built index
publish	atomic pointer swap to the new version	leave two versions live

Replayability

HeuristicEvery stage must be idempotent and replayable

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Pipelines fail halfway: a node dies, an API rate-limits, a deploy lands mid-run. Recovery must be "run it again", which requires each stage to be a pure function of its inputs and its configuration:

Address outputs by a content hash of (input, code version, config), so a re-run writes the same key and a completed unit is skipped rather than redone.
Never use now() inside a transform. Pass the logical processing date in as a parameter, so a backfill for last Tuesday produces what last Tuesday's run would have.
Make writes upserts keyed on a stable document id, never appends.

def unit_key(doc, code_version, config) -> str:
    return sha256(f"{doc.id}:{doc.content_hash}:{code_version}:{config.hash}".encode()).hexdigest()

async def embed_stage(docs, code_version, config):
    for doc in docs:
        key = unit_key(doc, code_version, config)
        if await store.exists(key):
            continue                              # already done, in a previous attempt
        await store.put(key, await embedder.embed(doc.chunks))

Key idea

The content-hash key does three jobs at once: it makes retries free, it makes backfills incremental (only changed documents re-run), and it makes a config change visibly invalidate everything it should — because the hash changes and every unit is recomputed. A pipeline keyed on document id alone silently serves stale artefacts after a config change, which is the bug that takes weeks to find.

Lineage

DefinitionThe lineage record

Every artefact carries the inputs and the code that produced it:

{ "artifact": "chunk:doc-8812:14",
  "produced_at": "2026-09-21T09:14:02Z",
  "logical_date": "2026-09-20",
  "source": { "uri": "s3://corpus/policies/refunds.md",
              "content_hash": "sha256:3f1a...", "fetched_at": "2026-09-20T02:11:00Z" },
  "code": { "pipeline": "corpus-v4", "git_sha": "a91f3c2" },
  "config": { "chunker": "markdown-sections", "size": 512, "overlap": 64,
              "embedder": "embed-v3", "config_hash": "sha256:77be..." },
  "index": { "name": "policies", "version": 41 } }

With this, three otherwise unanswerable questions become lookups:

Why did the model say that? → the response cites chunk ids → the chunks cite source documents and versions.
This document changed — what is now stale? → find artefacts whose source.content_hash is superseded.
We are deprecating embed-v3 — what depends on it? → query artefacts by config.embedder.

Skew

DefinitionTraining–serving skew

A feature is computed one way when the model is built and another way when it is served. The model is correct on the offline data and wrong in production, and every offline metric says it is fine.

Common forms in a retrieval system:

Skew	How it happens
Different chunker	the eval set was chunked with the old settings
Different embedder	the index was built with `embed-v2`, queries are embedded with `v3`
Different normalisation	offline lower-cases and strips accents, online does not
Time leakage	the eval corpus contains documents published after the query date

Key idea

The structural defence is that there is one implementation, used by both paths, parameterised by configuration rather than duplicated. When offline and online are separate code, they will drift, and the drift is undetectable by construction — both sides work, they simply disagree.

The detection of last resort is an assertion in the serving path: embed a fixed canary string and compare its vector against the one recorded when the index was built. If they differ, the query embedder and the index embedder are not the same, and that is a hard failure rather than a degradation.

Migrations

PropositionRe-embedding is the dominant cost of changing an embedding model

Let a corpus have $M$ chunks and let embedding cost $c_e$ per chunk. Changing embedding model requires re-embedding every chunk, costing $M c_e$ , regardless of query volume — the vectors from two different models are not comparable, so there is no incremental path.

Key idea

Two practical consequences. First, the migration is a batch job whose cost you can compute in advance — $M c_e$ — and it is usually large enough to need its own budget approval. Second, it must be done into a new index version, with an atomic pointer swap at the end, because a half-migrated index mixes two embedding spaces and returns nonsense for the portion that has moved.

The dual-index pattern is the standard shape: build version $n+1$ alongside $n$ , shadow-query both, compare recall on a golden set, then swap. It costs double storage for the duration and it is the only way to find out whether the new model is actually better before committing.

What this does not say

This page treats the corpus as a batch. Streaming ingestion — documents arriving continuously, needing to be searchable within seconds — changes the index stage substantially: most vector indexes are cheaper to build in bulk than to update incrementally, so a streaming design usually maintains a small hot index of recent documents alongside a large periodically-rebuilt cold one and queries both. The freshness/latency trade that implies is covered in Vector Search Infrastructure.

References

PaperD. Sculley et al. (2015). Hidden Technical Debt in Machine Learning SystemsSource of the glue-code, configuration-debt and entanglement failure modes described here.
PaperE. Breck, N. Polyzotis, S. Roy, S. Whang, M. Zinkevich (2019). Data Validation for Machine LearningThe schema-and-statistics validation approach used in the validation stage below.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionThe stages, and what each may not doIngest, validate, transform, embed, index and publish, with validation as a gate rather than a filter.
HeuristicEvery stage must be idempotent and replayablePipelines fail partway, so each stage must be safe to re-run over the same input and produce the same output.
DefinitionThe lineage recordEach artefact records the exact inputs, code version and configuration that produced it, so any output can be traced back to its sources.
DefinitionTraining–serving skewA discrepancy between how a feature is computed offline and online, which produces a model that is correct in evaluation and wrong in production.
PropositionRe-embedding is the dominant cost of changing an embedding modelproof includedSwitching embedding models requires re-embedding the whole corpus, so the migration cost is linear in corpus size and independent of query volume.