Build and instrument a minimal RAG pipeline

CodingAdvanced⏱ 1 dayRAGragimplementationinstrumentationevaluation

Theory: Retrieval as a Latent-Variable Model →

Problem

Implement a minimal retrieval pipeline end to end and instrument it so that each stage's recall is measurable.

Chunk a corpus, embed, index, retrieve top- $k$ , and construct a context.
Measure chunk recall, index recall (against brute force) and truncation recall separately.
Show the effect of normalisation on ranking, and of chunk size on chunk recall.
Demonstrate the recall ceiling by computing end-to-end accuracy against the product of stage recalls.

Implement it, then check the implementation against something you can verify independently.

Write it

Solve it here10 tests · Python runs in your browser

import numpy as np

def chunk_spans(n_tokens, size, overlap):
    """Split [0, n_tokens) into overlapping chunks.

Return a list of (start, end) pairs. Chunks start at 0, size - overlap,
    2*(size - overlap), ... for as long as the start is below n_tokens, and
    each one ends at min(start + size, n_tokens). `overlap` is strictly less
    than `size`.
    """
    raise NotImplementedError

def chunk_recall(spans, answer):
    """1.0 if some chunk contains the whole answer span, else 0.0.

`answer` is a (start, end) pair. An answer split across two chunks is
    unreachable no matter how good the retriever is, which is why this is the
    first ceiling on the pipeline.
    """
    raise NotImplementedError

def top_k(query, index, k, normalise=True):
    """Indices of the k best matches in `index` (shape (n, d)), best first.

With normalise=True rank by cosine similarity; with normalise=False rank
    by the raw dot product. The two orderings are not the same, and which one
    you get is decided by whether you remembered to normalise.
    """
    raise NotImplementedError

def recall_at_k(retrieved, relevant):
    """Fraction of `relevant` ids that appear in `retrieved`."""
    raise NotImplementedError

def pipeline_recall(*stage_recalls):
    """The end-to-end ceiling: the product of the stage recalls.

Every stage can only lose documents, so the best the pipeline can do is
    the product — which is why three "95% good" stages give 86%.
    """
    raise NotImplementedError

⌘/Ctrl + ↵ to run · limit 60s · draft saved in this browser

10 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
End-to-end recall arithmetic for a RAG system	RAG	Numerical	Beginner	15 min
Perplexity is not comparable across tokenisers	LLMs	Numerical	Beginner	15 min
Implement backpropagation from scratch run	Deep Learning	Coding	Intermediate	3 hours
Implement kernel ridge regression run	Classical ML	Coding	Intermediate	1 hour
Implement multi-head attention from scratch run	Deep Learning	Coding	Intermediate	3 hours

Problem

Write it

Solution

Related problems