The RAG evidence lower bound
ProofAdvanced⏱ 30 minRAGelbovariationalretriever trainingposterior
Problem
Derive the evidence lower bound for the RAG marginal likelihood and identify the distribution that closes the gap. Explain what this says about what a retriever should be trained to approximate.
Write a complete argument. Every inequality you use should be one you could state precisely.
Solution
Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.
Related problems
All problems| Solved | Problem | Field | Type | Difficulty | Time |
|---|---|---|---|---|---|
| End-to-end recall arithmetic for a RAG system | RAG | Numerical | 15 min | ||
| RAG-sequence and RAG-token are different models | RAG | Proof | 30 min | ||
| Build and instrument a minimal RAG pipeline run | RAG | Coding | 1 day | ||
| Closed form of the KL-regularised RL optimum abyss | LLMs | Proof | 1 hour | ||
| Retrieval error propagates into generation abyss | RAG | Proof | 1 hour |