Simulate an agent pipeline end to end

CodingAdvanced⏱ 1 dayAgentssimulationagentscorrelationverification

Theory: Error Compounding in Agent Trajectories →

Problem

Build a simulator that makes the theory's assumptions visible.

Simulate trajectories with independent steps and verify $p^n$ empirically.
Add correlated failures via a shared latent "task difficulty" and measure the gap from $p^n$ .
Add detection and retry; verify the $p_{\text{eff}}$ formula and then break it by making detection correlated with the failure.
Add majority voting and measure the effective independent vote count.

Implement it, then check the implementation against something you can verify independently.

Write it

Solve it here9 tests · Python runs in your browser

import numpy as np

def independent_success(n_steps, p, trials=20000, seed=0):
    """Fraction of trajectories in which all n_steps succeed.

Each step succeeds independently with probability p. Simulate `trials`
    trajectories with a generator seeded with `seed` and return the observed
    fraction. This is the baseline the theory predicts at p**n_steps.
    """
    raise NotImplementedError

def correlated_success(n_steps, p_easy, p_hard, prob_hard, trials=20000, seed=0):
    """The same measurement when failures share a cause.

Each trajectory first draws a latent difficulty: hard with probability
    prob_hard, easy otherwise. Every step of that trajectory then succeeds
    independently with p_hard or p_easy accordingly. Return the fraction of
    trajectories that complete.

The per-step marginal is still p_bar = prob_hard*p_hard +
    (1 - prob_hard)*p_easy, but the trajectory rate is not p_bar**n_steps.
    """
    raise NotImplementedError

def p_effective(p, detect, retries):
    """Per-step success once failures can be caught and retried.

A step succeeds with probability p. If it fails it is detected with
    probability `detect`, and a detected failure is retried — up to `retries`
    extra attempts, each an independent fresh attempt with success p. An
    undetected failure ends the step as a failure.

Return the resulting probability that the step ultimately succeeds, in
    closed form rather than by simulation.
    """
    raise NotImplementedError

def majority_vote_accuracy(p, k, trials=20000, seed=0):
    """Accuracy of a majority vote over k independent attempts.

Each of the k attempts is correct independently with probability p; the
    vote is correct when strictly more than k/2 of them are. Take k odd.
    """
    raise NotImplementedError

⌘/Ctrl + ↵ to run · limit 90s · draft saved in this browser

9 tests will run against your code. The first run downloads the Python runtime — several megabytes, once per browser.

Solution

Opening this before you have a real attempt on paper is the fastest way to feel like you understand something you cannot reproduce.

Problem	Field	Type	Difficulty	Time
Trajectory success from per-step reliability	Agents	Numerical	Foundation	15 min
Simulate distance contrast run	Classical ML	Coding	Beginner	30 min
The reliability wall	Agents	Numerical	Beginner	15 min
Balls into bins, with one choice and with two run	AI Systems	Coding	Intermediate	1 hour
Cache hit rates under Zipf popularity run	AI Systems	Coding	Intermediate	1 hour

Problem

Write it

Solution

Related problems