Tool Loops in Production

The agent loop as a piece of software: schema design, validation, parallelism, truncation, side effects and the termination conditions that stop it running forever.

AI EngineeringGraduate⏱ 3 hours6 min readagentstool callingschemasidempotencyretries

AssumesThe Model Context Protocol · Error Compounding in Agent Trajectories

The loop

DefinitionThe loop

async def run(conversation, tools, budget: Budget) -> Result:
    while True:
        budget.check()                                  # raises on any exhausted dimension
        reply = await model.generate(conversation, tools=tools, deadline=budget.deadline)
        conversation.append(reply)

        calls = reply.tool_calls
        if not calls:
            return Result(reply, budget.spent())        # the model is done

        results = await asyncio.gather(*[
            execute(c, deadline=budget.remaining()) for c in calls
        ], return_exceptions=True)

        for call, result in zip(calls, results):
            conversation.append(tool_result(call.id, render(result)))
        budget.charge(reply.usage, results)

Three properties are load-bearing and easy to lose. Tool results are appended in the same order as the calls and tagged with the call id, or the model cannot tell which result belongs to which request. Failures are appended as results with an error flag rather than raised, so the model can react to them. And the budget is checked at the top of every iteration, not only at entry.

Making it stop

DefinitionFour independent stopping conditions

Condition	Guards against	Typical value
model returns no tool calls	the normal case	—
step budget	a loop that alternates between two tools forever	10–50
token budget	a loop that makes progress but costs unboundedly	a hard ceiling
wall-clock deadline	a slow tool chain that will outlive the caller	inherited from the request

All four are required and none subsumes another: a loop can be cheap and endless, expensive in few steps, or fast and unbounded. Exhausting any of them should return a partial result with a reason the caller can see, not an exception that discards the work already done.

Context growth

PropositionTool results grow the context quadratically in cost

Each step resends the entire transcript. If step $i$ appends $r$ tokens of tool output and the conversation starts at $c_0$ tokens, the prefill cost over $n$ steps is

\sum_{i=1}^{n}\bigl(c_0 + (i-1)r\bigr) \;=\; n c_0 + r\,\frac{n(n-1)}{2} \;=\; O(r n^2) .

Key idea

Twenty steps each returning a 5,000-token tool result is $5000 \times 190 = 950{,}000$ prefill tokens of accumulated re-reading on top of the conversation itself. Prefix caching (Caching — Hit Rates, Zipf and the Working Set) removes most of it — the transcript is an exact prefix that only grows — which is why prefix caching is not optional for agents. Where it is unavailable, the quadratic is the dominant cost of the whole system.

Truncation that does not destroy the trace

Large results must be reduced before they enter the transcript, and how matters:

def render(result, limit=2000) -> str:
    if result.tokens <= limit:
        return result.text
    # Keep the head and the tail: schemas and headers live at the top, errors and
    # totals at the bottom. Cutting only the tail loses exactly the useful half.
    head, tail = result.head(limit // 2), result.tail(limit // 2 - 40)
    return (f"{head}\n\n[... {result.tokens - limit} tokens elided; "
            f"full output at {result.artifact_uri} ...]\n\n{tail}")

Storing the full output somewhere addressable and giving the model a handle preserves the option of fetching more, which naive truncation removes.

Side effects

HeuristicMake tools idempotent or make them confirmable

Heuristic — a reasoning shortcut that predicts the right answer in the regimes discussed here. It is not a proof and can fail outside them.

Tool calls get retried — by the loop on timeout, by the caller on a dropped connection, by the model because it did not see the result. Every tool must therefore be in one of two categories:

Idempotent — running twice is the same as running once. Achieved the same way as in API Design for Model Endpoints: the loop generates a key per logical call and passes it through.

await execute(call, idempotency_key=f"{run_id}:{step}:{hash(call.arguments)}")

Confirmable — the tool is split into a proposal and a commit, with the commit requiring a token the model cannot produce:

create_transfer(...)  → {"status": "pending", "confirm_token": "ct_9f2b...",
                         "summary": "Transfer £4,200 to account ****8891"}
                        ← host shows the summary to the user, who approves
confirm(ct_9f2b...)   → executed once, token consumed

A tool that is neither is a latent duplicate-side-effect bug, and it will fire during an incident rather than during testing.

How many tools

Empirical observationSelection accuracy degrades with tool count

Empirical observation — a regularity seen in experiments. No theorem on this page establishes it in general.

Published tool-use evaluations report that a model's ability to select the right tool falls as the number of simultaneously exposed tools grows, with overlapping or similarly-described tools degrading it fastest.

Assumptions

This is an empirical observation about current models, not a property of the protocol, and the numbers vary by model and by how distinct the tool descriptions are. It is stated as an observation because that is what it is; the engineering response below does not depend on the exact figures.

The response is to treat the tool list as a retrieval problem rather than a fixed manifest:

Filter by context — expose only tools relevant to the current task or the user's permissions. A read-only user does not need the write tools in their context at all.
Group behind a router — one database tool with an operation argument rather than fifteen flat tools, when the fifteen share a domain.
Retrieve tools by embedding — for large catalogues, select the top- $k$ tools by similarity to the task before the model sees them, which is the same mechanism as document retrieval pointed at the manifest.
Write descriptions that discriminate. Most selection errors are between two tools whose descriptions do not say how they differ. Say it explicitly: "use this for X, not for Y — for Y use other_tool."

What this does not say

This page covers one loop with one model. Multi-agent systems, where one loop spawns others, add a supervision problem this does not address: budgets must be hierarchical or a child loop can exhaust the parent's, failures must propagate with enough context to be actionable, and the compounding analysis in Error Compounding in Agent Trajectories applies at each level, so per-step reliability has to be raised to the power of the product of the horizons rather than to either alone.

References

Lecture notesAnthropic (2024). Model Context Protocol — SpecificationTool schema and result shapes assumed throughout.
PaperS. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao (2023). ReAct: Synergizing Reasoning and Acting in Language ModelsThe interleaved reason-act loop this page treats as the implementation baseline.

These works are cited as sources to read, not reproduced. All exposition, examples and problems on this site are written independently; where an argument is standard, the citation points to a canonical treatment.

Results stated on this page

DefinitionThe loopAppend the user turn, call the model, execute any requested tools, append their results, repeat until the model stops asking or a budget is exhausted.
DefinitionFour independent stopping conditionsA production loop terminates on model completion, step budget, token budget or wall-clock deadline, and all four must exist.
PropositionTool results grow the context quadratically in costproof includedBecause every step resends the whole transcript, appending r tokens per step costs O(r n squared) prefill tokens over n steps.
HeuristicMake tools idempotent or make them confirmableA retried or replayed tool call must either be safe to run twice or require an explicit confirmation step that the model cannot forge.
Empirical observationSelection accuracy degrades with tool countReported evaluations find tool-selection accuracy falls as the number of simultaneously exposed tools grows, motivating retrieval over the tool list itself.