Tool results and context

Every tool result you send back becomes part of the conversation, and the whole conversation is sent again on every following request. In a long tool loop, results are usually the largest and fastest-growing part of the prompt. This page covers how to keep them small, how to keep the prompt cacheable while they accumulate, and what to do when a conversation grows too large.

Size results on purpose

Set hard caps in your tool code (characters, lines, items) and say in the tool’s description what those caps are, so the model knows to ask for the next page when it needs more.

Order messages for caching

The prompt cache matches identical prompt beginnings. In a tool loop, each round re-sends everything before it unchanged, so most of the prompt is served from cache as long as you:

• keep the system message and tools identical across rounds;
• append new messages at the end, never edit earlier ones;
• avoid timestamps or random ids in early messages.

You can check what was cached from usage:

def cache_ratio(usage) -> float:
    u = usage.model_dump() if hasattr(usage, "model_dump") else usage
    details = u.get("prompt_tokens_details") or {}
    cached = u.get("cached_tokens") or details.get("cached_tokens") or 0
    return cached / max(1, u["prompt_tokens"])
 
# after each round of the loop:
# print(f"round {n}: {reply.usage.prompt_tokens} prompt tokens, {cache_ratio(reply.usage):.0%} cached")

function cacheRatio(usage) {
  const cached = usage.cached_tokens ?? usage.prompt_tokens_details?.cached_tokens ?? 0
  return cached / Math.max(1, usage.prompt_tokens)
}
// console.log(`round ${n}: ${reply.usage.prompt_tokens} prompt tokens, ${(cacheRatio(reply.usage) * 100).toFixed(0)}% cached`)

From the second round onwards, a well-ordered loop typically shows most of the prompt as cached. A low ratio usually means something early in the conversation changes between rounds.

Shrink old results as the loop goes on

Once the model has used a result, it rarely needs the full text again. Replace old results with a short note, keeping the tool_call_id so the conversation stays valid:

KEEP_LAST = 3   # most recent tool results kept in full
 
def compact_tool_results(messages: list) -> list:
    tool_idx = [i for i, m in enumerate(messages) if m.get("role") == "tool"]
    for i in tool_idx[:-KEEP_LAST]:
        content = messages[i]["content"]
        if len(content) > 300:
            messages[i] = {**messages[i], "content": content[:200] + f"... [trimmed; {len(content)} chars originally]"}
    return messages

This edits earlier messages, so the cache restarts from the first changed message. Do it occasionally (for example every ten rounds, or when the prompt passes a size threshold), not on every round.

When the conversation gets too large

The context window is 262,144 tokens for prompt and reply, and a request is limited to about 2 MB and 500 messages. Before you get close:

1. Compact old tool results as above.
2. Summarise the oldest part of the conversation into one message (see Build a chatbot with memory), keeping tool call and result pairs intact or dropping them together.
3. For agents, end the run and start a fresh one with a summary of progress as the first message.

Track usage.prompt_tokens per round and act on a threshold you choose (for example 150,000 tokens), rather than waiting for an error.

Related pages

• Design good tools
• Prompt caching
• Context windows