Why does the bill grow when token prices fall?
Cost per task is calls per task × tokens per call × price per token. Vendors keep cutting the third factor. An agent pushes up the first two (the loop itself is described in how AI agents work). One task means a plan step, several tool calls, an observation after each, a retry or two, and each call carries the full conversation so far. Without a cached prefix, input cost grows faster than the step count, because step 12 pays again for the eleven steps before it.
The research matches what we see in logs. Bai et al. (arXiv:2604.22750, revised 2 October 2026) studied eight frontier models on SWE-bench Verified. Agentic tasks used about 1,000× more tokens than code chat, and input tokens drove the cost even with caching on. Repeated runs of the same task differed by up to 30× in total tokens. Accuracy did not keep rising with spend.
Finance teams have noticed. The FinOps Foundation's State of FinOps 2026 (1,192 respondents, $83bn+ annual cloud spend) reports that 98% of respondents now manage AI spend, up from 63% in 2025 and 31% in 2024. Their top challenge with AI is visibility, and the most requested capability is granular monitoring of tokens and LLM requests. The report does not say how far AI projects overran their budgets. You will see that claim in blog posts, and we could not trace it to this report.
Harness's vendor survey (700 engineering leaders, May–June 2026) is blunter. 72% had an unexpected AI cost spike in the past 12 months, 79% would need a full day or longer to trace one, and 52% say nobody clearly owns AI cost.
Measure cost per task before touching a prompt
Log one row per model call, keyed by task, and compute everything else from it. The number that matters is cost per completed task. Cost per call hides retries and dead-end runs. Next to it we track cache hit rate, retries per task and the share of input tokens that came from tool results. That last one is usually the surprise.
Both major APIs return what you need. Anthropic's usage object has input_tokens, cache_creation_input_tokens and cache_read_input_tokens, and the docs say plainly that zero in both cache fields means nothing was cached. OpenAI reports usage.input_tokens_details.cached_tokens and, on GPT-5.6 and later, cache_write_tokens. Our row looks like this:
{
"task_id": "inv-2026-10-16-0042",
"tenant_id": "acme-eu",
"step": 7,
"model": "claude-sonnet-5-5",
"input_tokens": 1840,
"cache_read_tokens": 21500,
"cache_write_tokens": 0,
"output_tokens": 412,
"tool_result_tokens": 9300,
"latency_ms": 2710,
"outcome": "tool_call"
}outcome on the final row (done, escalated, aborted_budget) is what turns call logs into unit economics. Without it, you cannot tell a cheap success from a cheap failure.
Cache hits are a prompt-design problem
Caching only pays when the prefix is byte-identical between calls. Put stable content first and volatile content last, and never edit earlier turns. On Anthropic the prefix order is tools, then system, then messages, and a change at one level invalidates that level and everything after it. Change one tool definition and you have thrown away the whole cache.
Current Anthropic prices (docs accessed 16 October 2026): a 5-minute cache write costs 1.25× the base input price, a 1-hour write 2×, and a read 0.1×. Claude Opus 5.5 and Sonnet 5.5 read at 0.05× ($0.20 against a $4 base per million tokens for Opus 5.5). The default TTL is five minutes and it refreshes free on every hit. You get up to four breakpoints, and the minimum cacheable prompt is 512 tokens on current models (up to 4,096 on Opus 4.5/4.6 and Haiku 4.5). Below the minimum the request simply runs uncached, with no error, which is easy to miss.
The arithmetic decides when not to bother. A prefix used once costs 1.25× instead of 1×, so a one-off loses 25%. Used twice, it costs 1.35× instead of 2×. An agent that makes ten calls inside five minutes pays for the prefix roughly 2.15× in total, against 10× uncached.
The other vendors differ in the details. OpenAI caches automatically. On GPT-5.6 and later the minimum is 1,024 tokens, reads are 0.1× on most models, writes are 1.25× and the TTL is 30 minutes. Gemini's implicit caching is on by default for 2.5 and newer, with a 4,096-token minimum on the Gemini 3.x Flash models and Gemini 4 Argon. Its docs give no discount figure, so check the pricing page for your model.
The mistakes we look for first:
- a timestamp or request ID in the system prompt (it kills every hit after it);
- tools added or removed per request (OpenAI's docs suggest
allowed_toolsortool_choice: "none"instead); - history "cleaned up" by rewriting old turns instead of appending.
Trimming tool results also breaks the cache. Anthropic's tool result clearing (clear_tool_uses_20250919) invalidates the cached prefix wherever it removes content. That is why it has a clear_at_least parameter: clear a big chunk once and pay one cache write, or keep clearing small chunks and pay a write every time.
Where agent tokens go and what we do about it
Use it as a checklist for a log review. The trade-off column is why none of these fixes is free.
| Cost source | Symptom in logs | Fix | Trade-off |
|---|---|---|---|
| Re-sent history | Input tokens climb step by step within a task | Cache the stable prefix; summarise turns older than N steps | Summaries lose detail and rewrite the prefix |
| Tool result bloat | `tool_result_tokens` over half of input | Return IDs and fields, not whole documents; clear old results | The model may need to call the tool again |
| Retries and loops | Same tool + same arguments 3+ times in one task | Repeat-call breaker, per-task token cap | Some hard tasks get cut off and escalated |
| Wrong model tier | Frontier model on classification or formatting steps | Route by step type, gated by evals | Extra code path, extra eval upkeep |
| Uncached stable prefix | Both cache fields at 0 on repeat calls | Move volatile content to the end, freeze tool order | Less freedom to tweak prompts per request |
| Oversized RAG context | 20+ chunks per call, low citation rate | Rerank and send the top few | Recall risk on long-tail questions |
Routing: a smaller model has to pass the same evals
Route by step type. Classification, field extraction and tool-argument formatting go to the small model; planning, judgement and anything customer-facing go to the strong one. Below a confidence threshold, the step falls back to the larger model. We accept a cheaper tier for a step only after it passes the same eval set, built from the client's own data, that the expensive tier passed.
That rule is slow. It is also the only defence against a router that saves money on 95% of calls and quietly breaks the other 5%, which tend to be the expensive ones to get wrong.
The costs of routing are real: one more network hop, one more failure mode (the classifier itself), and one more thing to re-evaluate whenever a vendor ships a model. The Bai et al. data adds a twist. Models consumed very different token volumes on identical tasks (Kimi-K2 and Claude Sonnet 4.5 averaged over 1.5 million more tokens than GPT-5), so a lower list price can still give a higher cost per task. We compare tiers on cost per completed task from the eval run. We don't publish a savings percentage because it depends entirely on the mix of steps.
Hard stops for runaway loops
Every task gets a token cap and a step cap. Every tenant gets a daily budget with alerts at fixed thresholds. A breaker trips on repeated identical tool calls, and anything that hits a limit goes to a human with the trace attached. We cap by tokens first because a step cap alone lets one step that reads a 200-page PDF through.
class BudgetExceeded(Exception):
pass
class TaskBudget:
def __init__(self, max_tokens=150_000, max_steps=25, max_repeats=3):
self.max_tokens, self.max_steps, self.max_repeats = max_tokens, max_steps, max_repeats
self.tokens, self.steps, self.calls = 0, 0, {}
def charge(self, usage, tool_name=None, tool_args=None):
self.steps += 1
self.tokens += (usage["input_tokens"] + usage["output_tokens"]
+ usage.get("cache_read_tokens", 0) + usage.get("cache_write_tokens", 0))
if tool_name:
key = (tool_name, repr(sorted(tool_args.items())))
self.calls[key] = self.calls.get(key, 0) + 1
if self.calls[key] > self.max_repeats:
raise BudgetExceeded(f"repeated call {tool_name}")
if self.tokens > self.max_tokens or self.steps > self.max_steps:
raise BudgetExceeded(f"{self.tokens} tokens, {self.steps} steps")Cached reads count against the cap at full weight here. That is on purpose: the cap guards against runaway behaviour, not just money, and a loop on a fully cached context is still a loop. Provider-side spend limits sit on top as the last fence, set well above the sum of tenant budgets.
For anything that is not interactive (nightly classification, backfills, eval runs) we use batch APIs. Anthropic's Message Batches cost 50% less, most batches finish within an hour, and unfinished requests expire after 24 hours unbilled. The docs suggest the 1-hour cache TTL for batches, since five minutes is often too short.
RAG and long context: fewer tokens in
For retrieval-heavy agents, the cheapest token is the one never sent. We rerank before the model and pass the top few chunks, return fields instead of whole documents from tools, and summarise turns older than a set window instead of sending the full history. The RAG production checklist covers the retrieval side. RAG, fine-tuning or long context covers when a cached long context beats retrieval on cost.
One warning from fresh research. Satish et al. (arXiv:2609.32961, 26 September 2026) ran nearly 35,000 agent runs and found that compression policies using about a third of the tokens could take 20–80% longer. Fewer tokens did not always mean cheaper or faster. Compression changes the prefix, the number of calls and the cache behaviour all at once. So we measure cost per completed task before and after, never tokens alone.
What to report to finance
Report cost per business outcome: per ticket resolved, per invoice processed, per document reviewed. Split it by workflow and by team or client (showback), with the escalation rate next to it. A falling token count with a rising escalation rate is not a saving.
The standards are catching up. The FOCUS 1.5 working draft adds optional TokenDirection (Input/Output) and TokenCacheAction (Uncached/Read/Write) properties plus model identity fields, with ratification targeted for 3 December 2026. Until then FOCUS 1.4 is the published version, so we keep our own schema close to those names. The Linux Foundation launched the Tokenomics Foundation on 4 August 2026, with token cost telemetry in FOCUS and a "cost to serve" method on its roadmap. OpenCost added "AI Inference Costs v1" in release 1.121.0 (20 July 2026). None of these replaces per-task logging inside the agent, because they see invoices and clusters, not tasks.
What we leave alone
A prompt that runs once a day does not need caching (it loses 25% on the write). A workflow that costs less per month than a day of engineering time does not need a router. We also don't route quality-critical steps, such as final answers to customers or anything with legal or financial effect, to a cheaper tier just because it passed an average-case eval. Security controls stay out of the cost discussion entirely; see securing AI agents in production. If you want the per-task log and budget guard built into an existing agent, that is part of our AI development work.