🧨The failure mode: invoice shock
Every team running agents in production has a version of this story. A prompt change, a retry loop, or a traffic spike quietly multiplies token consumption. Nothing errors. Dashboards stay green, because they measure uptime and latency, not spend. Thirty days later the provider invoice arrives and it is four times the usual number, and the incident review starts with "when did this start?" — a question nobody can answer because nobody was measuring cost as a first-class signal.
Agents make this worse than ordinary API usage for a structural reason: a single user action triggers a variable, unbounded amount of work. A chat completion costs one call. An agent task costs one call, or twelve, or — if the planner cannot make progress and nobody capped the loop — four hundred. The variance is not a bug in the pricing; it is the nature of autonomous execution. Your monitoring has to match that shape.
The fix is not a bigger budget. It is treating cost as an operational metric with the same discipline as latency: attributed, alerted on, and enforced at runtime. This article is the working setup.
If your cost data lives only in the provider billing console, your detection latency is one billing cycle. Every control in this article exists to move detection from thirty days to thirty seconds.
🏷️Cost attribution: per agent, per task, per user
You cannot control what you cannot attribute. Provider dashboards give you organization-level totals, sometimes split by API key or model. That granularity answers "what did we spend?" but not the questions that matter operationally: which agent spent it, on which task type, on behalf of which user or tenant.
The mechanism is straightforward: tag every model call with a structured cost context at the point of invocation. Minimum viable set: agent identifier, task or session identifier, user or tenant identifier, feature or workflow name, and model name. Multiply tokens by your per-model rates (input and output separately — output tokens cost several times input on most pricing sheets) and emit the result as a metric and a log line. Do this in a single wrapper around your model client, not scattered across call sites.
Two subtleties separate useful attribution from noise. First, attribute at the task level, not the request level. An agent task is a tree of model calls and tool calls; the number you want is the total cost of the tree, rolled up under one task ID. Second, record the retry lineage. When a call fails and the framework retries with a longer context, the retry is where the money goes — and retries that do not carry the parent task ID become unattributed spend, which is how invoice shock hides.
Store the rollups somewhere queryable by the people who need the answers — which includes finance and product, not just engineering. A task-type cost table that a product manager can sort is worth more than a trace viewer only engineers open.
The pricing side: AI agent cost per task model
| Attribution dimension | Question it answers | Typical surprise it surfaces |
|---|---|---|
| Per agent | Which agent is the fleet cost leader? | The least-used agent is often the most expensive per task |
| Per task type | What does a summarize vs research vs draft task cost? | One workflow costs 10× the others and nobody priced it |
| Per user / tenant | Is any customer unprofitable at current pricing? | A free-tier power user burning paid-tier margin |
| Per model | Is the routing policy actually being followed? | Half the traffic silently fell through to the expensive default |
| Per retry / step | Where inside a task does spend concentrate? | The final "polish" call costs more than the whole draft |
🔁Loop detection is cost control
The single most common cause of runaway agent spend is not traffic — it is loops. An agent that cannot make progress does not stop; it re-plans, retries the same tool, re-reads the same context, and bills you for every lap. Because each individual call is legal and successful, nothing pages. Only the aggregate betrays it, and only if you are watching the aggregate per task.
Detect loops at three levels. Cheapest first: iteration caps. Every agent run gets a hard maximum on model calls and wall-clock time, sized generously for the legitimate task and enforced absolutely. This is the backstop that converts every loop bug from "unbounded spend" to "one expensive task."
Second: no-progress detection. Track a cheap progress signal per step — new information retrieved, state changed, plan items closed. N consecutive steps with no state change is a loop, even if every step is syntactically different. More sophisticated: hash the (tool, arguments) pairs and kill the run on a repeated signature, which catches the classic exact-retry loop immediately.
Third: cost-based tripwires. A task whose cost crosses, say, three times the rolling median for its task type gets flagged for review even if it completes successfully. Successful-but-expensive tasks are tomorrow incident report; loops that eventually succeed still burned the money.
One design note: make loop kills graceful. A killed run should return a partial result and a clear "stopped: no progress" status to the user, not an opaque error. Users retry opaque errors — which starts the meter again.
| Loop pattern | Symptom in the data | Cheapest effective control |
|---|---|---|
| Exact retry (same tool, same args) | Repeated identical call signatures | Signature hashing with a repeat limit |
| Thrashing (re-plan every step) | High call count, no state change | No-progress detector over N steps |
| Context ballooning (re-read everything each step) | Input tokens growing linearly per step | Context budget per step; summarization |
| Multi-agent delegation ping-pong | Task cost = N agents × expected cost | Delegation depth cap + aggregate task budget |
| User-triggered retries after opaque failure | Task cost multiplied by retry count | Graceful partial results on kill |
🛑Budget circuit breakers that actually stop the bleeding
Alerts tell you money is burning; circuit breakers stop the burn. The distinction matters because the response time of a human reading an alert at 2 a.m. is measured in hours, and a loop bug bills at the speed of the API.
Build breakers at three scopes, each with a defined action. Per-run: a maximum cost or step count per task, enforced by the agent runtime — breach it and the run halts gracefully. Per-identity: a daily or hourly budget per user, tenant, or API key, enforced by your gateway or wrapper — breach it and the identity gets queued, throttled, or a paywall message depending on your product. Per-system: a global daily budget per agent or per environment, breach of which disables the feature flag and pages someone.
Two implementation rules. First, enforce at the client, not the invoice: estimate cost before the call and check it against remaining budget, because provider-side spend limits (where offered) lag and are too coarse for per-identity control. Second, fail closed for the expensive paths. If the budget service is down, the default for a runaway-capable feature should be deny-or-degrade, not allow-everything. Degrade can mean routing to a cheaper model, which is often the right behavior anyway.
Test your breakers. A circuit breaker that has never fired in staging is a hypothesis. Chaos-test it: point an agent at a stubbed tool that always returns "try again," and verify the run dies at the cap, the task is attributed, and the alert fires. That thirty-minute exercise is the difference between a control and a hope.
Per-run cap
Max steps, max cost, max wall-clock per task. Enforced by the runtime, graceful on breach. Converts loop bugs from unbounded to bounded.
Per-identity budget
Hourly or daily allowance per user or tenant. Enforced at the gateway before the call. Protects you from power users and abusive clients.
Per-system breaker
Daily fleet budget per agent tied to a feature flag. Breach disables the feature and pages. This is the invoice-shock killer.
Fail closed on the expensive path
Budget service unreachable means degrade to a cheap model or deny, never unrestricted spend. Write this down before the incident, not during.
🗄️Cutting the baseline, part one: caching wins
Detection stops disasters; the baseline is cut by engineering. Two levers do most of the work — caching and routing — and both compound with attribution: you cannot verify a cache win without per-task cost data.
Caching comes in three useful flavors. Provider prompt caching (offered by the major providers, with discounted rates on cached input tokens — verify current terms before budgeting on it) rewards stable prompt prefixes: put the system prompt, tool definitions, and long static documents first and keep them byte-identical across calls. Exact-match response caching deduplicates identical requests, which happens more than you expect in retry-heavy frameworks. Semantic caching stores responses keyed by embedding similarity and answers near-duplicate questions for free; use it for read-heavy, low-stakes paths and never for anything time-sensitive or user-specific without care.
The unglamorous detail that decides whether caching pays: prefix stability. A single dynamic timestamp or reordered tool list near the top of the prompt invalidates the cache for everything below it. Audit your prompt assembly for churn before blaming the cache hit rate.
✂️Cutting the baseline, part two: model routing
Model routing is the bigger structural win. Most agent workloads are a minority of hard steps and a majority of mechanical ones — extraction, formatting, classification, short summaries. A routing policy that sends the mechanical steps to a small, cheap model and reserves the frontier model for planning and synthesis routinely cuts fleet cost by half or more without a measurable quality drop, but only if routing is measured per task type rather than tuned by vibes.
A second, quieter lever: context discipline. The dominant cost in long agent runs is input tokens, and input tokens are your own code re-sending the conversation. Trimming tool outputs, summarizing completed subtasks, and capping retrieval results are cost decisions that look like prompt engineering. Attribute them and they become legible: you can point at the step where context trimming saved more than the model swap did.
The table below summarizes the full baseline-cutting toolbox, caching included, because the choice between levers is workload-specific and the attribution data is what chooses for you.
Self-hosted vs API models: the cost trade
| Lever | Mechanism | Where it wins | Watch out for |
|---|---|---|---|
| Prompt caching | Provider discounts repeated input prefixes | Long stable system prompts and documents | Cache misses from trivial prompt churn |
| Exact-match cache | Deduplicate identical requests | Retry-heavy frameworks, idempotent reads | Stale responses on non-idempotent tools |
| Semantic cache | Answer near-duplicates from embeddings | FAQ-style, read-heavy agent paths | Wrong answers on time-sensitive queries |
| Model routing | Small models for mechanical steps | Mixed-difficulty workloads (most of them) | Routing by vibes; measure per task type |
| Context discipline | Trim, summarize, cap retrieval | Long-running multi-step tasks | Summaries that drop decision-critical detail |
📟Alerting and the weekly cost review
Cost alerts should fire on shape, not just totals. A fleet that doubles its spend because traffic doubled is healthy; a fleet whose cost per task doubled at flat traffic has a bug. Alert on cost per task type against its rolling baseline, on unattributed spend as a share of total (it should be near zero), and on breaker events — a circuit breaker firing is by definition an anomaly worth a human.
Keep the alert count low. Three good alerts beat fifteen noisy ones, because noisy cost alerts get muted and muted alerts are how invoice shock survives a monitoring program. Route spend alerts to the channel that owns the feature flag, not a general observability list — the response to a cost alert is almost always "turn it down or off," and the person reading it needs the authority to do that.
Then run the slow loop deliberately: a weekly thirty-minute review of the attribution table by one engineer and whoever owns the product P&L. Sort by cost per task type, look at the top movers, and ask one question per row: is this spend buying something the product values? That meeting is where "the polish step costs more than the draft" turns into a routing change, and it is where pricing and packaging decisions get real data.
The mature end state is boring: cost per task is a dashboard next to latency and error rate, every task is attributed, breakers have fired in staging and once in prod, and the provider invoice has never surprised anyone twice.
Alert on cost per task against baseline, not on raw spend. Traffic-driven spend growth is a business event; per-task spend growth at flat traffic is an engineering incident.