⚡The honest multiplier: 2× to 4× the pilot cost
The gap between a working demo and a production system is not a quality preference — it is a category difference. A pilot proves the model can do the task on curated inputs, under supervision, at trivial volume. Production means the system does the task on whatever arrives, unsupervised, at real volume, with a cost ceiling, an audit trail and an answer for the security team.
Across the engagements we see, the jump lands between 2× and 4× the pilot budget. The low end is an internal tool with forgiving stakes and clean data access. The high end is anything customer-facing, anything that writes to systems of record, and anything in a regulated industry — where the compliance and audit work alone can match the original pilot cost.
Our AI agent PoC-to-production service
| System type | Typical pilot cost | Typical production cost | What dominates the multiplier |
|---|---|---|---|
| Internal assistant (read-only, low stakes) | $15,000 – $40,000 | $40,000 – $90,000 | Eval harness, permission-aware retrieval, adoption work |
| Customer-facing assistant | $25,000 – $50,000 | $80,000 – $180,000 | Guardrails, refusal behaviour, load handling, legal review |
| Agent with write access (refunds, tickets, records) | $30,000 – $60,000 | $120,000 – $300,000+ | Approval gates, enforcement outside the model, audit logging, staged rollout |
⚰️Why pilots die (and it is rarely the model)
Most enterprise AI pilots do not fail — they stall. The demo worked, the steering committee was impressed, and then the project entered the long corridor between "promising" and "deployed" where it quietly ran out of budget and attention. The causes repeat with remarkable consistency, and none of them are fixed by a better model.
How to model AI running cost per task
The pilot was proven on demo data
Curated documents and hand-picked queries hide the parsing failures, permission problems and near-duplicates that real corpora are made of. Production data is the first honest test the system ever faces.
Nobody defined "correct"
Without a scored evaluation set, every change is a coin flip and every regression is discovered by a user. Teams without evals become afraid to touch the system, and a system nobody can improve is a system that gets shelved.
Security review arrived at month three
When InfoSec, legal and procurement meet the project after the pilot instead of before it, their entirely reasonable requirements land as rework. The review itself is not the cost — the surprise is.
No owner with a budget
A pilot has a sponsor; a production system needs an owner — someone accountable for eval scores, incident response and the run cost. Orphaned systems decay, visibly, in front of their users.
The run cost was never modelled
A pilot at fifty queries a day tells you nothing about the invoice at five thousand. Cost per successful task, including retries and escalations, has to be computed before the business case is signed.
📏The evaluation harness: the line item pilots skip
If one workstream defines the pilot-to-production jump, it is this one. An eval harness is a scored set of real inputs with known-good outputs, run automatically every time anything changes — prompt, model, retrieval pipeline, tool definitions. It is the only mechanism that converts "we think it got better" into a number.
Building it is real work. Someone who knows what correct looks like must produce and review several hundred representative cases, including the hostile and weird ones. The harness then needs maintaining as the product and its data evolve — it is permanent infrastructure, not a launch task. Expect it to be 15 to 25 percent of the production budget, and expect every quote that omits it to be describing a larger pilot, not a production system.
Regression safety
Every model upgrade and prompt change gets scored before it ships. Without this, "the provider updated the model" becomes an outage cause rather than a non-event.
A shared definition of quality
The eval set is where business stakeholders and engineers agree, in concrete examples, what the system is for. That agreement is worth as much as the test coverage.
A kill switch for scope creep
New feature requests get a measurable question attached: does the eval score move? It is the difference between a roadmap and a wish list.
🛡️The hardening work, priced by workstream
This is the bulk of the multiplier. Each row below is work a pilot legitimately skips and a production system cannot. The shares are honest ranges from scoping real engagements — they overlap, and they sum to more than the whole because some work serves two rows.
| Workstream | What it involves | Typical share of production budget |
|---|---|---|
| Security & permissions | Scoped tool permissions, enforcement of limits outside the model, injection defences, permission-aware data access, audit logging | 15–30% |
| Observability | Full traces of every run, per-request cost tracking, quality dashboards, alerting on spend and failure-rate anomalies | 10–20% |
| Reliability & scale | Fallback models, retry policy, rate-limit handling, load testing, latency budgets, graceful degradation | 10–20% |
| Integration | Real authentication to systems of record, error handling for their failure modes, data contracts that survive upstream changes | 15–30% |
| Evaluation | Golden set construction, scoring automation, regression gates in CI, ongoing curation | 15–25% |
| Rollout & change management | Staged release, user training, feedback loops, the human escalation path and its staffing | 10–20% |
A useful test for any production quote: find the security, observability and evaluation lines. If they are absent, the quote is for the demo again — just at a bigger number.
💸Do not forget the run cost
The build multiplier has a monthly sibling. Pilots run at volumes where inference is a rounding error; production runs at volumes where it is a budget line. The run cost has three components, and only one of them is tokens.
First, inference: requests per month multiplied by tokens per request, multiplied again by the retry multiplier — real workloads spend meaningfully more than the happy-path estimate because failures happen late, after most tokens are spent. Second, infrastructure and tooling: vector storage, tracing, eval automation, environments. Third, the human layer: whoever reviews escalations, curates the eval set and owns the system. On most production deployments we see, the third line rivals the first — and it is the one absent from vendor business cases.
There is also a fourth line that only appears at production scale: continuous improvement. Model providers deprecate versions, your data drifts, users find new ways to ask things. A production AI system needs a maintenance allocation in the same way any software does — plan 15 to 25 percent of the production build cost per year, reviewed against the eval scores. A system with no improvement budget does not stay still; it quietly gets worse as the world moves around it.
LLM integration cost: API vs fine-tuning vs self-hostingRAG system cost: build vs buy
🗺️How to plan the jump so it actually happens
Scope the pilot as phase one of production
Write the production requirements — security, evals, run-cost model — before the pilot starts. The pilot then answers the questions production depends on, instead of generating a demo that proves nothing transferable.
Bring security and legal in at week one
A one-hour requirements conversation before the pilot saves a quarter of rework after it. Their requirements are stable and predictable; only the timing of the conversation is a choice.
Instrument the pilot like it is production
Log every run, every tool call, every token. The pilot's telemetry is the dataset your production cost model and eval set are built from.
Name the owner and the kill criteria
Decide in advance what result funds the jump and what result ends the project. Pilots without kill criteria do not die cleanly — they decay expensively.
Budget the jump as a separate, honest line
Present the 2–4× multiplier to the steering committee on day one. The projects that reach production are the ones whose sponsors were never surprised by the real number.
✅What production day one should actually look like
It is worth ending the planning with a concrete picture, because "production" is used loosely enough to hide most of the cost. A system has genuinely crossed the line when all of the following are true — and each one maps to a line in the budget that a pilot did not have.
A change can ship safely on a Tuesday
Prompt, model or pipeline changes go through the eval gate and deploy without a meeting. If every change requires heroics, you have a demo with users attached.
The system fails politely
Low-confidence cases escalate to a human with full context; outages degrade to a fallback rather than a blank screen; spend ceilings terminate runaway loops before the invoice does.
Someone can answer "what happened?" in minutes
Every run is traced — inputs, tool calls, tokens, cost, outcome. A wrong answer is a queryable record, not an anecdote from a screenshot in a Slack thread.
The unit economics are on a dashboard
Cost per successful task, success and escalation rates, and the 95th-percentile cost — reviewed by the owner monthly. If nobody is watching these numbers, the system is unsupervised, whatever its uptime.
Security can defend it in an audit
Documented data flows, scoped permissions enforced server-side, retention policies, and an answer to "what stops it from doing X?" that is a control, not a prompt instruction.
If that list sounds like a lot, it is — that is the multiplier, made tangible. The good news is that it is known, bounded work. The teams who reach production are simply the ones who priced it before they started.