How to Evaluate AI Agents Before Production (Evals Guide)

🎭Why demo-driven shipping fails

Every failed agent deployment we have been called in to rescue started the same way: someone built a promising prototype, ran it against ten hand-picked examples in a meeting, and the room decided it worked. Demos are not evidence. They are a sample of size ten, selected by the person with the most incentive to pick easy cases, evaluated by an audience watching for plausibility rather than correctness.

The core problem is that agent behavior is a distribution, not a function. A traditional feature either works or throws; an agent produces a spread of outcomes across inputs, and the spread shifts every time you change the prompt, swap the model, upgrade a tool, or the underlying model provider quietly updates the weights. A demo measures one point in that distribution on one day. Production exposes the whole distribution, every day, on inputs nobody curated.

Agents compound the problem because errors multiply across steps. If each step in a five-step workflow is right 90 percent of the time, end-to-end success is roughly 59 percent. The demo showed the 90 percent. Your users live in the 59. Evaluation infrastructure exists to measure the end-to-end number honestly, track it over time, and tell you when a change made it worse before your customers do.

The mental shift is treating evals as a product feature, not a QA afterthought. Teams that ship reliable agents spend a visible fraction of engineering effort — in our experience 20 to 30 percent of the agent budget — on evaluation and observability. Teams that skip it spend the same money later, on incident response and churn.

🥇Golden datasets: the foundation everything sits on

A golden dataset is a curated set of representative inputs with verified expected outcomes, versioned like code and treated as the ground truth for every evaluation run. Without one, every eval conversation degenerates into opinion. With one, "did the change help" becomes a measurable question.

Build it from real tasks, not synthetic ones. Pull from support tickets, sales calls, workflow logs — the actual distribution the agent will face, including the ugly cases: ambiguous requests, missing data, hostile phrasing, inputs that should be refused. A dataset of only clean, well-formed requests measures a product your users will never see. Aim for 50 to 200 cases to start; below 50 the variance swamps the signal, and past a few hundred you get diminishing returns for most single-workflow agents.

Each case needs three things: the input, the expected outcome (or at least the rubric that defines a good outcome), and tags that slice the data — task type, difficulty, edge case category. The tags matter more than teams expect, because "overall score dropped 3 points" is useless but "score on multi-step refund cases dropped 20 points" is actionable.

Maintenance is the part everyone underfunds. Golden datasets rot: the product changes, user behavior drifts, new failure modes appear in production. Assign an owner, review the dataset monthly, and feed confirmed production failures back in as new cases. The dataset is a living asset; a stale one gives you confident measurements of a system that no longer exists.

One practical note: keep the golden dataset out of the prompt and out of any fine-tuning data. Contamination turns your test set into training data and every subsequent score into fiction.

🎯Task-completion metrics: measure outcomes, not vibes

The metric that matters for an agent is whether it completed the task, and almost nothing else. Tone, fluency, and confidence are decoys — language models are maximally fluent when they are maximally wrong. Define task completion as a verifiable end state: the ticket is resolved, the order is cancelled in the system of record, the code change passes its tests, the report contains the correct figures.

Wherever possible, verify outcomes programmatically rather than judging text. If the agent books meetings, check the calendar API. If it updates CRM records, diff the database. State-based verification is deterministic, cheap, and cannot be sweet-talked. Reserve text judgment for the tasks where no machine-checkable end state exists, which is fewer than most teams assume.

Track a small set of metrics, not a dashboard of thirty. The working set for most agents: end-to-end task success rate, per-step success rate (so you can find which step breaks), tool-call accuracy (right tool, right arguments), refusal correctness (did it decline what it should decline, and not decline what it should do), cost per completed task, and latency. Everything else is secondary.

Report metrics by slice, never only in aggregate. An agent at 85 percent overall might be at 97 percent on simple cases and 40 percent on the hard tier — and the hard tier is exactly where your most valuable tasks live. Aggregate scores are how teams ship regressions with a straight face.

⚖️LLM-as-judge: powerful, biased, and still worth it

Some agent outputs genuinely have no programmatic check — a drafted email, a research summary, a negotiated reply. For those, LLM-as-judge is the practical answer: a strong model scores the output against a rubric, at scale, for pennies per case. Frameworks like Braintrust, LangSmith, Promptfoo, and OpenAI Evals-style harnesses all implement some version of this pattern, and it has become the default for open-ended evaluation.

But the judge is a model, with model failure modes. It prefers confident, well-structured, longer answers regardless of correctness. It agrees with whatever framing the rubric implies. It grades its own model family generously. And it cannot verify facts — a judge will happily award full marks to a fluent summary citing figures that do not exist. As of writing, the research literature on judge bias is consistent on these points; verify current findings before designing a high-stakes pipeline around any single judge.

The mitigations are well understood. Use pairwise comparison (is output A better than output B) instead of absolute scores where possible — humans calibrate judges more easily on comparisons. Keep rubrics specific and binary per criterion rather than a single 1-to-5 scale. Swap judge models periodically and measure inter-judge agreement. And never let the judge be the same model, or model family, as the agent under test.

The non-negotiable step is human calibration. Sample judged outputs regularly — 5 to 10 percent is a workable range — and have a human score them blind. Track agreement between judge and human. When agreement drifts below your threshold, the judge prompt or the judge model needs work. An uncalibrated judge is not an evaluation; it is a second opinion from an employee you have never checked.

🔄Regression suites: evals wired into the pipeline

A regression suite is the golden dataset run automatically, on every change that could move behavior: prompt edits, model version bumps, tool schema changes, framework upgrades, retrieval index rebuilds. The output is a diff — this change moved success on slice X from 88 to 79 percent — that blocks the deploy until a human signs off.

The architecture is simple. Evals live in the repo as code. CI runs the suite on pull requests that touch anything in the agent path, using seeded or recorded model calls where determinism matters and live calls where it does not. Results are stored per run so trends are visible over weeks, not just per commit. Treat a failing eval suite exactly like a failing test suite: the merge does not happen.

Two traps to design around from the start. First, cost: a full suite against a frontier model on every commit gets expensive, so tier it — a fast smoke subset on every commit, the full suite nightly and before releases. Second, flakiness: model outputs vary run to run, so set pass thresholds on aggregates, run critical cases multiple times, and quarantine genuinely non-deterministic checks rather than letting the team learn to ignore red builds.

The regression suite is also your negotiation tool with model providers. When a provider updates a model and your scores drop, a versioned eval history turns a frustrating support ticket into a documented before-and-after. More than once, that artifact is what justified pinning a model version or accelerating a migration.

📡Production monitoring: the eval that never stops

Pre-production evals tell you how the agent performs on tasks you anticipated. Production monitoring tells you what it is actually doing. Both are necessary because the input distribution in production always diverges from your dataset — users find phrasings, tasks, and failure modes nobody predicted. This is the discipline the industry has settled on calling agent observability: full traces of every run, sampled quality scoring, and drift detection on live traffic.

The foundation is tracing. Every agent run should produce a structured trace — the inputs, each model call with its prompt and response, every tool call with arguments and results, latency, and token usage — stored and queryable. LangSmith, Langfuse, Arize Phoenix, and Braintrust all provide this layer, and OpenTelemetry-based instrumentation is becoming the common substrate. Without traces, debugging a production failure is archaeology; with them, it is reading.

On top of traces, run continuous quality sampling. Score a percentage of live runs with the same judges and rubrics used pre-production, and alert when the live score distribution drifts from the eval baseline. Add hard guardrail metrics too: tool-error rates, refusal rates, cost per task, and escalation rates. A sudden move in any of them is an incident even when quality scores look fine.

Close the loop back into the dataset. Confirmed production failures — found through sampling, user reports, or guardrail alerts — become new golden cases. This is the flywheel that makes the whole system improve: production finds the gaps, the dataset absorbs them, the regression suite makes them permanent.

How we build agent observability into production systems→

🗓️The eval cadence: who runs what, and when

Evaluation infrastructure decays without an operating rhythm. The cadence below is what we have seen work on production agents — adjust the frequencies to your change rate, but keep the structure: something runs continuously, something runs on every change, a human reviews on a fixed schedule, and the dataset gets fed from production.

The weekly review is the heartbeat. One engineer and whoever owns the product outcome sit down with three artifacts: the regression trend line, the live-monitoring sample scores, and the new failures harvested from production. Decisions come out of that meeting — accept a regression for a capability gain, promote a production failure into the dataset, re-calibrate a judge. Evals without this meeting become a dashboard nobody opens.

Assign a named owner. On teams where evals are everyone’s job they are no one’s job, and the first sign is a golden dataset that has not changed in three months.

💰What evaluation infrastructure costs

Honest labelled ranges, because the real number depends on how many workflows the agent covers and how much of the stack you build versus rent. A minimal but real setup — golden dataset, programmatic checks, CI smoke suite, basic tracing — typically runs $15,000 to $40,000 of engineering effort to stand up alongside the agent build. A fuller program with LLM-as-judge calibration, tiered regression suites, and production sampling dashboards lands in the $40,000 to $100,000 range as part of a serious agent build.

Recurring costs are smaller but permanent. Eval platform tooling (LangSmith, Braintrust, Langfuse Cloud, and similar) runs from free tiers to a few hundred dollars per month at team scale, usage-priced. Judge model calls and eval compute are usage costs that scale with dataset size and run frequency — usually tens to low hundreds of dollars per month for a single-workflow agent, more when the full suite runs nightly against frontier models. The largest recurring cost is human: the weekly review and monthly dataset work, roughly one to two engineer-days per month.

Compare that against the alternative. One production incident where an agent quietly did the wrong thing for two weeks — wrong refunds, mangled CRM records, fabricated figures in customer-facing output — routinely costs more than the entire eval program, before you count the trust damage. Evaluation is not the expensive option; it is the cheap one that feels optional until the day it was not.

🤝When to bring in a team

If your team has shipped and operated production LLM systems before, this guide is enough to build the eval layer in-house — the patterns are established and the tooling is mature. The case for outside help is when this is the first production agent, when the cost of a silent failure is high (money movement, customer data, regulated output), or when the team is already behind on the build itself.

What you are buying is scar tissue: someone who has already watched a judge go uncalibrated, a dataset rot, and a regression ship because the smoke suite was too small, and designs around those failures from day one. Ask any prospective partner to show you their eval artifacts — a real golden dataset structure, a regression report, a monitoring dashboard — before you look at a single demo. Teams that run evals can show them instantly; teams that demo cannot.

We build agents with the evaluation and observability layer included from the first sprint, because retrofitting it after an incident is the most expensive way to get it. Codazz has delivered 500+ projects since 2018 with 200+ engineers across Edmonton and Chandigarh, and the agent observability practice exists precisely because this layer is where agent projects live or die.

Agent observability services→AI agent development at Codazz→