Eval-Driven Agent Development: A Working Method

⚡The method in one paragraph

Eval-driven development (EDD) applies the test-driven loop to systems whose correctness cannot be asserted with equality checks. Before changing a prompt, model, or tool, you write or extend the eval that defines "correct" for the behavior you are about to touch. You run the suite. You make the change. You run the suite again. If the target eval improves and nothing else regresses beyond its tolerance, the change ships. Otherwise it does not, no matter how good the demo looked.

That last clause is the entire method. Agents fail by vibes: a prompt tweak fixes the case you were looking at and quietly degrades five you were not. Without a gate, every improvement is a random walk. With a gate, changes accumulate. The difference between teams whose agents get steadily better and teams whose agents oscillate is almost never model choice — it is whether changes were gated on a fixed, trusted yardstick.

Our pre-production evaluation guide covers how to assess an agent before launch: capability probes, red-teaming, readiness sign-off. This article picks up from there — the day-two reality of a living agent whose prompts, models, and tools change every week. Eval-driven development is how you keep that change rate without losing your mind or your users.

The pre-production companion: how to evaluate AI agents→Agent observability, built and operated→

📝Write the eval before the prompt

The discipline is borrowed from TDD and adapted for stochastic systems. In TDD you write a failing test, then make it pass. In EDD you write a failing eval — a case your agent currently gets wrong — then change the prompt, model, or tool surface until it passes, under the constraint that the rest of the suite stays green. The eval is the specification; the prompt is an implementation detail.

This inverts the usual workflow, which is: tweak prompt, eyeball two outputs in a playground, ship. Eyeballing has three known failure modes: you check the case you were fixing and nothing else, you anchor on whatever the model produced last, and you cannot distinguish "fixed" from "different." A written eval eliminates all three, and it compounds — every bug you have ever fixed stays fixed, because its eval is still in the suite.

The practical unit is small. An eval case is an input, any setup the agent needs, and a checkable assertion about the outcome — not necessarily an exact string match, more on graders below. When a user reports a failure in production, the first artifact of the fix is a failing eval case reproducing it. The prompt change is the second artifact. Teams that adopt just this one rule — no fix without a reproducing case — see repeat regressions roughly disappear, because the suite grows teeth with every incident.

🏆Golden set curation: the asset that appreciates

The golden set is the curated collection of cases your gates run against, and it is the most valuable artifact in an agent codebase — more durable than any prompt, any model version, any framework. Treat it accordingly: versioned in the repo, reviewed like code, owned by a named person.

Sources, in descending order of value. Production failures: every incident, every user complaint, every thumbs-down becomes a case — these are gold because they are real inputs where the stakes are proven. Production successes: a sample of real tasks the agent handled well, so the suite measures regression on the common path, not just the edge cases. Synthetic cases: authored to probe specific capabilities, constraints, and adversarial inputs — necessary but biased toward what you already thought of, which is why they rank last.

Size honestly: a useful golden set for a focused agent is a few dozen to a few hundred cases, not thousands. Past that, run cost and review burden grow faster than signal, and stale cases accumulate. Every case should earn its place by failing at least once in its life — a case that has never caught anything is decoration. Prune quarterly: delete cases that no longer correspond to behaviors the product has, and de-duplicate near-identical cases that double-count one behavior in your aggregate score.

Label quality matters more than quantity. For each case, record what correct means and why — including the rejected answers and the reason they are wrong. That rationale is what lets a future engineer (or an LLM judge) grade borderline outputs consistently, and it is what survives team turnover. A golden set without rationales decays into a list of sacred outputs nobody dares touch.

🧱The three eval types you actually need

Assertion evals are the cheapest and most trustworthy: deterministic checks on the final artifact or the trace. The refund email cites the correct policy ID. The SQL the agent ran contains no DELETE. The task completed within the step budget. Any check you can express in code, express in code — these are fast, free to re-run, and never disagree with themselves.

Graded evals use a rubric: a human-authored checklist that a grader — human or model — applies to the output. "Acknowledges the constraint," "does not promise a timeline," "includes the caveat when data is missing." Rubric evals are where LLM-as-judge belongs: a strong model applying a specific, decomposed rubric is consistent enough to gate on, provided you have calibrated it. Calibration means a labeled sample where humans and the judge disagree, measured, with the rubric revised until agreement is acceptable. An uncalibrated judge is a random number generator with a confidence score.

Trajectory evals grade how the agent got there, not just where it ended: which tools it called, in what order, at what cost, with what dead ends. Two runs can produce the same correct answer, one in four clean steps and one in forty billable meanders. If you only grade outcomes, you will ship the forty-step version and discover it on the invoice. Trajectory checks — expected tool sets, maximum steps, forbidden sequences — are how efficiency and safety properties get gated like correctness.

A healthy suite is mostly assertion evals, a meaningful slice of calibrated rubric evals for judgment-heavy outputs, and trajectory checks on every workflow where cost or safety depends on the path. If your suite is 90 percent LLM-judged vibes, your gates are noise, and the sections below will not save you.

🚦CI integration and eval gates

Evals that run when someone remembers to run them are documentation. The method only works if the suite runs automatically, on the triggers that matter: any change to a prompt, a model version, a tool definition, or orchestration code. Most teams wire this as a CI job keyed on the paths that contain agent configuration, plus a nightly full run, because per-commit runs on a large suite cost real money and time.

The gate policy is the part teams under-specify. Define it in writing: which evals are blocking (assertion and trajectory checks on core workflows), which are advisory (rubric evals on long-tail outputs), what regression tolerance applies (exactly zero on blocking evals; a small, pre-agreed band on aggregate scores to absorb judge noise), and who can override a gate and how that override is recorded. An undocumented gate is a suggestion, and suggestions lose to deadlines.

Sampling keeps the loop affordable. The full golden set on every prompt change is ideal when the set is a few hundred cases; as it grows, run the blocking subset per commit and the full suite nightly. Never sample randomly per run for blocking decisions — the same change must face the same cases, or pass/fail becomes a coin flip you will learn to game by re-running CI.

Model upgrades deserve a special gate, because they are the change you do not control. When a provider deprecates a model version or you evaluate a new one, run the full suite against old and new on identical cases and diff the results per case, not in aggregate. Aggregate scores hide compensating regressions — five cases improve, five degrade, the average holds, and you ship a different agent.

🔍Regression triage: what to do when the gate goes red

A red eval run is the start of a workflow, not the end of one. The teams that drown are the ones whose response to a failed gate is re-running the suite hoping for green. The workflow that works has four steps, and it is fast precisely because it is boring.

Step one: reproduce and isolate. Re-run the failing cases in isolation with recorded traces. Nondeterminism means some failures flake — but a flake is a finding, not an exemption. A case that fails one run in five is a case your users hit one time in five.

Step two: classify the regression. Genuine behavior break (the change made things worse — revert or fix). Eval rot (the case asserted something the product no longer requires — update or delete the case, with the rationale recorded). Judge drift (the LLM judge changed its mind, not the agent — recalibrate). Rubric ambiguity (the output is fine and the rubric cannot tell — tighten the rubric). These four look identical in a red build and require opposite responses, which is why classification is a step, not a vibe.

Step three: decide and record. Whatever the resolution — revert, fix forward, update the eval — it goes in the case file with a one-line rationale. The suite accumulates an institutional memory of why it says what it says.

Step four: feed back. Every genuine regression that reached the gate without a pre-existing case exposing it means the suite had a hole. Add the case. The gate is the immune system; the triage loop is how it learns.

🔄Team rituals that keep the suite honest

Eval suites decay like test suites, only faster, because the ground truth shifts: products change, policies change, models change underneath you. The rituals below are the maintenance schedule. None are heavy; all are mandatory if you want the suite to mean something in month six.

A per-change norm: no prompt, model, or tool change merges without the eval diff in the PR description — what moved, by how much, on which cases. Reviewers read eval diffs the way they read test results. This single norm kills the "improved the prompt" PR with no evidence, which is the most common genre of agent regression.

A quarterly pruning and calibration review: delete stale cases, re-calibrate judges against fresh human labels, and re-baseline tolerances. Judges drift as models update; a judge calibrated in January is making up its own rubric by June.

Ownership: one named owner for the suite. Not a committee. Committees let red builds stay red; an owner gets paged by them.

🧰Honest tooling notes

The tooling landscape as of writing — verify before committing, this space moves quarterly — splits into three layers, and you need something in each. Evaluation frameworks for defining and running suites (Promptfoo, Braintrust, LangSmith evaluations, and the eval features of Langfuse are the names we see most in real pipelines). Observability platforms for the traces your trajectory evals and failure harvest consume (LangSmith, Langfuse, Arize Phoenix, Helicone). And your own CI glue, because no vendor ships your gate policy.

Buy versus build is more modest than the vendor demos suggest. Buy the trace storage and the UI; building those is a waste of your engineers. Be skeptical of anything that promises evals without authoring effort — auto-generated eval suites measure what is easy to measure, which is rarely what your product depends on. The golden set and the rubrics are yours to write; no tool knows that a refund answer must cite the policy ID.

On LLM-as-judge specifically: it is a legitimate grader for decomposed, specific rubric items, calibrated against human labels, and re-calibrated when the judge model changes. It is not a legitimate grader for "overall quality," and any dashboard showing an uncalibrated quality score is decorative. The failure mode to watch is quiet: a judge model update shifts your scores with zero code changes on your side, and the gate either blocks good work or waves through regressions. Pin judge models and treat judge updates like model upgrades — full diff, per case.

The stack we end up recommending most often is boring: evals as code in the repo, traces in one observability platform, gates in CI, and a spreadsheet-grade weekly review. Fancy tooling helps at the margins. The method is the thing.

Observability for agents in production→Talk through your eval setup with us→