Without evaluation, teams cannot tell whether a change improved the system or broke an edge case, and they will not notice when a provider updates the underlying model and shifts behaviour beneath them. The predictable end state is a team too afraid to change anything — a worse position than having a few failing tests.
For tasks with no single correct answer, evaluation combines deterministic checks on the objective parts — did it call the right tool, was the output well-formed, did it stay within policy — with model-graded scoring for the subjective remainder. The critical step is calibrating that grader against human ratings, because an uncalibrated judge produces confident numbers that mean nothing.
Codazz builds this in production — Evaluation & Observability.