🌫️Why traditional APM is not enough
Classic application monitoring assumes failures are legible: a 500, an exception, a latency spike. Language model systems fail illegibly. The request returns 200 in 900 milliseconds, and the answer is confidently wrong. The model provider has an outage and your retry logic silently routes to a fallback model with different behavior. A prompt change ships on Tuesday and quality decays by Friday with no error anywhere in the pipeline. None of these events exist in a traditional dashboard.
LLM observability exists to close that gap, and it does so by treating three things as first-class telemetry that APM never had: the content of calls (prompts, completions, retrieved documents), the economics of calls (tokens, model version, cache hits, dollars), and the quality of calls (evaluator scores, user feedback, task success). The glossary page defines the term; this article is the build guide.
One mental model is worth adopting early: an LLM application is a data pipeline where the transform is probabilistic. You would never run a data pipeline without row counts, schema checks, and spot audits of the output. LLM observability is the same discipline, with the audit done partly by machines.
🔭What to trace: spans per LLM call and tool call
The trace is the spine of the whole stack. Model it as a tree: one root span per user request or agent run, with child spans for every LLM call, every tool call, every retrieval step, and every guardrail check. For a multi-step agent, the difference between "the run took 40 seconds" and a span tree showing that 31 of those seconds were one retried tool call is the difference between guessing and knowing.
Each span needs a standard payload. The OpenTelemetry GenAI semantic conventions have emerged as the shared vocabulary — attributes for the model name, token usage, temperature, and finish reasons — and instrumenting against them keeps you portable across backends. Beyond the standard fields, capture your own identifiers on every span: application version, prompt template version, feature name, tenant, and a session or conversation ID so multi-turn behavior can be reconstructed.
The contentious decision is content capture: whether to store full prompts and completions with each span. You want them — debugging a quality regression without seeing the actual text is archaeology. But prompts carry user data and sometimes PII, which makes content capture a privacy decision, not just an engineering one. The workable pattern is capture-by-default with redaction at the instrumentation edge, plus a strict-retention mode where content is hashed or dropped and only metadata and metrics are stored. Decide this deliberately per environment; do not let it be decided by a default.
Tool calls deserve the same rigor as model calls: record the tool name, the arguments (redacted), latency, success, and the error class on failure. In agentic systems the tool spans are where the reliability problems live, and they are exactly what model-centric dashboards tend to undercount.
| Span type | Must-capture fields | Common mistake |
|---|---|---|
| LLM call | Model and version, token usage, latency, finish reason, prompt version, cost | Logging only the final answer and losing the retry chain |
| Tool call | Tool name, redacted arguments, latency, success or error class | Recording the call but not the arguments that explain it |
| Retrieval | Query, document IDs returned, scores, embedding model version | No link between the answer span and the documents it saw |
| Guardrail check | Check name, verdict, latency, what was redacted or blocked | Silently swallowing blocks so incidents look like good traffic |
| Agent run (root) | Session ID, user or tenant, feature, total tokens, total cost, outcome | No terminal outcome field, so success rate is unmeasurable |
🧾Token and cost accounting per trace
Cost in LLM systems is per-request and variable, which makes it an observability concern rather than a monthly finance surprise. Every span should carry input tokens, output tokens, and — increasingly important — cached tokens, since cached input tokens are priced differently with most providers. Multiply by the price table for the exact model version, and store the computed dollar cost on the span. Price tables change; store the computed value, not just the inputs to it.
The dimensions are where the value is. Aggregate cost by feature, by tenant, by user tier, and by model route. This is how you find that one feature accounts for half the bill, that a single enterprise tenant is unprofitable at its current price, or that a fallback route you added for resilience is serving eight percent of traffic at triple the cost. None of that is visible in the provider invoice.
Two traps to avoid. First, hidden token consumers: guardrail prompts, judge calls, and summarization steps all spend tokens that never appear in the user-visible exchange — instrument them as spans or your accounting will be systematically optimistic. Second, retries: a request that called the model three times cost three calls, and the trace should say so plainly.
Cost per successful task — not cost per request — is the number to watch. A cheaper model that needs two attempts and a judge pass can cost more per solved problem than the expensive model that gets it right once.
📡Quality signals on live traffic
Latency and cost are measurable with counters. Quality is not, and this is where LLM observability diverges hardest from APM. Production quality measurement rests on three signal families, and a serious stack runs all three because each has a distinct blind spot.
Explicit user signals are the cheapest and the most honest: thumbs, accepted-versus-edited suggestions, regenerate clicks, abandoned sessions, ticket resolution. Their weakness is coverage and bias — a small, unrepresentative fraction of users votes. Implicit task-success signals fill the gap where a ground truth exists: did the code compile, did the booking complete, did the user return with the same question within a day.
Reference-free evaluators — LLM-as-judge scoring of sampled production traces against rubrics for groundedness, relevance, or policy compliance — give you coverage at the cost of judge error and judge cost. The discipline that makes them trustworthy: calibrate the judge against human-labeled samples from your own traffic before trusting its trend line, version the judge prompt like code, and alert on changes in the judge score distribution rather than individual scores.
The fourth signal is drift: shifts in the distribution of inputs (new user populations, seasonal topics) and outputs (length, refusal rate, language mix) that precede quality complaints. Distribution monitors on embeddings and simple statistics catch the slow failures that evaluators tuned for point-in-time quality miss.
| Signal family | Coverage | Cost | Blind spot |
|---|---|---|---|
| Explicit feedback (thumbs, edits) | Low — a minority of users | Near zero | Selection bias; unhappy users vote more |
| Implicit task success | Medium — where ground truth exists | Low | Many tasks have no observable success marker |
| LLM-as-judge on samples | High — any sampled trace | Moderate — a judge inference bill | Judge error and rubric blind spots |
| Distribution drift monitors | High | Low | Says something changed, not whether it is worse |
🚨Alerting thresholds that survive contact with reality
LLM alerting fails in two directions: paging on single bad outputs (alert fatigue within a week) or aggregating so loosely that real regressions hide (the dashboard is green while the feature is broken). The fix is to alert like an SLO system, on windows and rates, with quality metrics promoted to the same status as error rates.
The core set: provider error rate and p95 latency per model route, cost per request or per task with a delta alert against a rolling baseline, refusal rate, judge score rolling averages per feature, and guardrail block rates. Each gets a threshold and a window, and the window is sized to your traffic — a feature with fifty requests a day cannot support the same alerting math as one with fifty thousand.
Two rules learned the hard way. Segment alerts by feature and model route before setting thresholds; a global average hides a broken feature behind healthy volume. And page on rate-of-change as well as level: a judge score sliding half a point over three days is a more reliable regression signal than any single-day threshold, because prompts and models drift rather than break.
| Alert | Signal | Sensible starting shape |
|---|---|---|
| Provider errors | Error rate per model route | Page above a low single-digit percent over a short window |
| Latency | p95 per route | Page at 1.5 to 2 times the rolling baseline |
| Cost | Dollars per successful task | Ticket on a sustained delta versus baseline; page only on step changes |
| Quality | Rolling judge score per feature | Ticket on multi-day drift; page on sharp drops |
| Refusals | Refusal rate per feature | Page on spikes — usually a prompt or model change |
| Guardrail blocks | Block rate per check | Page on spikes (attack or broken filter); ticket on slow creep |
🗄️Sampling strategies and data retention
Full-fidelity tracing of every request is the ideal and frequently the wrong default at scale. Storage cost, judge cost, and privacy exposure all scale with volume, so mature stacks sample deliberately. The standard pattern is tail-biased sampling: keep every trace with an error, a guardrail flag, negative feedback, or a low judge score, and sample the healthy remainder at one to ten percent. Errors are rare and informative; healthy traffic is redundant.
A refinement worth the effort is stratified sampling across features and tenants, so that a low-volume feature is not invisible in your dashboards just because it contributes few traces. Per-feature sampling rates are an operational knob you will actually turn.
Retention is the privacy half of the same decision. Trace content contains user data, so retention windows, regional storage, and access controls apply to your observability backend exactly as they do to your primary database — a point teams routinely discover during a compliance review rather than before one. Self-hosted observability, or a vendor arrangement with explicit data terms, stops being a preference and becomes a requirement in regulated contexts. As a working pattern: short retention for full content, long retention for redacted metadata and aggregates.
🧰The tooling landscape, as of writing
The market has settled into recognizable categories — verify current capabilities before committing, because feature sets converge and diverge quickly. LangSmith is the tracing platform from the LangChain ecosystem, with strong evaluation workflows and both hosted and enterprise self-hosted options; it is framework-agnostic in practice, though smoothest with LangChain and LangGraph. Langfuse is the leading open-source option, self-hostable, with tracing, prompt management, and scoring; teams with data-residency constraints shortlist it first. Helicone takes a gateway approach, sitting as a proxy in front of model providers, which makes adoption nearly free and makes it strong on cost and caching analytics.
Arize Phoenix is open source with an evaluation and experimentation emphasis, built on the OpenInference conventions. Braintrust positions around evals-first development with production observability attached. Datadog LLM Observability matters for organizations already standardized on Datadog, where the integration into existing alerting and incident workflows outweighs best-of-breed features. For vendor-neutral instrumentation, the OpenTelemetry GenAI semantic conventions — with instrumentations like OpenLLMetry and OpenInference — let you emit once and route to multiple backends.
Selection advice in one paragraph: if data residency is a constraint, start with Langfuse or Phoenix self-hosted. If you live in the LangChain ecosystem, LangSmith is the path of least resistance. If you want cost and usage analytics this afternoon, a gateway like Helicone is the fastest instrument you will ever install. Whatever you choose, instrument with OpenTelemetry-compatible attributes so switching costs stay low — the layer you will keep is your own span schema.
How to evaluate AI agents before productionWhat production AI actually costs to run