Skip to main content
Architecture & Engineering

Observability

Observability is the ability to explain a system's internal state from its outputs — structured logs, metrics and traces — without deploying new code to find out. In distributed systems it is the difference between "the site is slow" and "the checkout service's connection pool is exhausted", reached in minutes instead of days.

The three classic signals have distinct economics. Metrics are cheap aggregates — request rates, error rates, latencies — good for dashboards and alerts, poor at detail. Logs carry detail but are expensive at volume and useless without structure. Traces record the path of a single request across services and are what made microservices debuggable at all. Correlation IDs tying the three together matter more than any individual signal.

Monitoring and observability answer different questions. Monitoring checks known failure modes against dashboards — the known unknowns. Observability handles the unknown unknowns: the novel incident nobody wrote a check for. The enabling property is high-cardinality data — being able to break down latency by user, endpoint or deploy version after the fact — which lets engineers ask questions of production that nobody anticipated when the instrumentation was written.

OpenTelemetry standardised the instrumentation layer, so code is instrumented once and the backend — Jaeger, Grafana, Datadog, Honeycomb — can change without re-instrumenting. Practice matters more than platform: alerts keyed to SLO burn rate rather than symptom spam, sampling and retention tiers that keep costs proportional to value, and tracing on the critical request paths first. Observability budgeted as a feature ships; observability treated as an afterthought gets bought after the first multi-day incident.

Cardinality is the discipline that keeps the whole thing affordable. A label for the endpoint is useful; a label for the user ID multiplied across every metric is an invoice and a query slowdown at once. The working rule is that high-cardinality detail belongs in traces and logs while metrics stay coarse — and teams that ignore the rule learn it from the monitoring bill before they learn it from a book.

The payoff shows up in incident reviews. With traces on the critical paths, "what happened" is a query rather than a debate, and the debugging conversation moves from theories to evidence within the first ten minutes. Exemplars — links from a metric spike straight into representative traces — shorten the jump from "something is wrong" to "this request, this service, this dependency", which is where mean time to resolution is actually won. That speed compounds: every incident that resolves quickly is also one that teaches the team something specific.

Codazz builds this in production — Cloud & DevOps.

FAQ

Observability
FAQ.

Common questions about observability.

Ask Us Anything

Monitoring answers questions you anticipated — dashboards and alerts for known failure modes. Observability lets you answer questions you did not anticipate, by capturing high-cardinality traces, logs and metrics that can be sliced new ways after an incident begins. You monitor for the failures you can imagine; you need observability for the ones you cannot.

Distributed tracing on the critical request paths, plus RED metrics — rate, errors, duration — per service. That combination answers most "what broke and where" questions in a microservices system. Structured logs come next, then SLOs once the data exists to set them honestly. Instrument with OpenTelemetry from the start so the backend choice stays reversible.

There is no universal percentage, but the honest check is proportionality: telemetry spend should track the downtime and debugging cost it prevents, and it should be reviewed like any other infrastructure line. Bills explode through unplanned cardinality, debug-level logging left on, and full-fidelity retention kept forever. Sampling, tiered retention and a regular review of what is actually queried keep the system honest.