Traditional observability stacks were built for deterministic services. If latency goes up or error rates spike, you know something is wrong and you can often trace it to a specific change.
LLM systems break this mental model. The core behaviour is stochastic, heavily data‑dependent, and often mediated by orchestration layers, tools, and memory components. Looking only at logs and metrics is like watching the engine temperature of a car without ever looking at the road.
From telemetry to evidence
For LLM systems, the useful unit of observability is closer to “evidence” than raw telemetry. Each interaction should leave behind artefacts that make it possible to answer simple questions:
- What did the model see and in which context?
- Which tools or retrieval steps did it use, and with what results?
- How would this behaviour score against the evaluations we care about?
Evaluations as first‑class signals
Instead of manually reading samples, we can turn evaluations into structured signals that flow through the same pipelines as metrics:
- Task‑specific automatic checks (safety, hallucination, adherence to constraints).
- Periodic human‑in‑the‑loop review, sampled from real traffic.
- Scenario‑based test suites for changes to prompts, models, or tools.
These signals are what you actually want on dashboards and alerting rules, not just CPU or token counts.
Closing the loop
Observability is only useful if it drives action. In a mature LLM system, evidence and evaluations feed back into:
- Prompt and policy updates.
- Model selection and routing decisions.
- Retrieval and tool reliability improvements.
The long‑term goal is a control loop where behaviour is continuously measured and nudged toward the semantics you care about, rather than manually patched after incidents.
Observability mermaid schema
A simple way to visualise the flow is as an evidence pipeline around the core model: