📡 AI Observability

AI Observability: AI observability is a hospital's vital-signs monitor, not a one-time checkup — and the mechanism matches one-to-one, not loosely: a checkup catches a problem AT

AI observability is a hospital's vital-signs monitor, not a one-time checkup — and the mechanism matches one-to-one, not loosely: a checkup catches a problem AT THAT MOMENT, in the room, while a vital-signs monitor watches continuously and alerts BEFORE a crisis becomes visible to the naked eye. Here is the question worth sitting on: you already have an eval set that passed with green checkmarks the day this feature shipped — why isn't that enough? Why would you need to keep watching afterward? Because an eval run is a snapshot exactly like that checkup, and nothing about a passing snapshot on day one tells you anything about a retrieval setting silently lowered during an unrelated deploy on day five — the exact "AI in Production" tab's evals catch regressions when you RUN them, but running them once at ship time only proves the system was healthy in that one moment, not that it stays healthy. Java comparison: this is the precise difference between a unit test suite that runs once at CI time and an APM tool (Datadog, New Relic) watching a running service continuously — green tests before deploy say nothing about a slow memory leak or a degrading dependency that only shows up hours or days into production, and a silently-lowered retrieval top_k is the AI-pipeline equivalent of that memory leak. The QA stake: a hallucination rate creeping from 3% up to 11% over a week is invisible to anyone glancing at the app, because every individual answer still sounds fluent and confident — observability is the only mechanism that catches this specific kind of drift before it becomes a flood of support tickets or a real incident, which is exactly the spike you are about to investigate in the dashboard below.

What a Dashboard Actually Watches

A real AI observability setup tracks three distinct kinds of signal, and each answers a different question. Hallucination rate (or grounding/faithfulness score, from a continuously-running eval sample) answers "is quality degrading over time?" — the same rubric-based scoring from the Judge Playground and RAG Lab, just run repeatedly in production instead of once, offline. Latency and token-cost distribution answers "did a change make calls slower or more expensive even if the output still looks fine?" — a completely separate axis from correctness. Trace-level detail — the exact retrieval step, prompt, and token breakdown of ONE specific call — answers the question an aggregate number never can: not "that something changed" but "which stage of the pipeline changed and why".

Below, the hallucination-rate trend crosses its threshold on the final day — the aggregate metric tells you something is wrong, but not what. Press "Investigate the Spike" and work through the trace one stage at a time: prompt, token count, retrieval, model version, latency. Four of the five will report "looks normal" — exactly the trap of the "Deterministic vs Stochastic" tab's fluent-but-wrong answers, except here it's a metric that looks fine, not an answer. Only checking each stage individually isolates the one that silently regressed.

🎬 What Does a Dashboard Actually Watch?

Latency (response time)

Output Quality Score

In production, EVERY AI request drops into a dashboard as a record — seeing one request isn't enough, the TREND matters.

Latency gets tracked: if a normally-800ms response suddenly jumps to 4 seconds, that's a warning signal.

Token usage gets tracked: if a user group suddenly starts consuming 10x more tokens, it's either abuse or a bug.

Output quality score gets tracked: an automated eval scores every response — if the average score drops over time, it could be a MODEL UPDATE or prompt degradation.

Final — these three metrics together trigger an ANOMALY ALERT. The dashboard doesn't answer "is it working", it answers "HOW WELL and HOW EXPENSIVELY is it working".

Step by Step: Isolating an Anomaly in a Trace

Aggregate metric signals a problem