claudio.cool
CCDV-PLessons

CCDV-P · Observability and Production Monitoring

SLIs, SLOs, alerting and quality drift

⏱ 3 min readDraft

A summarization service has an all-green dashboard: 99.9% availability, stable p95 latency. Yet for two weeks users have said the summaries “got worse”. Infrastructure metrics don’t see quality. In an LLM system, answering doesn’t mean answering well.

SLIs and SLOs for an LLM service

An SLI (service level indicator) is a measure of how the service is doing; an SLO (service level objective) is the target for that SLI. There are three families:

Family Example SLIs
Availability and latency successful request rate, time to first token, total p95 latency
Quality rate of outputs valid against the schema, online evaluation score on a sample, negative feedback rate, human-escalation rate
Cost and usage cost per request or per completed task, tokens per request, cache hit rate

The quality SLO is the most often omitted and the one that matters most. An error budget formalizes how much shortfall you tolerate before freezing changes to stabilize. For latency targets, define which one you measure: time to first token matters in a streaming chat UI; total latency matters in a batch process.

Alerts and dashboards

A good alert tells you something actionable. Useful rules:

  • Alert on error-budget burn or sustained rates, not a one-minute spike.
  • Split errors by type (rate limits, overload, refusals, length cut-offs): one “errors” alert mixes causes with different responses.
  • Include token and cost telemetry in dashboards: tokens per request, cost per tenant, cache rate. A prompt change that doubles output tokens breaks nothing visible but doubles the bill.
  • One dashboard per question (is it healthy? what does it cost? did quality degrade?), not one that mixes everything.

Quality drift and regressions

Quality drift is gradual worsening with no visible failure. The most common sources: changes in user data or queries, changes to the prompt, tools or retrieved content, and model updates. Detecting it requires measuring quality continuously:

  • an online evaluation on a sample of traffic, compared with a baseline;
  • a fixed set of reference cases run on a schedule against production (a canary), to see changes even when traffic varies;
  • output distributions: length, format, refusal rate, share of each category. An abrupt shift in the distribution is a behavioral regression even if no error metric moves.

After a model update, you compare the same metrics before and after, by cohort, and keep the previous version available to go back.

Anomalous usage

Anomalous usage includes token spikes from one tenant, agent loops that never end, and patterns of abuse or repeated attempts to bypass controls. You detect it with thresholds relative to each tenant’s history, and respond with limits and, if needed, blocking.

The usual traps

  • Measuring only availability and latency.
  • Alerting on every individual error until nobody reads them.
  • A quality SLO with no way to measure it.
  • Comparing with last week without accounting for traffic changes.
  • Updating the model with no baseline to compare against.

On the exam

If the panel is green and quality drops, the answer is a quality SLI with online evaluation and reference cases, not more infrastructure metrics. After a model update, per-cohort comparison against the baseline.

Quick question

A practice exam question on this domain. It counts towards your progress.

A document-QA agent's task-success rate, measured on graded samples, fell from 91% to 88% over two weeks. The on-call engineer reads fifty failing traces at random and finds a variety of different-looking problems: a timeout here, a poor retrieval there, an odd answer elsewhere. No single cause stands out, and a colleague suggests that the failures are simply noise from model nondeterminism.

The team has 200,000 traces from the period, each with attributes such as prompt version, tool set used, document language, document length, tenant tier, and whether a particular search tool was called.

What is the best next step?

Practise this domain in a practice exam →