CCDV-P · Observability and Production Monitoring
Trace analysis at scale, sampling, aggregation and failure cohorts
An agent processes 200,000 tasks a day. The team manually reviews twenty failed traces a week and each has a different, plausible cause: a timeout, a slow tool, an odd response. None suggests a pattern. But 4% of tasks fail, and all the failures share something nobody sees because they’re looked at one by one.
Why reading traces isn’t enough
At scale, individual reading has two problems: it’s a small, biased sample (you read the eye-catching ones) and it doesn’t quantify. A systemic issue shows up as a statistical difference between groups, not as something strange in one trace. The question changes from “what happened to this one?” to “what do the failing runs have in common that the healthy ones don’t?”.
Sampling that doesn’t hide failures
Storing every trace can be expensive, so you sample. How you sample defines what you’ll be able to see:
- Uniform sampling (a fixed percentage): cheap and representative of normal traffic; with low failure rates, few of the stored traces are failures.
- Tail-based sampling: decided after the trace finishes, to keep all (or nearly all) the ones that failed, were slow, were expensive or got negative feedback, plus a percentage of the good ones.
- Stratified sampling: a percentage per tenant, feature or prompt version, so small groups don’t vanish.
Always record the sampling rate with each trace so you can reweight and compute real rates. Aggregate metrics (counts, tokens, latencies) are computed over 100% of traffic, not over the sample.
Aggregation and cohorts
Aggregation turns millions of events into distributions: failure rate, tokens and latency by dimension. A failure cohort is a group of runs sharing a characteristic. The analysis is segmenting and comparing:
- by version (prompt, model, tool, deployment);
- by tenant, language, task type or channel;
- by trace structure: number of turns, tool sequence, result sizes, the first step that failed;
- by time: before and after a change.
Example: the 4% failure concentrates in tasks with more than 15 turns that use a search tool whose results exceed a certain size. That points to context saturated by bulky results, and it’s quantifiable (how many tasks, how much cost) for prioritizing. Then you test the hypothesis: fix it for that cohort and verify the rate drops where expected.
Watch the conclusions
A cohort can be correlated without being the cause: long tasks may fail more simply because they’re hard. Compare with a similar cohort, check when the difference began and validate with a controlled change.
The usual traps
- Deciding what to fix from the most eye-catching trace.
- Uniform sampling with low failure rates, concluding “there are hardly any failures”.
- Not recording the sampling rate, so real rates can’t be computed.
- Computing global metrics over the sample instead of full traffic.
- Confusing correlation with cause.
On the exam
If individual failures look random but the total rate matters, choose the option that aggregates and segments by cohort. If the problem is that failures vanish from the sample, tail-based or stratified sampling.
Quick question
A practice exam question on this domain. It counts towards your progress.
A fraud-review agent produces 40 million traces per day. Storing all of them with full span detail is too expensive, so the team samples 1% of traces at the start of each request. The failures the team cares about, such as tool errors, runaway loops, and low-quality outputs, occur in about 0.4% of requests. When investigating an incident, the team often finds no failing traces in the stored sample.
The budget cannot grow more than a little, and the team still needs an unbiased view of healthy behavior.
Which sampling strategy is the best fit?