CCDV-P · Evaluation Engineering and Quality Assurance
Measuring quality on live traffic, sampling, A/B tests and feedback
The offline suite is clean and controlled, but production brings what the suite never imagined: odd queries, other languages, users who paste entire documents. Online evaluation measures quality on real traffic, asking the same questions you ask offline but about what actually happens.
Live-traffic sampling
You can’t review everything. Live-traffic sampling chooses what gets evaluated:
- a random sample to estimate overall quality;
- targeted samples on risky ground: outputs flagged by users, long or expensive responses, refusals, human escalations, new tenants;
- automatic evaluation (rules and a calibrated judge) on a larger sample, and human review on a smaller one to check the judge.
Mind privacy: content samples follow the system’s retention and redaction rules.
A/B tests
An A/B test compares two versions (prompt, model, flow) on real traffic split at random. What makes it valid:
- random assignment consistent per user or session (the same person doesn’t see both);
- a hypothesis and a primary metric set in advance, plus guardrail metrics (cost, latency, refusal rate) so you don’t improve one thing by breaking another;
- enough sample size and duration to tell the difference from noise; you don’t stop the moment it “looks better”;
- the rest of the system held constant during the test.
An earlier, safer step is shadow mode: the new version runs on real traffic but isn’t shown; you compare it against the current one with no risk to the user.
The feedback loop
The feedback loop turns usage signals into improvements:
- explicit feedback (thumbs up or down, reports) and implicit signals (the user rephrases, copies the answer, abandons, asks for a human);
- signals are tied to the trace and version, to know what caused them;
- reviewed negative cases enter the regression suite and the golden set.
Caution: explicit feedback is scarce and biased (the very happy or very angry speak up), and implicit signals are proxies. A proxy metric such as “copy rate” is a hint, not quality; contrast it with direct evaluation.
Metrics by task type
| Task type | Useful metrics |
|---|---|
| Extraction | per-field accuracy, valid-output rate, missing-field rate |
| Generation | quality rubric with a calibrated judge, human preference, subsequent edit rate |
| Agentic | task completion rate, number of steps, cost per task, human interventions |
| Grounded | groundedness, verifiable-citation rate, correct “I don’t know” rate |
Picking the wrong metric leads to optimizing what doesn’t matter: measuring an agentic task only by the quality of the last answer ignores whether it completed the task.
The usual traps
- Relying only on the offline suite.
- An A/B with no guardrail metrics, or one stopped as soon as a difference shows.
- Treating thumbs-up as quality.
- A generation metric for an extraction task (or the reverse).
- Feedback that never returns to the evaluation suite.
On the exam
To choose between two versions with user risk: shadow mode or A/B with a primary and guardrail metrics. To decide whether feedback is useful: is it tied to a trace and does it return to the suite? To pick metrics, look at the task type.
Quick question
A practice exam question on this domain. It counts towards your progress.
A team operates a Claude-based writing assistant used by thousands of customers. Their only online quality metric is the thumbs-down rate, which has stayed around 1.5 percent for months. Yet an account manager reports that several large customers stopped using the assistant because "it keeps ignoring our style guide".
Only about 2 percent of responses ever receive any feedback, and most of that feedback comes from a small group of enthusiastic power users. Leadership wants a single weekly number it can trust, but the team suspects that the current metric describes only a small and unrepresentative slice of its customers.
What should the team add to measure online quality more reliably?