claudio.cool
CCDV-PLessons

CCDV-P · Evaluation Engineering and Quality Assurance

Designing the evaluation system, golden sets, regression and release gates

⏱ 3 min readDraft

A team changes an assistant’s prompt every week. Every change “looks better” in manual tests with five examples. Two months later a customer discovers that a case that used to work stopped working and nobody knows when. Without an evaluation system, every change is a bet.

The pieces

An offline evaluation system is made of:

  • Golden set: representative cases with the expected output or criterion, reviewed by people. It includes typical cases, edges and ones that failed before. Its size must be enough to tell real changes from noise.
  • Regression suite: the cases that must never fail again (each incident adds one). It runs in full on every change.
  • Behavioral checks: tests of properties rather than exact text: follows the format, refuses what it should, doesn’t reveal another user’s data, uses the right tool, stays under a length. Some are verified with code (deterministic) and others with a judge.
  • Metrics by system type: in a retrieval system, retrieval precision and recall separate from answer quality, and groundedness (what share of claims are supported by the retrieved sources).

Wherever possible, prefer deterministic checks (valid schema, value matches, the expected tool was called) over a model’s judgment.

The CI/CD gate

An evaluation gate runs the suite in the pipeline when a prompt, model, tool or agent configuration changes, and blocks the merge if the release criteria aren’t met. Good criteria:

  • thresholds per category, not just a global average, which hides a category that got worse;
  • no regressions on critical cases (security, compliance, previously failed cases);
  • cost and latency limits as well as quality, because an improvement that triples the bill may not be worth it;
  • comparison against the current version on the same cases, not against an absolute number;
  • for non-deterministic outputs, multiple runs per case, looking at the pass rate rather than a single result.

For model and agent changes you add agentic behaviors: tool sequence, number of steps, completion rate.

Keeping the suite valid

A suite that’s good today degrades:

  • Contamination: suite cases leak into the data you use to tune the prompt (or show up as an example in the prompt), and the system “learns the test”. Keep a held-out subset not used for iteration.
  • Saturation: when nearly every case passes, the suite stops distinguishing improvements. Add harder cases.
  • Coverage gaps: production brings cases the suite lacks. Feed it from real failures, feedback and sampled traces; each incident adds a case.
  • Stale cases: ones whose correct answer changed (a new policy) are updated or retired.

The usual traps

  • Evaluating with five hand-picked examples.
  • A single global average as the criterion.
  • Tuning the prompt while looking at the same cases you measure with.
  • Thresholds lowered every time a change fails to pass.
  • A suite that never updates from what happens in production.

On the exam

If a change “improves” the average but worsens a critical category, the gate should block. If nearly everything always passes, that’s saturation. If the model does very well on the suite and poorly in production, suspect contamination or coverage gaps.

Quick question

A practice exam question on this domain. It counts towards your progress.

A legal-tech company runs a retrieval-augmented assistant that answers questions about internal contract templates. Reviewers report that about one answer in six contains a clause that is not in the retrieved documents or cites the wrong template.

The team has only one end-to-end score: a judge model rates each final answer from 1 to 5. When the score drops, they cannot tell whether the retriever returned the wrong passages or Claude ignored the right ones, so every investigation starts from scratch.

Which change to the offline evaluation would make failures diagnosable?

Practise this domain in a practice exam →