CCDV-P · Evaluation Engineering and Quality Assurance
An LLM judge must be calibrated, rubrics, biases and agreement with people
A team evaluates a writing assistant’s responses with an LLM-as-judge: a model that scores each response from 1 to 5. The average score rises every week, but human editors say the responses are getting longer and more generic. The judge is rewarding something people don’t value.
A judge is a measuring instrument
An LLM judge is useful because it scales, but it is an instrument, and like any instrument it must be verified against a reference. The reference is human labels on a sample: expert people score the same cases with the same rubric. What you measure is the agreement between the judge and the people, before trusting it.
Calibration
Calibration is making the judge’s score correspond to human judgment. The process:
- Build a human-labeled set, with examples of each level, including hard cases.
- Run the judge and measure agreement (percent match, or a chance-adjusted metric such as Cohen’s kappa, or correlation if the score is ordinal).
- Examine the disagreements one by one: is the rubric, the judge prompt or the human label at fault?
- Adjust the rubric or prompt and re-measure on cases not used for tuning.
- Set an agreement threshold. Until it’s reached, the judge reports but doesn’t block releases.
Rubric design
A good rubric breaks the criterion down: instead of “is the response good?”, separate criteria (accuracy, coverage of what was asked, tone, faithfulness to sources), each with levels described with examples. Ask for brief reasoning before the score, and structured output so it can be aggregated. When a reference answer or a verifiable rule exists, use it rather than asking the judge for an opinion.
Known biases
- Position effect: in pairwise comparisons, a tendency to pick the response shown first (or second). Control it by randomizing or swapping order and counting only a verdict that holds in both orders.
- Verbosity bias: preferring longer responses. Mitigate it in the rubric (length earns nothing) and detect it by correlating score with length.
- Self-preference: favoring responses from a model of its own family or style; ideally the judge isn’t the same model being evaluated, or at least you measure it.
- Format sensitivity: bullets and bold influencing more than they should.
Inter-rater reliability and drift
Inter-rater reliability measures whether two evaluators, human or the judge, agree with each other. If the people themselves disagree, the rubric is ambiguous: the problem is upstream of the judge. Also, the judge is a model and it changes when its version or prompt changes: judge drift means today’s scores can’t be compared with those from three months ago. You pin the judge version, re-measure agreement on the labeled set after any change, and keep the history.
The usual traps
- Giving decision power to a judge without measuring its agreement with people.
- Moving to a bigger model as the only fix for bias.
- A single general-question rubric.
- Comparing scores across different judge versions.
- Tuning the judge on the same cases used to validate it.
On the exam
A judge with low human agreement or a visible bias isn’t trustworthy yet: control the bias, re-measure against the labels and remove its blocking authority until agreement is acceptable. When the judge’s version changes, recalibrate.
Quick question
A practice exam question on this domain. It counts towards your progress.
A retailer runs a Claude agent that processes refund requests by calling order, policy, and payment tools. The online dashboard tracks how often the agent's final message says the request was resolved, which is currently 96 percent. An audit of a random sample shows that in 9 percent of "resolved" cases the refund amount was wrong or a refund was issued for an ineligible order.
The team is choosing metrics for a new online evaluation, and wants to compare two versions of the agent through an A/B test on live traffic.
Which primary metric best fits this agentic task?