Explicit criteria instead of "be conservative"

An agent that reviews pull requests in CI flags almost everything. A month later, the team resolves the comments without reading them. The problem is rarely the model: it is that the prompt asks for a judgement (“check that comments are accurate”, “flag anything risky”) without saying what counts as a finding.

What a criterion is

An explicit criterion describes something observable and checkable by reading the code: a behaviour, a pattern, a condition. “Use your judgement”, “be thorough” or “flag what is risky” are general values; every run interprets them differently.

Vague	Explicit
“Check that comments are accurate”	“Flag a comment only when the behaviour it describes contradicts what the code does”
“Flag SQL injection risks”	“Flag only when external input (request, file, another source) reaches a query; skip constants and parameterised placeholders”
“Report only real bugs”	“Report behaviour that can break at runtime; not style, naming or formatting”

A good criterion says both things: what gets flagged and what gets skipped. The list of exclusions (local team patterns, minor style, cases already guarded earlier on the path) is what cuts false positives the most.

Why “be conservative” does not work

“Be conservative” or “report only high-confidence findings” set no boundary. And self-reported confidence is poorly calibrated: the model can be very sure about a false finding. A 0.8 threshold as the only filter fixes nothing. The right order is criteria first, to decide what a finding is, and only then, if you want, confidence for routing.

Capitals, repeating “DO NOT report style” or lowering the temperature do not help either. If the model keeps flagging style, add examples of code that looks like a style issue and is not reported, together with the skip criterion.

False positives cost trust

A noisy category drags the others down: if half the “variable naming” comments are irrelevant, developers stop reading the security ones too. That is why the guide proposes temporarily disabling the category with the most false positives to restore trust while you improve its prompt, and turning it back on when measurements show the noise has dropped. That is not the same as switching categories off blindly or lowering their severity so they bother people less: that hides the real findings and keeps the false ones.

Severity with examples

If “critical”, “high” and “minor” come out differently on every run, describing the levels in prose is not enough. What stabilises the classification is severity criteria with one concrete code example per level and the reason it is classified there. The model compares against cases instead of interpreting adjectives.

The same goes for making a finding actionable: ask for file and line, the failing condition and one concrete change.

Measure the change

To know whether the new prompt is better, run both versions on a labelled set of past diffs and compare the false positive rate per category. The noise “seeming” lower the following week is not evidence.

The typical traps

• “Be careful”, “only high confidence” or a confidence threshold as the only filter.
• Emphasis: capitals, repeating the rule, temperature 0.
• Lowering the severity of the noisy category instead of fixing its criterion.
• A second model pass that repeats the same vague judgement.

On the exam

The scenario is usually Claude Code in CI with a team that ignores the comments. Look for the option that names concrete patterns to flag and to skip. If the question is about inconsistent severities, the answer is one example per level. If it is what to do right now with a category that is destroying trust, disable it temporarily while it is improved.