Human in the loop by risk, and how to avoid automation bias

An insurer is debating its claims-approval agent. Legal wants an adjuster to sign off on every decision; operations wants to automate everything and let customer complaints correct the mistakes. In another department, reviewers approve 99.6% of the letters the AI drafts in four seconds on average, and errors still reach customers. Both situations are the same problem badly solved: human review placed where it doesn’t add value.

Neither everything nor nothing

Human-in-the-loop is an expensive, limited control: people’s attention. Reviewing everything wears it out on routine work; reviewing nothing lets through the mistakes that hurt most. Risk-based review concentrates it where it matters:

Factor	Example	Review
High impact	large amounts, legal or health decisions	before executing
Irreversibility	a payment, a shipment, a deletion	before executing
Adverse decision for a person	a rejection, a cancellation, a penalty	before communicating it
Low confidence or failed validations	a doubtful photo, data that doesn’t add up, an unusual case	before executing
Low-risk routine	what evaluation shows comes out right	goes through on its own, with audit sampling

Sampling is what keeps automation honest: a random share of what goes through on its own is reviewed afterwards, and if the error rate rises, the thresholds are adjusted. Reviewers’ corrections also go back into the evaluation suite.

Automation bias

Automation bias is the tendency to accept what the system proposes without checking it. The signs are clear: approvals near 100%, seconds per review and errors that show up later. The review exists on paper, not in practice. It’s fixed by design:

• send only substantive cases to review: if everything goes through the queue, everything gets approved on autopilot;
• show the evidence for each claim: where each figure comes from, which clause was applied, so checking is quick;
• seed known errors into the queue and measure how many reviewers catch (seeded errors);
• size the queue: if the load exceeds capacity, people approve to empty it.

Requiring a written justification for every approval or adding a second reviewer to everything adds work without addressing the cause: two people who trust the system still trust the system.

The usual traps

• A reviewer signing off on every decision “because any automated decision carries risk”.
• Automating everything and using customer appeals as quality control.
• Choosing what to automate by criteria unrelated to risk, such as customer tenure.
• Removing review because “they approve almost everything anyway”.
• Measuring reviewers on speed rather than errors caught.

On the exam

If the scenario frames it as “review everything or nothing”, look for the option that puts review on high-impact, low-confidence or adverse-decision cases and audits a sample of the rest. If reviewers approve almost everything in seconds, look for the combination: only substantive cases to review, evidence in view and seeded errors to measure. Rule out the ones that add bureaucracy or remove the control.