Human review and calibrated confidence

A system extracts data from forms with 97% accuracy. The team proposes to stop reviewing everything the model marks as high confidence. It sounds reasonable, but that 97% can hide a document type running at 70% or a critical field that fails one time in ten.

The average hides segments

Aggregate accuracy mixes everything together. If 90% of the documents are simple invoices, they can cover up the customs declarations that come out wrong. Before reducing human review:

• measure accuracy by document type and by field;
• automate only the segments that are already reliable.

Example: intake forms at 98.5% overall, but the diagnosis code is right 89% of the time. The four reliable fields are automated and the diagnosis code keeps human review for now.

Keep measuring what was automated

Once the high-confidence group goes through without review, how do you know whether it started failing? With a sample of that same group. Reviewing only the doubtful items never measures the errors the automation lets through with confidence, which are exactly the ones that matter.

Stratified random sampling draws the sample at random within each document type. With a uniform 1% sample, a rare document type barely shows up, and a change in its layout goes unnoticed. With stratification, every segment gets enough reviews on the same budget, and new error patterns get detected too.

If a supplier changes its invoice layout and that type’s accuracy drops, pause automation for that type, measure a labelled sample of the new layout and only then decide.

Field-level confidence, calibrated

The model can return a field-level confidence score with each value. But a number is useless until you compare it with the truth. Calibration is checked against a labelled validation set: documents whose correct values were verified by people. If it is calibrated, of the fields marked 0.9, about 90% are correct.

Scores do not mean the same thing in every field. For example, in bank forms, an IBAN at 0.90 may be right 71% of the time and an email at 0.70 right 98%. That is why thresholds are set per field, using the labelled set and according to the cost of the error: the IBAN needs a much stricter threshold.

Two runs giving the same value measures consistency, not accuracy: the model can be wrong the same way twice.

Who gets reviewed

With limited reviewer capacity (four people for 600 out of 2,000 daily declarations), reviewing in arrival order wastes time. Prioritise:

• fields with low confidence on the calibrated scale;
• ambiguous or self-contradicting documents (two different gross weights on two pages).

And the alert to the reviewer has to allow a quick decision: the field, the extracted value, the source passage and the reason for the flag.

The typical traps

• Automating based on aggregate accuracy.
• Sampling only low-confidence items.
• A uniform sample that ignores rare document types.
• A single threshold for every field, or trusting scores that were never compared with labelled data.

On the exam

If the scenario gives a high overall percentage and proposes removing review, the answer asks to measure by type and by field. If it asks how to keep monitoring, stratified sampling that includes high confidence. If it asks whether the scores can be trusted, compare them with a labelled set.