Sounding confident doesn't make it true: checking against the source

A coordinator asks Claude to summarize a new regulation. The summary is clear, well written and cites “subsection 4.b” with complete confidence. The subsection doesn’t exist.

This has a name: hallucination. A language model can invent very specific-looking details (an article number, a dated verbatim quote, a statistic from a named survey) and write them in the same tone as real data. The tone tells you nothing about whether the detail is true.

Two different checks

Reviewing an output means looking at two things separately:

• Accuracy: every figure and every claim has to be traceable to the source. If the summary says “18% total growth” and the report shows 18% for a single region, the figure is misattributed.
• Completeness: compare the output against what it had to cover. If the agenda had nine items and the meeting summary has six action items, it may be fine, but you have to confirm it against the agenda and the transcript.

Warning signs

These don’t prove there is an error, but they tell you where to look first:

• totals that don’t match the sum of their own line items;
• a figure that changes between sections (24 months in one place, 36 in another): an internal inconsistency, so at least one is wrong;
• very specific figures, quotes or references that weren’t in the material you gave it;
• similar cases described differently (that may be bias, covered in another lesson).

What is not a sign: the text sounding confident, not using hedging words, or having simplified the source’s language.

How to actually verify

Fact-checking is done against an authoritative source: the official text of the regulation, the original report, the speech transcript.

• Open the source. A citation proves nothing: the cited page may say something else.
• If you find one error, check the rest. One invented detail shows there may be others of the same kind. If one of five quotes can’t be found, all five become unverified.
• If you can’t confirm it, remove it. A statistic that doesn’t appear in any source doesn’t get published.
• Short on time, prioritize. Start with the claims that drive the decision (the 40% saving that justifies the investment), not the colour details.
• Prevent it in the request. Give Claude the documents and ask it to quote only from them; that way every quote can be checked.

Common traps

Option you’ll see	Why it’s wrong
Send it as is because Claude “was confident”	Confidence is not a sign of accuracy
Ask Claude whether the figure is real or to rate its confidence	The same model that invented it is not an independent check
Reword it more formally or change it to “around”	It changes the form, not the content
Fix only the error you found	It leaves the others of the same kind in place
Regenerate and keep what matches	Two model outputs are not a source

On the exam

If the scenario has a very concrete detail (a subsection number, a dated quote, a survey percentage) going to an audience that will rely on it (compliance, the board, customers), the answer is to verify it against the original source before sharing. Rule out anything that depends on Claude assessing itself and anything that only improves the wording. In “choose two signs” questions, keep the internal inconsistencies (totals that don’t add up, figures that change) and not the tone.