Diagnosing failures, prompt, hallucination, retrieval or model mismatch

After the document base is reloaded, the internal assistant starts giving confident, wrong answers. Latency is the same and the model ID hasn’t changed. In the crisis meeting, someone proposes moving to a larger model and someone else suggests lowering the temperature to zero. Neither proposal comes from looking at what changed.

First, what changed and where it fails

A serious diagnosis starts with two questions:

• What changed? The prompt, the documents or the index, the model, the tools, the kind of traffic. If the model ID is the same, don’t blame “silent model changes” without evidence: look first at what did change.
• Which segment fails? Everything, only complex cases, only one topic, only queries with no documents retrieved. A failure concentrated in one segment almost always points to the cause.

Symptom	Likely cause	First check
Confident but wrong answers after a document refresh	Retrieval failure: broken index, stale or irrelevant chunks	Look at which chunks were retrieved for the failing queries
Makes things up when no documents support it	Prompt-induced hallucination	Does the prompt demand a complete answer even when the context is empty?
With the same prompt, only complex cases fail on the new, smaller model	Model mismatch for that segment	Run the same prompt on the previous model, split by complexity
Ignores format or instructions on any model	Prompt failure	Check for ambiguous or contradictory instructions, or missing examples
Answers cut off mid-sentence	max_tokens was reached	Check the stop reason in the logs

Hallucinations: there’s almost always a reason

When the retrieved context is empty or not enough, and the instruction says “always answer completely and definitively”, the model fills in with whatever sounds plausible. The fixes go into the prompt and the design:

• give explicit permission to say it doesn’t have the information and hand off;
• external knowledge restriction: answer only from the documents provided, not from general knowledge;
• for long documents, ask for the relevant verbatim quotes first and answer from them;
• ask for every claim to cite its source and withdraw any claim without support.

Adding “similar policies” to the knowledge base so retrieval finds something makes the problem worse: the model answers with another product’s information.

Prompt or model

To separate them, keep the prompt fixed and change only the model, on the same set split by segment. If the previous model still does well with that prompt and the new one fails only on multi-step cases, the problem is the model for that segment. The way out is not rolling everything back: route complex cases to the more capable model and keep the smaller one for simple queries, which keeps almost all the savings.

If instead every model fails the same way, the problem is in the prompt or the data, and switching models only hides it.

The usual traps

• Moving to a larger model “because it knows more”: it doesn’t fix a broken index or a prompt that forces it to make things up.
• Lowering the temperature to zero: the made-up answer still comes out, just more repeatably.
• Opening a ticket about a supposed change in model weights without having looked at retrieval.
• Retries with backoff for truncated answers: it isn’t a transient error.

On the exam

If the failure appears right after documents are reloaded and nothing else changed, the first place to look is retrieval. If the model makes things up when there’s no context, look for the option that lets it say it doesn’t know and restricts it to the documents. If the failure appears after switching to a smaller model and only on hard cases, look for routing by complexity. Be wary of any option that blames the model without evidence.