CCDV-P · Advanced Debugging and Incident Response
Isolating where a failure originates, model, harness, tools or data
A back-office agent, on turn 9 of a task, issues an invoice with the wrong amount. Re-run with the same history, it sometimes gets it right. The team argues: does the model err, is the prompt ambiguous, did the tool return stale data? With no method, each person defends a favorite hypothesis.
Four places the problem can live
The problem origin sits among four candidates, and the fix depends on which:
| Origin | What it includes | Typical evidence |
|---|---|---|
| Model output | faulty reasoning, instruction not followed, hallucination | with correct input and tools, the output is wrong |
| Agent harness | the loop, context assembly, truncation, compaction, retries, limits | the model didn’t receive something it should have, or received something trimmed |
| Tools | errors, wrong or incomplete results, misleading descriptions, timeouts | the tool result was already bad before reaching the model |
| Data | stale source, wrong retrieval, unexpected format | the right data wasn’t there, or an old value was |
The method: reconstruct what the model saw at the failing step (the exact context, not what you assumed it saw), and compare with what it should have seen. If the input was already wrong, the origin is upstream. If the input was right and the output wasn’t, only then is it the model.
Multi-turn and multi-agent failures
In long tasks the visible error is often far from its cause. A wrong value introduced on turn 3 surfaces as a wrong amount on turn 9. You trace backwards: when did the incorrect value first appear in the context? In multi-agent systems, the question extends to the seams: did the subagent return it that way? Was it lost or distorted in the handoff? Was an empty error read as “no results”?
Non-deterministic failures
A failure that happens “sometimes” isn’t unexplainable. You replay the case multiple times with the same input and measure the failure rate, and compare what differs between good and bad traces (the tool called, the order, the retrieved content). Lowering temperature doesn’t eliminate non-determinism and isn’t a diagnosis. A failure at a 20% rate that drops to 2% after a fix is a verifiable fix; one that “hasn’t happened again” is not.
Refusals as a failure class
A model refusal is a response in which Claude declines, or a refusal stop_reason. It isn’t a technical error: it’s a signal that the request, or the context you assembled, triggered a safeguard. Before treating it as a bug, look at what was asked. Usual causes: the request really is inappropriate, or retrieved or injected content changed the intent, or an ambiguous prompt seems to ask for something prohibited. You diagnose it by reading the context, not by retrying.
Grounding failures
When the answer asserts something unsupported or cites the wrong source, apply the stage-by-stage analysis of retrieval: did the data exist? was it retrieved? did it reach the context? did the model use it? That breakdown stops you blaming the model for a stale index.
The usual traps
- Blaming the model before reconstructing its input.
- Fixing the turn-9 symptom without finding where it began.
- Treating a refusal as a transient error and retrying.
- Closing an intermittent failure because it “didn’t reproduce” in one attempt.
- Adding a prompt instruction for what is bad data.
On the exam
Evidence first: what did the model receive? If the input was already wrong, the origin is in tools, data or harness. For intermittent issues, measure rates and compare cohorts; for a refusal, read the context.
Quick question
A practice exam question on this domain. It counts towards your progress.
A travel-booking agent calls a search_flights tool with a travel date. Customers report that some searches return flights for the wrong month. Reviewing traces, the engineer sees that for a request like "the 3rd of April", the model sent the value "03/04/2026" in the date field. The tool's schema describes that field only as "date" with type string, and the backend parses strings using the US month-first convention.
The same model correctly produced ISO dates in other tools whose schemas specify the expected format.
Where is the most likely origin of the problem, and what is the best fix?