Read the trace, is the integration failing or the model's output?

A shipping service uses a create_shipment tool. On large orders, the input JSON fails to parse, and the team is already comparing models to see which one “builds JSON better”. Nobody noticed that without streaming the input always arrives perfectly. The model was never the problem: the code that assembled the fragments was. Before switching models or prompts, you have to read the trace.

What a trace must contain

For each call, enough to reconstruct what happened without guessing:

• the exact request your code built: system, messages, tools, parameters;
• the raw response: content blocks, stop_reason and usage (input and output tokens);
• the request ID returned by the API, the model ID and the prompt version;
• in an agent, each tool_use with its input, what your code did and the tool_result it returned;
• timings and errors, including error events inside a stream.

With only the final text saved, it’s impossible to know whether the problem was what the model asked for or what your code did afterwards.

Integration or model

The integration layer is everything between your application and the API: building the request, parsing the response, running tools, returning results. The key question is: was the model’s raw output correct?

Symptom in the trace	Where it usually is	What to do
stop_reason: max_tokens and output_tokens equal to the limit	Configuration	Raise max_tokens and treat the cutoff as incomplete
Broken JSON only with streaming; without streaming it arrives fine	Integration	Accumulate the input_json_delta events and parse when the block closes
The tool_use has the right input, but the result is wrong	Integration	Check the dispatcher: mapped name, types, units
A 400 naming a tool_use without a tool_result	Integration (harness)	Return one result per block, together
Empty end_turn responses after tools	Integration (harness)	Send only the tool_result blocks, with no text added after them
The model picks the wrong tool or invents a value that wasn’t there	Model, prompt or descriptions	Improve descriptions, prompt or examples; only then evaluate another model
stop_reason: refusal	Request content	Another path: rephrase, fallback or human review

How to isolate the origin

1. Reproduce the issue with a concrete case that fails.
2. Take the exact request from the trace and send it directly with the SDK, without your application in between. If the raw response is fine, the error is in your integration.
3. Change one thing at a time: with and without streaming, with and without the tool, with the previous prompt.
4. Only when the model’s raw output is consistently wrong, work on the prompt, descriptions or model, and measure it against a set of cases.

That process turns “it sometimes misbehaves” into a concrete failure mode: “on orders with more than forty items, the input gets parsed before the block finishes”.

The typical traps

• Switching models before reading the trace.
• Blaming the model for an error in how your code processes its output.
• Saving only the final text, without stop_reason, usage or the exact request.
• Testing several changes together and not knowing which one fixed the problem.
• Blindly retrying something that always fails the same way.

On the exam

Read the trace data the scenario gives you: stop_reason, output_tokens, whether it happens with and without streaming, what the tool returned. If the model’s raw output is fine, the answer is in the integration, so rule out options that switch models or add instructions. If the trace shows the model chose wrong or invented data, that’s when to look for prompt or description improvements.