Tokens, non-determinism, thinking and effort, fast mode and few-shot

A test compares the response byte for byte against a snapshot and fails every night with different cases. A twelve-label classifier returns stop_reason: max_tokens with no text. A team wants a code assistant’s patch to finish arriving sooner and tries a smaller model. All three problems are solved by understanding how the model generates and what options it has, not by switching models blindly.

Tokens and generation

The model processes and produces tokens (word fragments), one at a time: each new token is chosen based on everything before it. Three practical consequences follow:

• Everything that goes in (system prompt, history, tools, documents) and everything that comes out takes up the context window and is billed per token.
• Thinking is output tokens too: it counts toward max_tokens and is billed even when it isn’t shown.
• If the window fills up, the response stops with stop_reason: model_context_window_exceeded; if max_tokens runs out, with max_tokens.

Non-determinism

Generating means choosing among possible tokens, so the same input can produce different outputs. That is non-determinism, and it can’t be fully switched off: not even lowering the temperature on the models that accepted it guaranteed identical responses, and the newest models reject temperature, top_p and top_k outright. Behavior is guided with the prompt and verified with evaluations.

The consequence for tests: validate structure and properties (the correct category, the required fields, that the justification quotes the text), not the exact text.

Thinking and effort

• Extended thinking with budget_tokens: the developer set how many tokens the model could think. It’s the old scheme, and new models reject it.
• Adaptive thinking: the model decides on each request whether to think and how much. Some turns may have no thinking block.
• Effort, in output_config.effort (low, medium, high and, depending on the model, xhigh and max): the calibrated lever for how much work the model does. It affects everything it generates: thinking, text and tool calls.

If the model overthinks a simple task, lower the effort and confirm with your eval set that quality holds. Asking in the prompt “don’t think” is not reliable. If responses get cut off by max_tokens on tasks that really needed reasoning, raise max_tokens.

Fast mode

Fast mode, with speed: "fast", runs the same model with faster inference:

• same quality and same capabilities;
• more output tokens per second, not an earlier first token;
• premium pricing, and available only on some models;
• it can’t be combined with the Message Batches API.

It helps when the user waits for the end of a long response. If the problem is cost, it’s the wrong tool.

Zero-shot, one-shot and few-shot

No examples (zero-shot) is enough when the task is clear and the format is flexible. When the content comes out right but the format varies (dates written three ways, amounts in words), few-shot (multi-shot) prompting is the most direct fix: three or four varied cases with the exact expected output. A single one (one-shot) tends to make the model copy that case too closely.

The usual traps

• Lowering the temperature or pinning top_p to “make it deterministic”.
• Retrying a test until it matches, or regenerating the snapshots every night.
• Writing “answer without thinking” in the prompt instead of lowering the effort.
• Going back to budget_tokens on a model that uses adaptive thinking.
• Believing fast mode uses a smaller model or shortens time to first token.
• Switching to a larger model when what fails is the format.

On the exam

Read which symptom the scenario describes. Responses that vary while the content is correct: validate properties. Thinking that eats max_tokens on a simple task: lower the effort and measure. Users waiting for the end of a long output: fast mode. Inconsistent format with good accuracy: varied examples.