Message Batches: which jobs go in a batch

A team processes 20,000 documents every night and also runs a review before every merge. Both use the same model and the same prompt. One can go in a batch and the other cannot, and the difference is not the volume: it is whether someone is waiting for the answer.

What it offers

The Message Batches API processes many requests asynchronously:

• it costs 50% less than normal calls;
• each batch is processed within up to 24 hours (often much sooner), with no latency guarantee;
• each request carries its own custom_id, and results can come back in any order: the custom_id is the only thing that lets you match a response to its document.

What goes in a batch and what does not

Job	API
Overnight report, weekly audit, nightly test generation, reprocessing a historical archive	Batches
Pre-merge check that blocks the PR	Synchronous
Answering a customer in the support chat	Synchronous
Agent that calls tools and continues based on the result	Synchronous

The rule: batches fit latency-tolerant workloads that do not block anyone. A blocking workflow, such as a pre-merge check, cannot wait up to 24 hours.

Watch out for the last case: within a batch request there is no multi-turn tool calling. The model can request a tool, but nobody runs it and returns the result within the same request. An agentic flow that needs to go back and forth with tools does not fit in a batch.

How often to submit

If the business promises an SLA of 30 hours from document arrival to result, and the batch can take 24, the worst case is: the document arrives right after a submission, waits for the next window and then takes 24 hours. So:

submission window + 24 h ≤ SLA

With a 30-hour SLA, the window cannot exceed 6 hours. The guide uses 4-hour windows: the worst case stays at 28 hours and leaves some margin.

When something fails

A batch of 10,000 requests comes back with 3 errors. The other 9,997 are already fine:

• resubmit only the failed ones, identified by their custom_id;
• fix the cause first: if a document exceeded the context, split it into chunks; if it failed validation, send it with its specific error;
• cap the resubmissions, and whatever keeps failing goes to review.

Test before launching

Before sending 200,000 documents, refine the prompt on a sample. Every error that slips through in the prompt is multiplied by the volume and forces resubmissions. Measuring first on a labelled sample raises the first-pass success rate and lowers the cost of corrections. The same goes for cost changes: if you want to send simple documents to a smaller model, measure each route’s accuracy on labelled data before switching.

The typical traps

• Using batches for the pre-merge check “because it is cheaper”.
• Thinking batches work for an agent with tools.
• Resubmitting the whole batch because a few failed.
• Matching results by position instead of by custom_id.
• Launching the full volume without testing the prompt.

On the exam

Check whether the scenario has someone waiting (a blocked PR, a customer). If so, synchronous. If not, batch. If they ask for the submission frequency, subtract 24 hours from the SLA. If some requests failed, the answer mentions custom_id and resubmitting only those.