Real time or Message Batches, streaming, caching and Claude through third parties

Every month, a process labels ten thousand recipes with their dietary restrictions. They moved to batches to save money, and now the labels are correct but end up on the wrong recipe. Meanwhile, another team uses the real-time API for a nightly job nobody watches, and pays full price. Both chose the wrong way to call the API or used it wrong.

Real time or batches

If…	Use
Someone is waiting for the answer (a chat, a form, an interactive agent)	Messages API, in real time
The work can wait hours (classifying a historical archive, evaluations, nightly reports)	Message Batches API

The Message Batches API processes requests asynchronously, at half price. Most batches finish in under an hour, but the window can reach 24 hours; anything not processed in that time expires. Each request in the batch is a normal Messages API request, so it supports tools, images and a system prompt.

What comes up most:

• Each request carries a unique custom_id, for example the recipe ID.
• Results can come back in any order. They’re always matched by custom_id, never by position.
• Each result has a type: succeeded, errored, canceled or expired. The ones that failed or expired get resubmitted, not treated as good.

Streaming, vision and thinking, just enough

• Streaming: if a person is watching the screen and the answer is long, it’s sent in pieces so they see progress from the first second. It takes the same time, but it stops looking frozen. The events are in the SDK lesson.
• Vision: images and PDFs go in as content blocks (base64, by URL or by file ID, depending on the platform). It’s best to put the image before the question and to validate what gets extracted, just as with text.
• Thinking: the model’s reasoning arrives in its own blocks. In a tool loop, those blocks are passed back unchanged in the history; if your code drops or edits them, the API may reject the request.

Caching: prompt order matters

Prompt caching reuses an exact prefix of the request, in the order tools, system and messages. If the first line of the system prompt holds the user’s name and the time, the prefix changes on every call and the cache is always written and never read. The rule:

1. Fixed content first: tool definitions, instructions, reference manuals.
2. The cache breakpoint at the end of the fixed content.
3. Variable content (user, date, question) after it.

Cost and measurement are covered in the model selection domain.

Claude through third parties

Claude is also used from Amazon Bedrock, Google Cloud (Vertex AI) and other clouds. The message shape is the same, but several things change:

• Authentication: cloud credentials (for example, an IAM role on AWS), with the SDK client for that platform, not the Anthropic API key.
• Model IDs: each platform has its own format and its regional or global endpoints; that’s where data residency gets resolved.
• Feature availability: not everything is everywhere, or at the same time. On Bedrock, for example, there are no URL image sources and no Files API. Before migrating, you check every feature the application uses.
• Lifecycle: each platform sets its own retirement dates.

The typical traps

• Parallel synchronous calls for work that can wait.
• Matching batch results by order, or treating errored and expired results as good.
• Putting variable data at the start of the prompt and expecting cache hits.
• Migrating to a cloud by changing only the URL, without checking authentication, IDs and features.

On the exam

First ask whether someone is waiting for the answer: that decides between real time and batches. If the batch results are right but in the wrong place, look for custom_id. If the cache is never read, look for the option that moves variable content after the cache breakpoint. And if the scenario migrates to Bedrock or Vertex AI, choose the options about cloud credentials, platform IDs and checking features.