Optimizing cost and latency and monitoring quality, not just uptime

Every night two hundred thousand tickets are classified with real-time calls to the largest model, and cost grows faster than volume. On another team, the claims assistant’s dashboard is all green while handoffs to a person rise week after week and some replies end mid-sentence. They’re two sides of the same topic: knowing which lever to pull and what to measure so you find out in time.

Each lever targets something different

Lever	What it lowers	When to use it
Message Batches API	cost, and adds throughput	jobs that don’t need an immediate answer: nightly classification, reports, evaluations
Prompt caching	cost and time to first token for the repeated prefix	a system prompt, documents or tool definitions repeated on every request
Complexity-based routing	cost and latency	when most requests are simple and evaluation shows a smaller model is enough for them
Trimming context and output	tokens	only with evaluation: remove what the model doesn’t use, not what it needs
Streaming	perceived latency	interactive interfaces; it doesn’t lower cost or total time

Two rules for the exam. First, the batch API is asynchronous: most batches finish quickly, but there’s no instant answer, so it’s no good for a chat. Second, any optimization that can affect quality (a smaller model, trimmed context) is rerun against the evaluation suite before it ships. Caching has an extra advantage: on most models, tokens read from the cache don’t count toward the input tokens per minute limit, so it also frees up capacity.

Uptime is not quality

An agent can always answer, quickly and with no HTTP errors, and still resolve things worse and worse. Uptime and quality are different signals and need different dashboards:

• Operations: uptime, errors, p50 and p95 latency, rate limit usage.
• Quality: completed tasks, handoff rate to a person, user feedback, refusals, all by flow or segment.
• Cost: cost per transaction, cache hit rate, tokens per request.

An answer cut off by max_tokens is, for the API, a successful response: it doesn’t show up as an error. You only see it if you log the stop reason.

What to log

To diagnose at scale, every request leaves structured traces, not just the final answer:

• every model and tool call, in order, with its result;
• input, output and cache tokens, and latency per step;
• stop_reason, prompt version, model and flow;
• a request ID that ties everything together, with no personal data in the clear.

On top of that you build behavioral alerts: the share of max_tokens stops, the handoff rate or the cost per transaction moving out of their usual range. Storing full transcripts at debug level “just in case” fills the logs with sensitive data and doesn’t warn you of anything.

The usual traps

• Turning on streaming to lower the cost of a nightly job.
• Adding workers and retries to finish before hitting rate limits.
• Truncating every input to its first tokens without re-evaluating.
• Raising the uptime target when the problem is that the agent resolves worse.
• Relying on someone on the team to email when “something odd” happens.

On the exam

If nobody needs the result right away, look for the batch API, ideally combined with testing a smaller model on a labelled sample. If the request repeats a long, fixed prefix, look for caching. If the dashboard is green and users complain, look for the option that adds quality metrics separate from uptime and alerts on behavioral signals like the stop reason or the handoff rate.