CCDV-P · Cost and Latency Optimization at Scale
Latency engineering, first token or total, and what to move
A voice assistant for a call center has a goal: the person should start hearing the answer in under a second. The team measures an average latency of six seconds and wonders whether it needs a faster model. Before changing anything, you need to know what that latency is made of and which of the two latencies matters.
First token or total
They’re two different numbers, optimized with different levers:
- Time to first token (TTFT): how long until something appears. It’s what the person waiting in a chat or voice interface perceives. It depends on the size of the prompt to process, queueing, the model and any prior work (retrieval, tools).
- Total latency: until the full response finishes. It depends on how many output tokens are generated and how fast, and on an agent’s steps.
A goal of “answer fast” translates into one of the two. In a streaming chat, TTFT matters; in a process that consumes the full output (a JSON for another system), total matters, and streaming doesn’t reduce it.
The latency profile
You break a typical request down with a trace: time in retrieval, in each model call (input and output separately), in each tool, in waits and retries. You see which stretch dominates. A five-turn sequential agent calling two independent tools one after another has a different problem from a 50,000-token prompt. Look at the high percentile (p95, p99), not just the average: users suffer the tail.
The levers
| Lever | What it improves | Detail |
|---|---|---|
| Streaming | perceived TTFT | shows the response as it’s generated; doesn’t reduce total latency |
| Parallel tool execution | agent total latency | if tools are independent, call them in the same turn and run them at once instead of one per turn |
| Prompt caching with cache checkpoints | TTFT and cost | a long repeated prefix is processed faster when read from cache; put the checkpoint at the end of the stable stretch |
| Effort level | total latency | less reasoning and shorter outputs on simple tasks; raise it only where quality needs it |
| Smaller model | both | only if evaluation shows quality suffices |
| Shorter output | total latency | ask for what’s needed, bound max_tokens, compact formats |
| Fewer steps | total latency | move the predictable to a workflow, cut round trips |
To pick the lever, read the trace. If the prompt is long and repeats, cache. If the wait is a long output, effort, format and model. If tools are needlessly chained, parallelism. If the target is first token: streaming and cache; if total: output and steps.
Check the current names and parameters of these features (effort, caching, parallel tools) in the API documentation before implementing: they vary across models.
The usual traps
- Optimizing the average when the problem is p95.
- Streaming to cut the latency of an internal batch process.
- Changing the model before looking at the trace.
- Raising effort everywhere “for quality”, paying latency where it isn’t needed.
- A cached prefix that changes between calls, gaining nothing.
On the exam
First identify which latency is asked for (first token or total) and where the time goes; then choose the lever that attacks that stretch.
Quick question
A practice exam question on this domain. It counts towards your progress.
A code-review pipeline runs three Claude calls per pull request: it first labels the change type, then extracts the list of modified public APIs, and finally reasons about concurrency and security risks in the diff. All three calls use the highest effort setting with a large thinking budget. The p95 latency is 41 seconds against a target of 20 seconds.
Traces show that the first two calls take most of the time while producing short, mechanical outputs, and the team's evaluation suite shows their accuracy does not change between effort levels. The third call benefits clearly from deep reasoning.
Which action best meets the latency target while protecting quality?