claudio.cool
CCDV-PLessons

CCDV-P · Cost and Latency Optimization at Scale

Cost architecture, model, route, cache and cap

⏱ 3 min readDraft

A company’s Claude spend grew 4x in two months while usage only doubled. Finance asks for a forecast and a cap. The team has one number: the monthly bill. Optimizing without a breakdown is guessing.

A cost model

A cost model decomposes spend into what drives it. Per call: input tokens, output tokens, cache read and write tokens, times model price. Per task: calls per task (in an agent, turns and subagents) times cost per call. With usage telemetry you compute cost per feature, per tenant, per prompt version and per completed task. Cost per completed task is the useful unit: an option that’s cheaper per call can be dearer per task if it retries or fails more.

With that you forecast: expected volume times unit cost, with scenarios (growth, a large new customer, a prompt change) and a margin. Prices and models change: the model is updated with every price or version change.

Routing and cascades

Not every request needs the most capable model. In model routing a cheap classifier (or rules) sends each request to the right model in the family: smaller ones for classification, extraction and simple tasks; larger ones for complex reasoning and multi-step agents. In a model cascade you try the cheap model first and escalate to the large one if verification fails (an invalid schema, low confidence measured with verifiable signals, a check that doesn’t pass). It’s justified with data: evaluation tells you in which categories the small model suffices, and savings are measured net of escalation cost.

Cache: strategy and diagnostics

Prompt caching charges more for the first write and much less for reads (a read costs a fraction of the input price; the write carries a premium that depends on the TTL). It pays off when the same long prefix repeats before it expires. The strategy: fixed, long content first (tools, system prompt, reference documents) and variable content last, with cache breakpoints where each stable stretch ends.

Cache diagnostics looks at cache_read_input_tokens and cache_creation_input_tokens: many writes and few reads mean the prefix changes between calls (a timestamp, a user ID, tools in a different order) or the TTL expires between requests. You fix it by stabilizing the prefix or adjusting the TTL, not by changing models.

Batch or real time

The batch API offers a significant discount for work that can wait hours. The rule: anything with no person waiting (evaluations, bulk classification, reprocessing) goes to batch; interactive work doesn’t. And they combine: a batch with a cached prefix stacks both savings.

Per-tenant budget

So that one customer doesn’t consume everyone else’s margin, the per-tenant budget is enforced in your layer: you measure each tenant’s spend in tokens and cost, and apply limits (soft: alert and degrade to a cheaper model; hard: cut off or queue). Console workspaces and separate keys also let you attribute spend and set spend limits at the Anthropic account level. Enforcement has to live outside the prompt.

The usual traps

  • Optimizing from the total bill, with no breakdown by feature and tenant.
  • Moving everything to the smallest model without evaluating quality.
  • Caching a prefix that changes on every call.
  • Real time for deferrable work.
  • A per-tenant budget looked at only at month end.

On the exam

Breakdown first, lever second: route if the task mix is heterogeneous, cache if the prefix repeats, batch if it can wait, per-tenant cap if one customer dominates spend.

Quick question

A practice exam question on this domain. It counts towards your progress.

A compliance team classifies about two million archived documents each night into retention categories using Claude. The job is triggered by a scheduler at midnight, and the results are read by an analyst at nine the next morning. Today it calls the Messages API synchronously with a worker pool, and the monthly bill has become the largest line item of the project.

Nobody needs an individual result in less than a few hours, and a failed document can simply be retried the following night.

Which change gives the largest cost reduction without hurting the business need?

Practise this domain in a practice exam →