claudio.cool
CCDV-PLessons

CCDV-P · Production Integration, Reliability, and Knowledge Grounding

Scale and throughput, finding the binding constraint

⏱ 3 min readDraft

A document-analysis platform grew tenfold in a quarter. Users start seeing 429 errors at peak hours. The team proposes “ask for more quota”, but looking at the data they find that one large customer consumes most of the tokens, half the work isn’t interactive, and the limits being exhausted are input tokens, not requests per minute.

Which constraint is binding

Before acting, identify which limit is running out. API limits are measured on several dimensions (requests per minute, input tokens per minute, output tokens per minute) and each calls for a different move:

Constraint Signal Lever
Requests per minute many small requests batch up, avoid redundant calls, cache results
Input tokens per minute big requests, lots of repeated context trim context and use prompt caching (for most models, tokens read from cache don’t count against this limit; confirm in the rate-limit docs)
Output tokens per minute long responses bound max_tokens, ask for shorter formats
Economics cost per document outside margin batch processing, route to a smaller model

Asking for more quota is right when legitimate demand exceeds the limit even after optimizing; it doesn’t replace design.

Queues and backpressure

A system with no queue takes a spike and turns it into 429s for users. A queue between the app and the API smooths the work. Backpressure is the signal the system sends back to whoever produces the work when it can’t keep up: reject with a clear message, slow admission, prioritize interactive over deferrable. Without backpressure the queue grows unbounded, latencies explode and the system answers requests the user already abandoned.

Batches

Message Batches processes requests asynchronously at a discount (50% off standard pricing at the time of writing) with a window of up to 24 hours, and caps on requests and size per batch. It fits non-interactive work: bulk classification, evaluations, reprocessing. A batch pipeline is designed with a custom ID per request (to reconcile results), handling for individual requests that fail and retrying only those. It’s the economics and throughput lever for deferrable work, and it frees real-time quota for what’s truly interactive. It doesn’t fit when someone is waiting for the answer.

Multi-tenant: quotas and isolation

When several customers share your account, one can exhaust everyone’s limit. Multi-tenant isolation combines per-customer internal quotas (tokens and requests), separate queues or priorities, and splitting workloads across workspaces or keys when you need independent limits and spend. That way a noisy customer throttles itself, not the others.

Capacity planning

Capacity planning starts from data: requests and tokens per hour at peak, input/output ratio, cache hit rate, deferrable share. From that you forecast the limit you need with headroom and decide what you request as more quota, what moves to batches and what you optimize. Measure again after each prompt or model change, because tokens per request change.

The usual traps

  • Asking for more quota without checking which limit is exhausted or who consumes it.
  • Retrying harder instead of queueing with backpressure.
  • Sending work that needs an immediate answer to batches, or leaving deferrable work in real time.
  • A shared quota with no per-customer cap.
  • Sizing for the average instead of the peak.

On the exam

Constraint first (requests, tokens or economics?), lever second. Deferrable work, batches; spikes, queue with backpressure; one customer starving the rest, per-tenant quotas.

Quick question

A practice exam question on this domain. It counts towards your progress.

A bank's internal policy assistant retrieves passages from a policy repository and writes answers for relationship managers. An audit found two problems: some answers include a citation to a policy document that does not contain the claim made, and when the retrieved passages are only loosely related to the question, the assistant still writes a confident answer instead of saying it could not find the policy.

Managers rely on these answers for client conversations, and the compliance team wants measurable grounding rather than a general instruction to be careful.

Which TWO practices best improve grounding? (Select TWO.)

Select all that apply.

Practise this domain in a practice exam →