CCDV-P · Production Integration, Reliability, and Knowledge Grounding
Reliability, retries, circuit breakers and fallbacks that tell causes apart
A ticket-classification service wraps every Claude call in a retry that repeats up to five times on any failure. One Friday the provider degrades and the retry multiplies traffic exactly when that hurts most; meanwhile a batch of calls fails on an invalid parameter and is retried anyway, five times each, with no chance of success. The problem isn’t the number of retries: it’s that nobody distinguishes why each call ended.
First, classify the cause
The stop reason and the error code say what happened. Each cause needs a different response:
| What happened | Right response | Why not retry blindly |
|---|---|---|
Rate limit (HTTP 429, with a retry-after header), overload (529) and 5xx errors |
exponential backoff retry, with jitter and a cap | transient; retrying is correct |
stop_reason: max_tokens |
raise the output budget or ask for a shorter answer | repeating the same call reproduces the cut-off and burns tokens |
| Context-window stop | trim or compact the input | retrying doesn’t change the size |
stop_reason: refusal or a declining answer |
a safety-handling path: controlled response, escalation, no automatic retry | repeats the refusal and adds cost; sometimes retrying would be inappropriate |
| 400 (invalid request) | fix the request | never fixes itself |
| 401 or 403 | fix credentials or permissions | same |
Anthropic’s SDKs already retry some transient errors by default (with a configurable count). If you also retry in your own layer, retries multiply: assign a single layer to own them.
Circuit breaker and graceful degradation
When the provider is degraded, retrying harder makes it worse (a retry storm). A circuit breaker stops calls after a failure rate, lets a few probes through and closes again on recovery. While it’s open, the system applies graceful degradation: a cached response, a reduced version of the feature, a queue to process later or a clear message to the user. You decide in advance what gets sacrificed first.
Fallbacks across models, regions and platforms
A fallback to another model, region or platform (say, Anthropic’s API and a cloud such as Amazon Bedrock or Google Vertex AI) adds resilience but also differences:
- Feature parity: not every feature or beta tool is available on every platform or at the same time. A fallback that depends on a missing feature fails exactly when you need it. Check each platform’s feature matrix and exercise the fallback path with real or test traffic.
- Behavior: another model may answer differently; the fallback goes through the same evaluation as the primary.
- Quotas and data: the alternative region or platform has its own limits and data-residency requirements.
- The fallback fires for the right cause: switch to the secondary for an outage or saturation, not for a safety refusal (it would refuse the same, or worse).
The usual traps
- One retry path for every error.
- Raising retries and the backoff window “so it eventually works”.
- Retrying 400s or refusals as if they were transient.
- Nested retries (SDK, client, queue) that multiply.
- A permanent fallback to a cheaper model “because it’s answering”.
- Falling back to a platform that lacks a feature the flow needs.
On the exam
If the scenario lumps causes under one handler, pick the option that branches on stop reason and error code. If a fallback “doesn’t work” on the alternative platform, suspect feature parity. If the provider is down, the answer is circuit breaker and degradation, not more retries.
Quick question
A practice exam question on this domain. It counts towards your progress.
A document-processing service runs 200 concurrent workers that call the Messages API. After a traffic spike the API began returning 429 rate-limit errors. Each worker retries immediately after a fixed one-second pause, up to ten times. Instead of recovering, the service has stayed at a high error rate for twenty minutes, and workers now log "retries exhausted" for many documents.
The responses include a retry-after header, which the code ignores. The on-call engineer notes that error rates were normal an hour before the spike and that the queue of pending documents is growing.
Which change is most likely to restore stable operation?