CCDV-P · Advanced Debugging and Incident Response
Incidents, mitigate first and turn the learning into evaluations
On a Tuesday afternoon, support gets complaints: the assistant answers with another customer’s data, or doesn’t answer at all, or gives the same answer to everything. There was a prompt deploy in the morning and a tool update at noon. Someone proposes finding the root cause before touching anything. That’s the classic mistake: during an incident, you stop the damage first.
The order of an incident
Incident response has phases, done in order:
- Triage: how severe is it, who’s affected, is data exposed, can money be lost? Assign a severity and a coordinator. An assistant showing another customer’s data is top severity.
- Mitigation: reduce the damage now, before understanding everything. Options: rollback to the previous prompt or model, switch a feature or tool off with a kill switch, drop to a reduced version, redirect to humans. The fastest mitigation is almost always reverting the last change, which is why changes ship versioned and progressively.
- Diagnosis: with the damage contained, find the origin with traces and cohorts (see the diagnosis lesson). Compare what changed: prompt, model, tools, data, traffic.
- Resolution: a lasting fix, verified in a test environment and rolled out in stages.
- Communication, throughout, not at the end: who notifies whom (users, customers, internal teams, and regulators if personal data is exposed), how often, and what is said and what is still unknown. You don’t speculate about cause in external communications.
If data is exposed or actions are irreversible, you contain first: revoke access, halt the agent. Silencing alerts or “waiting to see” aren’t mitigations.
Quality regressions
An incident isn’t always an outage: it can be a quality regression with no errors. The response has the same structure: confirm with an evaluation or a cohort that there really is a regression, revert if the latest change is suspect, and diagnose afterwards. Rolling a model back to a previous version requires that version to still be available, another reason to pin versions.
Turning the incident into learning
An incident that ends when it “works again” will recur. A blameless postmortem reconstructs the timeline and the system factors that allowed it; it doesn’t hunt for a culprit. It yields concrete changes:
- Regression tests: the failing case joins the evaluation suite with its expected output, so a future change that reintroduces it won’t pass the gate.
- Monitoring coverage: what signal would have caught this earlier? Add that metric or alert (for example, a check that the response contains only the session customer’s data).
- Controls: if the incident revealed an unsafe path, a deterministic guardrail that closes it.
- Process: more gradual rollouts, review of tool changes, an updated runbook.
Each action has an owner and a date, and completion is verified.
The usual traps
- Hunting the root cause before mitigating.
- Fixing the symptom and not adding a test that will catch it next time.
- Communicating late, or speculating about cause externally.
- A postmortem that blames a person or concludes “be more careful”.
- Closing the incident with no change to evaluation or monitoring.
On the exam
The good sequence is contain, diagnose, resolve, communicate and learn. For “what first”, the answer is the fastest reversible mitigation (rollback, kill switch). For “what next”, turn the case into a regression test, monitoring and a process action.
Quick question
A practice exam question on this domain. It counts towards your progress.
After a migration to a newer model snapshot, the refusal rate for a category of billing questions rose from 0.5% to 9%. The change went out to 100% of traffic on day one. No alert covered refusals, the evaluation suite had no queries from that category, and the issue was found on day three through customer complaints. The postmortem meeting is about to start.
Some participants want the document to name the engineer who approved the rollout, while others say that nothing could have been done because model behavior changes cannot be fully predicted.
Which set of findings and actions would make the postmortem most useful?