CCDV-P · Agent Fleet Operations for Software Delivery
When generation outpaces review, AI code review in shadow mode
Since the team started using coding agents, the number of pull requests tripled but the number of people reviewing them didn’t. Reviews turn superficial (“looks fine”), unreviewed PRs pile up, and someone proposes switching on an automated AI review that blocks anything it doesn’t approve. The bottleneck is now verification, and the solution has to be as careful as the problem.
Why review breaks
When generation outpaces review, the risk isn’t that the code is worse: it’s that nobody really looks at it. Human reviews become a formality and defects slip through. The good answers aren’t “review faster” but reorganizing verification into layers, leaving people what only they can do.
Layers of verification
- Deterministic automated verification: compilation, types, linters, tests, security and dependency scanning. It’s the first and cheapest line; a change that fails here never reaches any review.
- Tests the agent writes and runs as part of the task, with a verifiable definition of done.
- AI code review: a model reads the change and flags problems (logic errors, edge cases, security, consistency with conventions).
- Human review focused on what matters: design, risk, intent, what the AI flagged, and sensitive changes.
Introducing AI review: shadow, calibration, gate
Giving a new reviewer, human or AI, blocking power without measuring it is risky: it can let defects through (false negatives) or stop good work (false positives) and people stop trusting it. The safe path:
- Shadow mode: the AI review runs on real PRs and leaves its comments, but doesn’t decide or block. Its findings are stored for comparison.
- Calibration against human review: compare what the AI flagged with what people found and with defects that surfaced later in production. Measure the share of useful findings (precision), the real defects it catches (recall) and the noise it generates. Adjust instructions, scope and thresholds.
- Merge gate: only when calibration shows acceptable performance does it get weight: for example, it blocks on high-severity findings in categories where it proved reliable, and the rest stays as suggestions. Keep measuring, with human sampling of what it approved, to detect drift.
It’s the same pattern as for an evaluation judge: no authority until it has been measured.
Designing human oversight
Human oversight is designed, not assumed: which changes always require a person (security, authentication, migrations, money, infrastructure configuration), what the person sees (the change summary, what the AI flagged, test results and the agent’s scope), who is accountable for approving and how to avoid mechanical approval (rotation, sampled deep reviews, review-time metrics). The agent that wrote the code and the one that reviews it should be independent (another instance, another context, ideally different instructions), not the same one correcting itself in the same conversation.
The usual traps
- Switching on AI review with blocking power on day one.
- Measuring AI review by number of comments instead of usefulness.
- Replacing human review with AI review on sensitive changes.
- The same agent generating and approving in the same conversation.
- Mass human approvals without looking, with no metrics to detect it.
On the exam
To introduce AI review, the correct answer starts in shadow mode, continues with calibration against human review and only then uses it to block. If the problem is review fatigue, strengthen automated verification and reserve people for the sensitive.
Quick question
A practice exam question on this domain. It counts towards your progress.
In a large company, teams share Claude Code prompts, Skills, and MCP servers through chat messages and personal gists. Last quarter, a Skill that was copied from a colleague's gist contained a command that uploaded repository contents to an external service, and another widely shared prompt silently produced incorrect SQL for a reporting team. Nobody could say who owned either artifact or which version was in use.
The platform group wants to encourage reuse across teams while setting a quality and safety bar.
Which governance model is most appropriate?