CCDV-P · Advanced Agent Architecture and Orchestration
Durable execution, checkpoints, sessions and resumability
A data-migration agent runs for about four hours. At hour three the container restarts for a deploy and the agent starts from scratch: it repeats calls it already made, rewrites records and spends the whole cost again. And when the team upgrades the model midweek, in-flight tasks behave differently when resumed.
The idea: state lives outside the process
Durable execution begins by accepting that the agent process will die. So what matters is not in process memory but in an external store:
- the conversation history (messages, tool results);
- the task state: which steps are done, which remain, which identifiers were already written;
- intermediate artifacts (files, partial results).
Session persistence stores that so you can pick up again. The Agent SDK and Claude Code can resume a session by its identifier and fork it to try another path without losing the original; if you build your own loop, you persist the message history yourself.
Checkpoints: when and what
A checkpoint is a consistent state you can resume from. Place them after every step with an external effect and before an expensive one. A good checkpoint has three properties:
- Useful granularity: not one per token and not one at the very end; the cost of redoing what was lost must be tolerable.
- Atomic write: state and progress are saved together, or not at all.
- Idempotent steps: if you resume from a checkpoint, repeating the last step must not duplicate its effect. An idempotency key per operation prevents the second charge, second email, second row.
Resumability isn’t “start over with the same prompt”: it’s reading the state, rebuilding the context you need and continuing from the step after the last confirmed one.
Memory: what to keep and what not
Memory architecture distinguishes:
| Type | What it holds | Lifetime |
|---|---|---|
| Working context | what’s in the window now | one session |
| Task state | progress and decisions | until the task ends |
| Persistent memory | reusable facts and preferences | across sessions |
You store data and decisions, not internal reasoning. Whatever can be queried again (a state in the database, a file in the repo) is better re-read than trusted from a stale copy in context.
Model change with tasks in flight
An agent resumed with a different model can decide differently from the same history. For long tasks, pin the model version for the task’s lifetime, or resume only at step boundaries and evaluate behavior on the new model before migrating live tasks. A model’s reasoning blocks are not a portable contract: what must survive is explicit state.
The usual traps
- Keeping everything in process memory or in the context.
- Restarting from scratch “because it’s simpler” when external effects already happened.
- Resuming without idempotency keys.
- Checkpoints only at the end of the task.
- Switching models mid-task without testing the resume path.
On the exam
For a long agent that gets interrupted, look for external state, checkpoints after effects and idempotent steps. Discard “raise the timeout”, “retry everything from the start” and “ask the model to remember where it was”.
Quick question
A practice exam question on this domain. It counts towards your progress.
A long-running onboarding agent works through three phases per customer: data collection, validation, and configuration. To keep it focused, the team rewrites the system prompt at the start of each phase so that it contains only that phase's instructions. Since this change, the cache read rate has dropped to near zero after each phase change, and input cost per customer has roughly doubled.
The conversation is long at that point, with a large shared history and a stable set of tool definitions.
How should the team deliver phase-specific instructions without losing the cache benefit?