Handoff is not the end, an owner, scheduled evaluations and thresholds that trigger iteration

The shopping assistant launched with excellent evaluation results. Six months later it recommends discontinued products and doesn’t know half of the new categories. Nobody touched the code. The project team thought operations was handling the evaluations; operations thought they belonged to the project team. The biggest campaign of the year is three weeks away.

The phases and who owns each one

A solution’s lifecycle doesn’t end at handoff:

Phase	What is delivered	Owner
Discovery	problem, metric, baseline, requirements	architect with the business
Design	architecture, ADRs, evaluation suite	architect and project team
Pilot and launch	results against acceptance criteria, tested runbook	project team
Handoff	shared operation for a while, documentation	both teams
Monitoring	quality, cost and operations dashboards, alerts	system owner
Iteration	changes evaluated and released in a controlled way	system owner

The row that’s usually missing is the last one. Without a system owner with a name and allocated time, evaluations stop running the day the project team leaves.

Why it degrades without anyone touching it

An LLM system drifts even when the code is frozen:

• the data changes: catalogue, policies, documents that feed retrieval;
• the users change: new questions, new ways of asking, seasons;
• the environment changes: models that are retired and need migrating, internal APIs that change format;
• the system itself changes: someone tweaks the prompt for one case and breaks another.

That’s why the launch evaluation expires: it measured the system against the data of that moment.

What has to exist after handoff

1. An owner responsible for quality, cost and changes.
2. Scheduled evaluation: the suite runs on a fixed calendar over a dataset refreshed with recent, anonymized production cases.
3. Trigger thresholds: agreed values that, when crossed, open an iteration without waiting for someone to complain.
4. A change process: every prompt, model or index change passes the suite before it ships and is rolled out gradually.

Metric	Example threshold	Action
Accuracy on the scheduled suite	drops more than 3 points	review retrieval and new data
Handoff rate to people	rises 20% in two weeks	analyze handed-off cases by topic
Cost per transaction	exceeds the agreed budget	review caching, routing and output length
Model retirement notice	date announced	evaluate the replacement with the suite and plan the migration

The usual traps

• Considering the project finished at handoff.
• Calling the project team back when complaints pass a certain level: it reacts late and leaves no owner.
• Freezing the data (the catalogue, the policies) so the assistant doesn’t go out of date.
• Fine-tuning on the new catalogue when what’s missing is updating retrieval and measuring.
• Evaluations “when there’s time”, on the same dataset used at launch.

On the exam

If a system degraded months after launch and nobody saw it coming, look for the option that assigns an owner, schedules evaluations on refreshed data and defines thresholds that trigger iteration. Rule out the ones that react to complaints, freeze the business or change the model without measuring.