Documenting to operate, runbook, ADRs and a reference implementation

In three weeks, the project team hands the support agent over to operations. What’s written down is the meeting notes and the team’s chat history. Nobody in operations knows what to do if answers start going wrong on a Saturday, why that model was chosen, or what breaks if the prompt is changed.

What the people who operate it need

A handoff works if the receiving team can operate the system and change it safely. Two documents are worth more than any other for that:

The runbook, to operate:

Section	What it contains
Known failures	what can go wrong and what it looks like (truncated answers, empty retrieval, rate limits)
Detection	which alerts and dashboards exist and what each threshold means
Rollback	how to go back to the previous version of the prompt, the model or the index
Escalation	who to call, in what order, with what data

Architecture decision records (ADRs), to change it: one per significant decision, with the context, the decision, the alternatives ruled out and the assumptions. The assumptions are the most useful part: “we chose this model because 80% of queries are simple” tells the next team when to reopen the decision.

Alongside them go the versioned prompts and the evaluation suite: without them, every change is made blind.

It’s written during the project

An ADR written three months later reconstructs the decision from memory and loses exactly the alternatives and the assumptions. A runbook written in the last week doesn’t include the failures that showed up in the pilot. What works:

• the ADR is written when the decision is made, in a few lines;
• the runbook grows with every incident in the pilot and is tested before the handoff, with someone from operations following the steps;
• the handoff includes a period when both teams operate together.

Guiding many teams

Another case: a shared Claude platform adopted by ten product teams. Some launch without evaluating, others log account data. The platform team has three people. A longer wiki page changes nothing, and reviewing every implementation doesn’t scale.

What scales is a reference implementation: an example service and templates with the required controls already built in (evaluation before launch, logs without personal data, rollback, cost limits), versioned and with an owner. Every team starts from there, and a short checklist provides the evidence the risk function asks for. The right thing becomes the easiest path.

The usual traps

• Exporting the chat history and meeting notes “so they can search there”.
• Prioritizing an executive slide deck over the runbook.
• Handing over the list of every prompt variant tried, without their results.
• Leaving all the documentation for the last week.
• Asking every lead to confirm they’ve read the wiki, or letting each team choose its own practices.

On the exam

If the scenario is a handoff with little documentation and you have to prioritize, look for the runbook (failures, alerts, rollback, escalation) and ADRs with the rationale and assumptions, together with versioned prompts and evaluations. If many teams are implementing unevenly, look for the reference implementation with templates and required controls, versioned and with an owner.