Comparing versions without fooling yourself, offline baseline, A/B tests and guardrail metrics

On Friday a new prompt went out for the sales assistant, to all traffic. On Monday conversion was down four points. That same weekend a competitor launched a promotion and it rained in the three largest cities. Nobody can say whether the prompt made anything worse, better or changed nothing at all: the experiment has nothing to compare against.

Two stages before deciding

Every change (prompt, model, retrieval parameters, tools) goes through two filters, in this order:

1. Offline, against the baseline. The new version runs on the same regression suite as the current version: the representative dataset from the previous lesson, with its edge cases. The pass criterion is set in advance: no regression in any critical segment and an improvement in what you were after. It’s cheap, fast and stops most mistakes before they reach a user.
2. Live, with an A/B test. Whatever passed the offline filter is tested with real traffic split at random between the two versions and at the same time, with the success metric and the sample size defined beforehand.

Running them simultaneously is what makes the result usable. Comparing “the week before against the week after” mixes the change with seasonality, campaigns, the weather and anything else that happened in between. With both versions running at once over randomly assigned users, the only thing that distinguishes the groups is the change.

Reading the result

What you see	What to do
Small lift after a few days, far from the planned sample size	Keep going to the agreed size: it may be noise
The primary metric improves, but cost per conversation or escalations get worse	Weigh the guardrail metrics before launching
Better on average, worse in one segment	Look by segment; it may make sense to launch only where it wins
Stable lift, guardrails within range, even segments	Roll out gradually and keep measuring

Guardrail metrics are the ones you’re not trying to improve but can’t break: cost, p95 latency, over-refusal rate, safety incidents, complaints. They’re defined together with the primary metric, not after seeing the numbers.

Iterate one change at a time

• One variable at a time: if you change the prompt and the model together and it improves, you don’t know which one did it.
• Versioned prompts, like code, with a record of what was tested, against what and with what result.
• The suite grows with production: every real failure that shows up is anonymized and added to the suite, so the next version doesn’t repeat it.
• Gradual rollout with fast rollback: first a small share of traffic, then the rest.

The usual traps

• Shipping to all traffic and comparing before and after, “adjusting by hand” for known events.
• Stopping the A/B after two days because the primary metric is already positive.
• Automatically rejecting the variant because cost goes up a little, without weighing how much it improves.
• Deciding by reading twenty transcripts of each version and voting on which one “sounds better”.
• Shipping to everyone and reverting if complaints exceed a threshold: users become the test bench.
• Rerunning the test on another model when what’s missing is sample size.

On the exam

If a change was shipped with no comparison and nobody knows what happened, look for the two-stage process: offline suite against the previous version, then an A/B with random, simultaneous traffic and a metric fixed in advance. If the A/B is already running and shows a small lift on a small sample, the answer is to complete the agreed size and look at guardrails and segments before deciding. Rule out the options that decide on noise or impressions.