Expectations and SLAs, no 100%, targets against the baseline and latency by percentile

The sponsor of a document-classification project said it at the kickoff meeting: “we launch when it’s 100% accurate”. Two weeks later, sales asks for every interaction with the new agent to take under two seconds, even though some tasks chain several tools and take between eight and twenty. Accepting both targets to avoid friction is the surest way for the project to fail.

Accuracy is agreed against the baseline

Nobody can promise 100% accuracy from a probabilistic system, and there’s no need to: the current process isn’t perfect either. The target is anchored to the baseline:

• measure the current process: if people classify 95% correctly today, that’s the point of comparison;
• agree the target against that number, by document type if errors don’t all cost the same;
• define an escalation path: low-confidence or poor-quality cases, such as an illegible scan, go to a person instead of forcing an answer;
• write it into the acceptance criteria, together with the test set it’s measured against.

Refusing to commit to any number because “the model is probabilistic” doesn’t work either: it leaves the business with no way to evaluate the project. And the provider’s public benchmarks aren’t a target; they measure other tasks on other data.

Latency: by flow and by percentile

A service level agreement (SLA) with a single average hides the tail: a two-second average can coexist with 10% of requests taking twenty. It’s defined by flow type and by p95:

Flow	What the user needs	How the SLA is expressed
Simple question	for it to feel immediate	time to first token at p95, with streaming
Multi-step task	to know it’s progressing	completion at p95, with step-by-step visible progress
Report or bulk job	to have it by a set time	asynchronous completion with a notification when it’s ready

Forcing everything onto the smallest model to hit two seconds sacrifices quality without asking anyone.

Communicating the trade-off

When a technical decision changes what the business asked for, it’s explained in the business’s terms and the owner of the outcome decides:

• your own data: accuracy measured on the company’s set, cost per transaction, hours spent correcting errors;
• options with a recommendation: for example, “the report is processed overnight, it’s ready by 8, it costs less; if you need it sooner, this is what changes”;
• no deciding quietly, and no technical note so the business can “choose on its own”.

A feedback loop

Expectations are aligned once and sustained with data: a pilot before the full launch, regular reviews with stakeholders against the agreed targets, real examples of hits and misses, and a process to prioritize change requests.

The usual traps

• Accepting 100% to keep the sponsor’s support and “adding review later”.
• Promising the accuracy from a public benchmark.
• A single average for every interaction, whether tight or generous.
• Meeting the SLA by moving everything to the smallest model without evaluating.
• Switching to batch quietly because “they read the report in the morning anyway”.

On the exam

If someone expects 100%, look for the option that agrees a target against the current process’s baseline, with escalation for doubtful cases, in writing. If they ask for a single time for everything, look for SLAs by flow and by p95, with streaming or visible progress. If a decision has to be justified to an executive, look for your own numbers in business terms and a recommendation.