Choose the model with evidence, not by size

A classifier runs on Opus “because it was the prototype’s model”. Now product wants the label in under a second and finance has flagged the monthly cost. In another team, someone moved everything to Haiku to save money and the hard cases started failing. Both mistakes share the same root: choosing the model without evidence.

The three axes and the referee

A model is chosen by balancing capability, speed and cost. The families line up roughly like this: Opus for the most demanding work, Sonnet for everyday work with a good balance, Haiku for what needs the lowest latency and price. But that table is a starting point, not the decision.

The referee is your own evaluation set: real cases from your task, edge cases included, run with your prompts and your data. With it you compare models on accuracy, quality and handling of hard cases, and only then do you weigh cost and latency.

• If the small model matches the large one on your evals for that task, the small one wins: it performs the same, costs less and responds faster.
• If the small model fails only on one segment, don’t throw it out entirely: route that segment.
• The decision holds for that task. Haiku matching Opus on ticket classification says nothing about the agent that reconciles payments.

There are two reasonable ways to start: efficiency-first (start with the small one and move up only if there are capability gaps) or capability-first (start with the strongest one, tune the prompt and move down when the evals allow it). Both end in the same place: measuring.

Mixed workload: route, don’t average

With a mixed workload, for example 85% simple lookups and 15% complex analysis, sending everything to a single model loses on both sides: everything to the large one blows the budget, everything to the small one breaks the hard cases.

Complexity-based routing sends each request to the model that is good enough. The routing signal has to be reliable: the request type the user picks, the category a classifier assigns or the application flow. Confirm with evals that each route meets the bar.

Before switching models, also look at effort: on models that support it, lowering or raising that parameter trades intelligence for latency and cost within the same model, and it is often a better lever than moving to another family.

How to recognize a model mismatch

A model mismatch is diagnosed with evidence: same prompt, same tools, and the complex cases fail on the small model but not on the previous one. If responses end normally and the failure reproduces on retry, it isn’t truncation or transient errors. The answer is to route that segment to the model that handles it, keeping the savings on the rest.

The usual traps

• “The largest model is always the safest choice in production”.
• Downsizing the model without running the evals, or with a public benchmark instead of your data.
• Generalizing an eval: “Haiku matched on this task, so let’s move everything to Haiku”.
• Cutting output tokens hoping it fixes the latency of an oversized model.
• Adding “think harder on difficult questions” instead of routing.
• Blaming “silent model changes” without comparing against the previous version.

In the exam

Check whether the scenario mentions an evaluation set built from real cases: if it exists and shows parity, the answer is to move that workload to the more efficient model. If it describes a mix of simple and complex requests with a clear signal to separate them, look for routing. Rule out options that move everything from one extreme to the other or that decide without measuring.