Start from the production decision
The eval that changed our routing used 84 prompts taken from real task language. Each response had to classify intent, domain, output format, and reasoning mode. Strict score required the complete structured answer to match. Partial credit was recorded only so we could see which field broke.
That design is more informative than a general ranking because an error has a known consequence. Misclassifying a dashboard request as a chat request changes the tools, the latency, and the evidence path. The eval measures the decision the router must make, on language the users actually type.
Repeat the run and publish the denominator
Each model ran the same set three times. Repetition exposed unstable classifications that a single pass would hide. We reported accuracy, severe-error rate, latency, and cost per accepted classification rather than one blended score, which can make a cheap failure look efficient.
| Route | Strict accuracy (%) |
|---|---|
| Local model | 96.7 |
| Higher-price model | 94.6 |
A local model reached 96.7 percent on this task. One more expensive model reached 94.6 percent. The result does not mean local models win in general. It means that on this bounded job, price and performance were weakly related, so buying the better-known name would have been a more expensive way to be slightly worse.
Inspect a surprising score before you believe it
A zero, or an unexpectedly low number, is often a broken parser, a schema mismatch, truncated output, or a label set that drifted. Before rejecting a model, read the raw responses and run a known-good fixture through the scorer. A harness can fail as confidently as a model, and the failure will still look like a measurement.
Mutation tests catch that class of error. Change one expected label, remove one required field, and feed malformed output. The scorer should fail for the intended reasons and name which contract was violated. If it cannot do that, the script is printing percentages rather than evaluating a contract.
Route on the cost of an accepted result
The cheapest call is not always the cheapest accepted result. Review time, retries, escalation, and severe errors belong in the cost. A model with a lower token price loses if its failures create expensive human inspection, which is how a program that appears to save money on models can spend the savings on review.
A compact routing table needs five columns: task class, quality floor, severe-error ceiling, cost per accepted result, and escalation rule. Re-run the eval when prompts, labels, or source data change. Last quarter's winner is a hypothesis until it is measured again.