The tradeoff is real but often measured wrong
Switching your agent from a larger, slower model to a smaller, faster one is a common optimization. The cost savings and latency reduction are measurable immediately. What's harder to measure is whether output quality dropped, and if so, by how much, and on what types of inputs.
Most teams making this switch do a manual review of a handful of outputs. The outputs look comparable. The switch ships. Weeks later there's a pattern in user feedback that's hard to attribute until someone goes back and looks at the eval scores, which don't exist, because no one was running them continuously.
The problem isn't the model switch. Smaller models can absolutely produce acceptable output for many tasks. The problem is that "looks comparable to me when I read five examples" is not a measurement. It's an impression. And impressions don't catch the failure modes that only appear on specific input types or that accumulate across many sessions.
Latency as a first-class eval metric
Latency is often treated as an infrastructure metric rather than a quality metric. Error rates, p50/p95/p99 latency, and trace counts go into the monitoring dashboard. Eval scores go into the eval dashboard. They're conceptually separate.
This separation creates a problem when making model optimization decisions. You have quality data and you have latency data but you don't have the tradeoff curve: at this latency level, what quality score are we getting? And at that quality score, what latency does it correspond to?
Capturing latency inside your trace data, at the individual LLM call level, is the change that enables this analysis. Orq captures p50 and p95 latency per trace as part of the standard trace payload, alongside the eval scores for that trace. When you want to evaluate whether a model swap is worth making, you can run both the latency comparison and the quality comparison against the same set of inputs.
Measuring the tradeoff across your actual input distribution
A latency-quality tradeoff measured on your happy-path test cases is not representative of the tradeoff across your full input distribution. Smaller models often show the biggest quality degradation on the harder inputs: longer contexts, more complex reasoning tasks, or specialized domain knowledge.
If your test corpus only includes simple inputs, a smaller model will look fine. Your P95 latency drops from 3.2 seconds to 1.8 seconds and your grounding score barely moves. Ship it. Then production arrives, and 15% of real users send queries that are significantly harder than your test corpus, and those are exactly the users where the smaller model performs meaningfully worse.
This is why the evaluation has to run against a corpus that represents your actual input distribution, not just the inputs you expect and like. Pull inputs from your production traces. Include the long ones, the ambiguous ones, the ones that required multiple sentence-level inferences. Run both models against that corpus and compare the quality scores across the input length distribution, not just in aggregate.
Setting a quality floor before optimizing for speed
The way to avoid the quality sacrifice disguised as a performance win is to set a quality floor before you start the latency optimization. Pick your most important quality metric, whatever dimension matters most for your feature, and define the minimum acceptable score. "Grounding score must not drop below 3.8 (on our 1-5 rubric) at any percentile of the input distribution." That's your constraint.
Now you're free to optimize for latency within that constraint. Try the smaller model, try quantized versions, try reducing max output tokens. Any change that keeps grounding above 3.8 is acceptable. Any change that drops it below 3.8 is not, regardless of how much latency it saves.
The floor is not a final target. It's a constraint on the optimization process. Once you've made the latency win while respecting the constraint, you can revisit the floor. Maybe you want to push quality higher on a future iteration. But having the floor explicit means you never accidentally sacrifice quality for a performance gain you didn't intend to make.
Where latency evals get complicated: multi-step agents
For agents that chain multiple LLM calls, latency isn't a single measurement. It's the sum of individual step latencies plus any waiting on tool calls, retries, or context-building. A model swap on step one might improve end-to-end latency or make it worse, depending on whether step one is the bottleneck and whether the swap introduces any retry patterns.
This is why step-level trace data matters. When you're looking at an agent where end-to-end latency is 8 seconds, you need to know which steps account for that 8 seconds before deciding where to optimize. In our own instrumentation, we've seen agents where the multi-step orchestration cost (assembling context, formatting tool results, chaining prompts) was larger than the sum of the actual LLM call latencies. Optimizing the model choice in that situation has limited impact; the bottleneck is elsewhere.
Per-step latency tracking, combined with per-step quality scoring where each step has a distinct output worth evaluating, gives you the data to make this analysis. Without it, end-to-end latency figures obscure more than they reveal.
What latency metrics alone will miss
A final note on what this approach doesn't cover: user-perceived quality has dimensions that latency and eval scores together still miss. A response that arrives in 1.5 seconds but requires the user to re-read it three times isn't actually faster from the user's perspective. A latency optimization that shifts work from the agent to the user, by producing outputs that require more interpretation or correction, is not a real win.
This is harder to measure systematically. The proxy we've found most useful is a "clarity" rubric dimension alongside grounding and task completion: does the output communicate its content without requiring interpretation or re-processing? It correlates weakly with traditional quality metrics but captures some of the user-experienced quality that latency measurements miss.
The point isn't to measure everything. It's to avoid the failure mode where you optimize for a metric that's easy to measure (latency, cost) while the metric that actually determines user satisfaction (quality) goes unmeasured and silently degrades.
Putting the tradeoff in writing before you make the call
One operational practice that helps: before making any model optimization decision, document the tradeoff you're accepting. "We are switching to Model B, which saves 1.1 seconds on P95 latency. Our grounding score distribution shows a 0.2 point mean drop on long documents, which stays above our threshold of 3.5. We accept this tradeoff."
Writing this down forces the analysis to actually happen. It also creates an audit trail: if a regression surfaces three weeks after the model switch, you can check whether the tradeoff you accepted was larger than you estimated, or whether something else changed alongside the model switch. Without this record, root cause analysis is significantly harder.
The Orq eval run API returns per-input score breakdowns that you can include in this documentation. For teams using GitHub Actions or similar CI systems, we've seen this kind of tradeoff analysis added as a comment to the pull request that makes the model configuration change. It makes the decision visible to the rest of the team and creates a record that survives personnel changes.
None of this is bureaucracy for its own sake. The goal is to make the cost of a speed optimization explicit before you accept it, rather than discovering the cost after users have been experiencing it for two weeks.