The false confidence of a large eval suite
Teams that invest in eval infrastructure often end up with a suite of eight or ten metrics covering a wide range of quality dimensions. The suite runs on every deploy. The dashboard shows all green. Everyone feels good about the reliability of their agent.
Then a regression slips through. A user finds it. Someone goes back to the eval scores and finds that the relevant metric was at 3.6 on a 1-5 scale for the last three deploys. The threshold was set at 3.5. The gate passed by 0.1 points while the agent was producing noticeably worse outputs for a specific input class.
A large suite with poorly calibrated thresholds provides false confidence, not real coverage. The number of metrics is not the quality measure. The question is whether the suite catches the regressions that would actually hurt users, and whether the thresholds are calibrated to detect them when they occur.
How suites stagnate
The typical lifecycle goes like this: the team ships the feature, writes an eval suite that reflects the quality concerns they had at launch, and moves on to other things. The suite runs quietly in CI. The thresholds were set once and never revisited. The rubrics were written once and never updated to reflect what the team learned from production usage.
Months later, the feature has matured. Users are using it in ways the team didn't originally anticipate. The failure modes have shifted. The original eval suite covers launch-time concerns well but has blind spots in the areas where users are actually encountering problems. The suite is still running, the metrics are all green, and the team has lost visibility into the regressions that actually matter.
Eval suite maintenance is not a one-time setup task. It's an ongoing process, analogous to test suite maintenance in software development. When a test suite stops catching the bugs that matter, engineers update it. The same discipline applies to eval suites.
The metric that doesn't fire on failures is the most dangerous one
When reviewing your eval suite, the question to ask about each metric is not "does it produce scores" but "would it have caught the last real regression we experienced." If the answer is no, the metric is either miscalibrated or measuring the wrong thing.
Running a retrospective on known regression incidents is the most useful eval calibration exercise. Take an incident where quality dropped, find the traces from that period, run your current eval suite against them, and check whether any of your metrics would have blocked the deploy that introduced the regression. If not, you have a coverage gap. Either the rubric doesn't capture the failure mode, or the threshold is too lenient to detect the level of quality drop that caused the incident.
We run this exercise internally when we update Orq's own eval pipeline. Every time we catch a quality drop in our own agent development, we ask whether our current suite would have caught it automatically. If not, we update the suite. The suite should encode our accumulated understanding of what failure looks like, not just our understanding at the time we first wrote it.
Threshold calibration: the work that's never done
Setting a threshold is not a one-time decision. Thresholds that were appropriate when the feature launched may be too lenient after you've improved quality, or too strict after you've changed the output format in a way that legitimately shifts score distributions.
The practical calibration process: periodically run your eval suite against a sample of recent production traces that you've manually reviewed. Check whether the scores agree with your manual assessment. If you'd manually approve an output that scores 3.2, your threshold might be too strict. If you'd reject an output that scores 4.1, your rubric isn't capturing what you care about.
Some teams review calibration quarterly. Others do it after any significant change to the feature or the underlying model. The right cadence depends on how fast your agent and its use cases are evolving. The wrong cadence is "we set it at launch and haven't revisited it since."
Adding coverage for failure modes you've encountered
Every production regression you experience is information about where your eval suite has a gap. When a regression slips through your gate and reaches users, the correct response is:
- Fix the regression in the agent.
- Write a rubric or update an existing rubric to cover this class of failure.
- Add test inputs to your corpus that represent the inputs where this failure appeared.
- Set a threshold that would have blocked the bad deploy.
- Verify that the updated suite would have caught the regression.
This process converts each regression incident into suite coverage. Over time, your suite represents the actual distribution of ways your agent can fail, because it was built from real failures, not from theoretical coverage analysis.
A suite built this way is fundamentally different from one built by reasoning about what might go wrong. The former is calibrated against reality. The latter is calibrated against your imagination. For LLM agents, where the failure modes are often surprising and input-dependent, the difference in coverage is significant.
The signal that your suite is working: how often it fires
A suite that never fires on a deploy, across many months of development, is probably not sensitive enough. Real development involves prompt changes, model version bumps, and context modifications that introduce regressions on at least some input types. A suite that catches zero of these regressions is either measuring the wrong things or has thresholds so lenient that any regression clears them.
A healthy eval suite fires occasionally. A deploy gets blocked a few times per quarter, someone reviews the regression, confirms it's real, and either fixes the agent or (if it turns out to be a rubric miscalibration) updates the rubric. The gate is working as a signal, not as a speed bump.
If your suite has never blocked a deploy, we'd recommend a calibration exercise: introduce a deliberate regression (lower grounding, or reduce task completion) and verify the suite catches it. If it doesn't, the suite has silent coverage gaps. Better to find that in a controlled test than when a real regression slips through.
Ownership: who maintains the eval suite
Eval suite maintenance tends to fall through the cracks when it's not explicitly owned. The team that built the feature ships it, sets up the initial evals, and moves to the next project. Months later, the eval suite hasn't been updated and the regressions it catches are increasingly misaligned with the regressions that actually matter.
Assigning explicit ownership to eval maintenance, someone who reviews gate failures, updates thresholds when the feature changes, and adds coverage for new failure modes, is the difference between a living eval suite and a stale one. This doesn't need to be a dedicated role. For most early-stage teams, it's one engineer spending a couple of hours per month: reviewing the last period's gate failures, deciding whether each was a real regression or a calibration issue, and making the appropriate update.
The eval suite is as much a product artifact as the agent itself. It encodes the team's current understanding of what quality means for the feature. That understanding evolves. The suite should too. A suite that's three months out of date is giving you false confidence, not real coverage, and false confidence is more dangerous than no coverage because it stops you from looking.