The assumption that broke
When we started Orq AI, we spent a few months talking to engineers who had shipped LLM-backed features at B2B software companies. We asked one question: how do you know when your agent's output quality has dropped? The most common answer was a version of "a customer tells us."
That answer points to a structural problem, not a careless team. These are people who write unit tests, track error rates, and page on-call when a service throws 500s. They are not skipping quality checks because they don't care. They're skipping them because the standard reliability toolkit doesn't have a slot for "the output was technically valid JSON but the answer it contained was wrong."
Why deterministic testing doesn't transfer
API regression testing works because given the same input, you get the same output. You encode that expectation in an assertion, you run it in CI, and if the assertion fails you know the code changed in a way you didn't intend. The feedback loop is fast and the signal is clean.
LLM outputs don't behave that way. Ask the same question twice and you get two answers that are semantically equivalent but textually different. Run the same prompt against a model version bump and the output may improve on some inputs, regress on others, and stay identical on most. There is no meaningful "expected output" to assert against.
This isn't a deficiency teams can engineer around with better test data. It's a property of how these models work. The fix isn't to write better assertions, it's to change the evaluation method entirely: from "does output == expected" to "does output score above threshold on these rubrics."
What fills the gap instead (and why it fails)
Most teams land on one of three substitutes when they first realize their existing tests won't catch LLM quality drops:
- Manual review before deploy: Someone on the team checks a handful of outputs and approves the release. This works until the team gets busy or the feature gets too complex to spot-check. It also only covers the happy path the reviewer happens to test.
- User feedback monitoring: Tracking thumbs-down, support tickets, or churn signals. Reactive by definition. A regression that only affects a specific input type can persist for weeks before enough users encounter it to show up in aggregate data.
- LLM output logging: Writing every input and output to a database. Better than nothing, but a pile of logs doesn't tell you whether quality has changed. You still have to build the analysis layer on top, and most teams never do.
All three approaches share a failure mode: they generate information after the fact, or they depend on someone manually interpreting that information before it becomes actionable. Neither works at the cadence a shipping team actually needs.
The timing problem
Software reliability tooling was designed around the assumption that a bug has a deterministic cause you can reproduce. You get a stack trace. You write a failing test. You fix the code. The test goes green. Done.
An LLM quality regression often has no stack trace. The agent didn't throw an error; it returned a response that was less grounded, less complete, or less accurate than it was last week. The cause might be a prompt change that shifted behavior on a specific input class, a model version bump, or a context window boundary you crossed as your data grew. None of these show up in error logs.
By the time a user files a complaint, the regression is already at least days old. If the agent is used across multiple sessions per day and the regression only affects, say, long documents, the detection lag can be weeks. You've delivered a degraded experience to many sessions before your monitoring system produces a signal you can act on.
The missing primitive: eval-gated deploys
The shift that breaks this pattern is treating every deploy as something that has to pass an eval suite before it reaches users, the same way every code change has to pass a test suite. That means:
- You have an eval suite that runs against a fixed set of traces and scores outputs on the dimensions that matter (grounding, completeness, task adherence, safety, whatever your feature needs).
- You have a score threshold. If a metric drops below it, the deploy doesn't proceed.
- The check runs automatically in your CI pipeline, not manually before each release.
This shifts the question from "did anyone check this before it shipped" to "did it clear the bar before it shipped." The former depends on memory and available bandwidth. The latter is a deterministic gate.
We're not saying this is easy to set up from scratch. Building an eval suite that reliably distinguishes real regressions from normal output variance takes iteration. The rubrics have to be calibrated against real examples. Thresholds need tuning. But the baseline version, three or four metrics and a passing threshold, is achievable in a day and already far more reliable than waiting for users to notice.
What this doesn't solve
Evals catch what you measured. If your suite doesn't include a metric for factual accuracy on a specific topic domain, a regression in that domain will pass your gates. The coverage problem is real, and it doesn't go away when you add a gate, it moves upstream. Now the question is "does your eval suite cover the things that would actually hurt users" rather than "did anyone look at this before it shipped."
That's a harder question to answer, but it's the right question to be asking. It forces you to think explicitly about what output quality means for your feature, rather than discovering the answer when a user tells you something is wrong.
If you're building on LangChain, LlamaIndex, or direct API calls to OpenAI or Anthropic, the tracing instrumentation to make this kind of eval possible is a few lines of setup. The eval suite itself takes more thought. We've written more on how to build one that actually catches the regressions that matter, starting with the article on minimal baselines in this series.
The goal isn't to build a perfect eval system before you ship anything. It's to break the pattern where user complaints are the first signal that quality has dropped. Any automated gate that closes before a degraded output reaches production is better than none.
The conversation that doesn't happen until it does
Most engineering teams don't have a conversation about LLM quality monitoring until after the first significant regression. Before that, quality feels like something that's being managed informally, through code review, prompt testing, and occasional output reviews. The first regression that reaches production changes this: suddenly there's an incident, and someone asks "why didn't we catch this."
The answer is usually structural. It's not that the team was careless. It's that the tooling they had wasn't capable of catching this class of failure. The same team that has a functioning alerting pipeline for service errors, latency spikes, and database failures has no equivalent alerting pipeline for LLM output quality. The gap isn't a missing process; it's a missing primitive.
Evals and release gating are that primitive. They don't require a large team to operate. They don't require months of setup. They require a few hours of thoughtful work to define what quality means for your specific feature, and then the discipline to treat the gate as a real gate rather than a suggestion. That combination is what breaks the pattern of discovering LLM bugs from users rather than from your own monitoring.