Where the assumptions break
API regression testing is built on two assumptions: that the same input reliably produces the same output, and that a correct output is precisely specifiable. When you write a test that sends a POST request and asserts the response contains a specific field with a specific value, both assumptions are in play. The service is deterministic (same state + same input = same output) and you know what "correct" looks like.
Neither assumption holds for an LLM. Ask the same question twice and you'll get two answers that are related but textually different. This isn't a flaw in the model or a bug in your implementation; it's a consequence of how these systems work. Temperature, sampling, and the complexity of learned representations all contribute to output variance.
More fundamentally, "correct" is not a precisely specifiable property for most LLM outputs. A correct API response either contains the right field value or it doesn't. A correct LLM summary is one that accurately represents the source material, is appropriately complete, and is communicated clearly. These are not conditions you can express as an assertion against an exact string.
What breaks in practice when you apply API testing assumptions
We've seen teams try a few approaches that fail in characteristic ways:
Exact match assertions
Record an output you like, use it as the expected value in future test runs. This fails immediately because even with temperature=0, subtle context changes produce different outputs. The test fails not because quality degraded but because the model generated a slightly different word order. Developers learn to ignore these failures, which means they'll also ignore the real regressions when they appear.
Contains-keyword checks
Assert that the output contains specific words or phrases. This is a step up from exact match but still produces both false positives and false negatives. An output can contain all the right keywords while being fundamentally wrong (the keywords appear in a different context than intended), and a correct output can fail the test because it used a synonym or restructured the same information differently.
Length thresholds
"The output must be between 100 and 400 words." Length correlates weakly with quality. A 380-word output that is completely off-topic passes this test. A 95-word output that is a concise and accurate answer fails it. Length is a hygiene check, not a quality signal.
The adapted approach: testing properties, not values
The shift that makes LLM regression testing work is moving from "does this output match what I expected" to "does this output have the properties that good outputs have." This is the rubric-based evaluation model.
Instead of asserting a specific response, you define what constitutes a passing response for your feature: grounded in the source material, complete on the required dimensions, within the appropriate scope. You write a rubric that makes these criteria explicit enough to score consistently. Then you run every output through the rubric and track whether scores stay above threshold.
The regression signal shifts from "output doesn't match baseline" to "scores have dropped below threshold." This signal is meaningful because it correlates with the quality dimensions that matter to users, not with exact textual similarity to a reference output you happened to capture on one run.
Corpus construction: where LLM regression testing diverges most from API testing
API regression tests are often generated automatically from request/response pairs captured in a staging environment. The corpus is comprehensive by construction if your staging traffic mirrors production.
LLM regression test corpora need more curation. Because LLM failures are often input-type-specific (a grounding regression that only appears on long documents, a task completion failure that only occurs on multi-part queries), your corpus needs to represent the full distribution of input types you care about, not just a random sample of requests.
The practical approach:
- Pull 40-80 inputs from production traces, covering the range of input lengths and types your agent handles.
- Include examples that previously caused quality issues, even if they've been fixed. These are the inputs where the agent has known difficulty and where regressions are most likely to surface.
- Add a few inputs that represent newly supported use cases each time you expand the feature. Coverage gaps in your corpus become detection gaps for regressions in those use cases.
Handling output variance in threshold setting
LLM outputs vary even on the same input on successive runs. This means your eval scores will have some natural variance that's not correlated with quality changes. A grounding metric might score 4.2 on one run and 3.9 on another run of the same input, just because the model produced slightly different wording each time and the scorer responded differently.
The solution is to set thresholds against the distribution rather than a single run. Run each input several times and look at the score distribution. Set your threshold conservatively enough that normal output variance doesn't trigger false gate failures, but strictly enough that a real regression (where scores drop systematically, not just by a point on one input) is detected.
This is different from how API test thresholds work. An API test either passes or fails, deterministically. An LLM eval threshold is a statistical judgment: are these scores consistent with normal variance, or are they low enough that something has actually changed? The practical difference is that you calibrate against observed variance before setting the threshold, not just against what a passing score should look like.
What LLM testing can borrow from API testing
The principles that do transfer:
- Testing must run automatically in CI, not manually before each release.
- The test corpus must be maintained and expanded over time, not just written once and forgotten.
- A failing gate must block a deploy, not just produce a warning that gets ignored.
- Test coverage gaps are coverage gaps. If a failure mode isn't in your corpus, you're not testing for it.
These are process disciplines, not technical ones. They're what makes any automated testing system reliable over time, for APIs and for LLMs alike. The technical differences in how you score outputs are significant, but the operational rigor is the same.
The teams that catch LLM regressions before users do are the teams that applied this operational rigor to their eval pipelines, calibrated their thresholds, maintained their corpora, and treated a gate failure as a real signal rather than an inconvenience to override.
The mental shift: from "did it break" to "is it still as good"
The deepest difference between API regression testing and LLM regression testing is the question being answered. API regression testing asks: did the behavior change from what it was? LLM regression testing asks: is the quality still at or above the level that meets user expectations?
These are fundamentally different questions. An API test can pass even if the behavior changed, as long as the behavior changed in an expected way. An LLM eval can flag even when nothing in the code changed, if the model's output distribution shifted due to a provider update or input pattern change.
This means the test suite for an LLM feature can never be fully static. The corpus needs to represent current real-world usage. The thresholds need to reflect current quality expectations, not the quality bar you set at launch. The rubrics need to encode your current understanding of what failure looks like, not your understanding from six months ago.
For teams coming from a software testing background, this feels like more work. And it is, in the sense that it requires ongoing calibration rather than one-time setup. But the alternative is the standard pattern: discovering that quality degraded from user complaints, weeks after it happened, with no trace data that tells you why. The ongoing calibration work is what prevents that pattern.