The problem with open-ended outputs and deterministic scoring
Writing an eval for a factual question with a known answer is straightforward: the output is correct or it isn't. "What is the capital of France" has one right answer, and you can assert against it. The score is binary.
Now consider an agent that generates a multi-paragraph summary of a research document, or one that produces a project plan given a set of requirements. There's no single correct output. A dozen different responses could all be reasonable, and a dozen different responses could all be useless, and the difference isn't captured by any string comparison.
This is where teams give up on evals entirely, or keep them so narrow that they only cover the easy factual cases. The result is an eval suite that systematically misses the failure modes that users actually complain about: "the summary was useless," "the plan didn't address half the requirements," "it made up a detail that wasn't in the source."
Rubric-based evaluation is the approach that works for this class of output. It's not new; it has roots in how humans score essay responses and generated text in research settings. Applied to LLM agents, it produces scores that correlate with actual quality in a way that string matching doesn't.
What a rubric actually is
A rubric is a description of what a good output looks like, written in enough detail that a scorer (human or LLM-based) can consistently apply it to novel outputs and produce a score that others would agree with.
A weak rubric: "The summary is accurate." This gives a scorer no guidance on what accuracy means for this feature, how complete the summary needs to be, or what to do with outputs that are partially accurate.
A strong rubric for the same metric: "The summary must accurately represent the main argument of the source document. Every claim in the summary must be traceable to a specific passage in the source. The summary should not introduce claims that are plausible but not present in the source. A summary that omits the main argument entirely, even if all included claims are accurate, scores below threshold."
The strong rubric is specific about what "accurate" means, what counts as a failure, and the edge case of accurate-but-incomplete. That specificity is what makes it possible to score consistently across many outputs.
The three dimensions most open-ended outputs need
Different features need different rubrics, but most open-ended LLM outputs can be evaluated along three core dimensions that cover the majority of real failure modes:
Grounding
Does the output only make claims supported by the provided context? Grounding failures are the most common cause of user complaints for document-processing agents. The model confidently includes a detail that isn't in the source, and the user, who trusts the agent, acts on wrong information. A grounding rubric asks: can every specific claim in the output be verified against the input?
Task completion
Did the output actually address what was asked? This is where summarizers that produce technically accurate but completely off-topic responses fail. A task completion rubric asks: does the output address the specific request, and does it address all required elements of that request? For a plan-generation agent, this might mean: are all the stated requirements represented in at least one plan step?
Appropriate scope
Did the output stay within the intended scope, or did it wander into territory the user didn't ask for, include caveats that belong in a different context, or produce more (or less) output than the task requires? Scope failures are subtler than grounding failures but just as disruptive for agents where users are relying on output length and focus to make decisions quickly.
Converting a rubric to a pass/fail signal
A rubric produces a description, not a binary. To use it as a gate in your CI pipeline, you need to convert it to a score. The common approach is a 1-5 scale for each dimension, where you define what each level means in concrete terms specific to your feature.
For grounding, one version might look like this:
- 5: All claims in the output are directly supported by the input context, with no inference beyond what the context clearly implies.
- 4: All major claims are supported. Minor inferences are present but clearly reasonable.
- 3: Most claims are supported. One claim is unsupported but the overall output would not mislead a careful reader.
- 2: Multiple unsupported claims. A reader relying on this output would likely be misled on at least one point.
- 1: The output contains fabricated content not present in the input.
Your pass threshold might be 3.5 on average across your test corpus, or 4 minimum (no output may score below 4), depending on how critical grounding is for your feature. The threshold is set by running the rubric against outputs you'd actually ship and outputs you wouldn't, and finding the score that separates them.
Rubric quality caps eval quality
Here is the limitation worth stating plainly: a rubric that's poorly written will produce scores that don't reflect actual quality. This is a ceiling, not just a floor. A high score on a bad rubric tells you nothing. This is why rubric calibration, running the rubric against outputs you've already judged manually and checking whether the scores agree with your judgment, is not optional work. It's the step that converts a scoring system into a reliable signal.
We calibrate our own rubrics against a set of "known good" and "known bad" outputs for each metric before we trust the scores in a gate decision. If the rubric scores a known-bad output as a 4 out of 5, the rubric is wrong. Fix the rubric, not the score.
When to use LLM-based scoring vs. human scoring
For eval suites that run on every CI deployment, human scoring isn't practical. LLM-based scoring, using a separate model to apply the rubric to each output, is the approach that scales. It introduces its own uncertainty (the scoring model can make mistakes), but for regression detection purposes, what you need is consistency, not perfection. If the scoring model consistently applies the same rubric and the score drops significantly, that's a real signal even if the absolute score values aren't perfectly calibrated.
Human scoring is valuable for calibration (building the reference set), for reviewing gate failures (was this a real regression or a rubric problem?), and for periodic audits of the scoring model's behavior. Mixing the two, automated scoring in CI, human review of gate failures, produces a feedback loop that improves both the rubrics and the agent over time.
You don't need perfect rubrics before you start. You need rubrics good enough to detect the regression that would cause a user complaint. Build from there.