Why "comprehensive" is the wrong target to start with
Every eval article I've read either describes a minimal baseline so thin it can't catch real regressions, or a mature system so involved that building it feels like a quarter-long project. Neither is useful if you're a small team that shipped an LLM feature two months ago and has been flying blind ever since.
The goal of this piece is a working eval baseline that catches the regressions that actually cause user complaints, set up in an afternoon, with Orq as the tracing layer. Not a perfect system. A working one.
What you need before you start
Three things:
- A set of test inputs. Twenty to forty is enough to start. These should come from real usage: actual queries or documents your agent handles. Not invented examples, not the happy path you're proud of. Pull inputs from your logs where the feature was used.
- A sense of what "good" looks like for your specific feature. Not a rubric yet. Just: when you read a response and it feels right, what properties does it have? Write those down in plain language.
- The Orq SDK installed and an API key. The quickstart takes about five minutes.
Step one: get traces flowing
If you're on LangChain, wrapping your agent with the Orq tracing callback is a few lines. For a direct OpenAI or Anthropic API call, you wrap the call itself. The exact SDK call depends on your setup, but the pattern is the same: every LLM call in your agent gets captured as a trace with inputs, outputs, latency, and token counts.
Run your twenty test inputs through the agent. You now have twenty traces in Orq. This is your baseline corpus.
Don't skip this step to do it "properly later." The traces you generate this afternoon are what you'll score against. An eval suite without traces is just a config file.
Step two: write exactly three metrics
Three is not a magic number. It's the number that's enough to catch most regressions without so much setup overhead that you abandon the project. Add more later when you find a regression type they don't cover.
For most LLM agents, these three metrics cover a high fraction of real failure modes:
- Grounding: Does the output only make claims that are supported by the input context? A response that invents facts not present in the input is a grounding failure. For retrieval-augmented agents, this is almost always the most important metric.
- Task completion: Did the agent actually do what was asked? A summary that omits a key point, a plan that's missing a required step, or a response that answers a different question than the one asked are all task completion failures. Write the rubric in terms of your specific feature.
- Output format: If your agent is supposed to return structured data, does the structure match? If it's supposed to follow a specific template, does it? Format failures are usually easy to detect but they break downstream dependencies silently if you're not checking.
For each metric, write the rubric in one or two sentences. "The response must only contain claims directly supported by the documents provided in the input. Any statement that can't be traced to the input documents is a grounding failure." That's enough for a rubric-based eval to score against.
Step three: set thresholds against your baseline
Run your metrics against the twenty traces you captured in step one. Look at the score distribution for each metric. Pick a threshold that would pass your current baseline: something like "90% of current outputs score above 0.7 on grounding." That's your starting gate.
This is important: the threshold is calibrated to your current outputs, not to some ideal. If your agent already has known weaknesses, you might set the threshold conservatively to avoid gating on a known existing issue. As you improve the agent, you tighten the thresholds. The baseline exists to detect regressions from the current state, not to enforce perfection.
Step four: connect to CI
The eval gate has to run automatically, on every deploy, or it won't function as a gate. A one-time eval run is just an audit. The CI integration in Orq can be wired via the API to your GitHub Actions workflow or whatever runner you use. The setup is a few lines: call the eval run endpoint on the baseline corpus after each deploy, and fail the pipeline if any metric drops below threshold.
You now have an automated gate. The next deploy that regresses grounding will be blocked before it ships. Not the one after the user complaint, this one, before anyone notices.
What this baseline will miss
Let's be direct: three metrics won't catch everything. A regression in output tone, a latency spike, or a failure mode that only appears on a specific input type you didn't include in your test corpus will slip through. This is not a flaw in the approach, it's the cost of starting small. The baseline is specifically designed to catch the regressions most likely to generate user complaints, not all regressions.
The right response to a gap in your eval coverage is to add a metric when you find a regression the suite missed. Not to delay starting until your suite is comprehensive. A suite that runs on every deploy and covers three things is worth more than a comprehensive suite that doesn't exist yet.
Also: the eval rubrics you write this afternoon will be imperfect. Some of them will be too lenient, some too strict, some measuring the wrong thing. That's fine. You'll calibrate them against real data. The calibration loop works much faster when you have traces and scores to look at than when you're reasoning in the abstract about what your metrics should be.
Calibration notes from building our own baseline
When we were building the internal eval pipeline we used to validate Orq, we started with a grounding metric that was far too strict. It flagged any output that contained a generalization as a failure, even when the generalization was reasonable given the input. Our pass rate on a synthetic 200-run corpus was around 60%, which made the gate useless: everything was flagging.
We adjusted the rubric to distinguish between "unsupported specific claims" (grounding failure) and "reasonable generalizations from the provided evidence" (acceptable). Pass rate moved to 85% on the same corpus, which was a calibration that actually reflected output quality rather than the quirks of our first rubric draft.
That calibration work took about two hours. The lesson is that a day of setup, followed by a day of calibration, gives you a working gate. That's a much more realistic timeline than "set up a comprehensive eval system before you ship."
The goal at the end of the afternoon is a gate that blocks a specific class of real regressions, not one that blocks everything. Build from there. Every real regression your suite catches is a user complaint that didn't happen.