What pilots actually measure
A pilot is a controlled test of a feature with a small group of users under some level of supervision. The supervision is the part that matters. Pilot users are often more patient, more willing to try again when something fails, and more likely to give you direct feedback about what's wrong. They're also not representative of how the feature will be used at scale, where users are less forgiving, input distribution is wider, and no one is monitoring closely.
This creates a gap that's easy to miss during the handoff. The pilot looked successful. Users were engaged. Quality seemed good. The decision to go to production feels justified. What nobody tracked is that the pilot cohort submitted queries that fell squarely within the feature's designed input space. The long tail of inputs that real users will send didn't appear during the pilot. Neither did the edge cases, the adversarial prompts, or the context lengths that push the model to its limits.
Production is where you find out what the pilot didn't test.
The input distribution problem
The most common failure mode at scale is an agent that performs well on the input types your team tested and degrades on the tail. In a pilot, you might see twenty to fifty distinct input patterns. A production feature with hundreds of daily users will see hundreds of distinct patterns within the first week, including ones no one on the team considered.
This is not a problem you can fully solve at pilot time. You can make your pilot coverage broader, but you can't replicate the actual input distribution of your user base before you have a user base. The practical response is to instrument broadly at production launch so that you're capturing the actual input distribution, and to have eval metrics that can detect quality degradation across the full distribution, not just the cases you expected.
Concretely: if your pilot only included short queries (under 200 tokens), but production users regularly send 1,500-token contexts, your grounding metric needs to be validated against long inputs too. A regression that only appears on long contexts will pass your pilot-calibrated thresholds and reach users undetected.
Latency that was acceptable at ten users breaks at a thousand
Pilots rarely surface latency issues because the request volume is too low to trigger the failure modes that appear under load. This isn't just about server capacity, it's about the patterns in how your agent chains calls.
Consider an agent that makes three sequential LLM calls per request. In a pilot, P95 latency might be 4 seconds. That's acceptable. Under load, if any of those calls hits a rate limit, the agent falls back to a retry with exponential backoff. What was a 4-second P95 becomes a 12-second P99. The retry behavior was present in the pilot code but never triggered.
More subtly: some LLM agents accumulate context across a session. During the pilot, most sessions were short. In production, sessions run longer, context windows fill up, and the model starts truncating or dropping early context. The behavior change is invisible in session traces unless you're specifically tracking context length and its correlation with output quality.
The monitoring gap between pilot and production
Pilots typically have one channel for detecting problems: someone on the team is watching. In production, that channel disappears. What takes its place determines whether you find out about quality regressions from your monitoring or from users.
The minimum monitoring setup for a production LLM feature at any scale includes:
- Automated traces on every agent run, capturing inputs, outputs, and latency
- Eval scores running against every trace on the metrics that matter for your feature
- A dashboard that shows score distribution over time, so a gradual drift is visible before it becomes a complaint
- Alerting on score drops, not just error rate spikes
The last point is the one most teams miss. A service that returns 200s but whose LLM outputs have degraded is invisible to conventional error monitoring. Eval scores are the only signal that can detect this. Without them, you're relying on users to notice, which brings back the detection lag we discussed in the article on regression costs.
What actually breaks at scale: a pattern we've seen repeatedly
In our experience building and instrumenting LLM agents, the failure modes that surface at production scale fall into a consistent set of categories. This isn't a dataset claim; it's a pattern from building our own pipeline and working with early-access teams testing Orq.
The most common is prompt boundary drift. As the user base grows, some users find inputs that push the agent into behavior that doesn't match the intended use case. Prompt constraints that worked in the pilot stop holding at scale. The agent is technically functioning; it's just answering a different kind of question than it was designed for.
Second is tool call error accumulation. Multi-step agents that call external tools will encounter errors from those tools in production at rates that don't appear in pilots. Most agents have some retry logic, but retry logic that hides errors rather than surfacing them creates a class of silent failures: the agent continues, returns a response, but a step in the chain used stale or fallback data.
Third is eval score drift after a model update. Providers update base models, sometimes in ways that change default behavior on specific input types. An agent that passed all your eval thresholds at launch may score slightly lower after a silent model update, and "slightly lower" multiplied across hundreds of daily runs means a meaningful fraction of users getting worse responses.
Scaling the reliability layer in parallel with the feature
The right time to build the monitoring infrastructure for a production LLM feature is before it goes to production, not after the first regression incident. This means:
- Extend your pilot's eval suite to cover the input types production users are likely to send. Broader coverage in the test corpus means your thresholds are calibrated to real variance, not just pilot variance.
- Validate your retry and error handling logic under simulated tool failures before launch. These are the paths that will be triggered in production and never triggered in a pilot.
- Set up your monitoring dashboards before the first production traffic arrives, so you have a baseline to compare against. A dashboard that starts tracking on day three has no baseline for the first three days.
- Build the alert rule for "eval score drops below X over the last 100 runs" before you need it. Alert rules written during an incident are worse than alert rules written when you're calm.
None of this is especially complex. It's operational discipline applied to a class of system that doesn't fit neatly into the standard reliability toolkit. The fundamental point is that a pilot that looked good is evidence of one thing: the feature works in pilot conditions. Production conditions are different. The reliability layer is how you find out whether the feature also works there.