Observability built for the shape of agent work
Agents are not APIs. They call LLMs in sequences, branch on outputs, and fail silently. Orq gives you a trace, a score, and a gate, in that order.
Capture every LLM call, tool use, and handoff
Orq instruments your agent at the span level. Every LLM call, every tool invocation, every agent handoff is recorded with its input, output, latency, and token counts.
- Full prompt and completion capture per span
- Tool call arguments and return values
- Latency breakdown across every step
- Token budget tracking per model per run
- Linked spans across multi-agent chains
Run your rubric on every trace
Define your eval metrics once: coherence, grounding, completeness, latency, safety, format compliance. Orq scores every new trace automatically against your suite and tracks drift over time.
- LLM-as-judge and deterministic checks
- Rubric-based scoring (pass/fail + numeric)
- Score history per metric per version
- Alert when any metric drifts out of threshold
Note: eval scoring tells you whether outputs match your rubric. It does not replace human review for safety-critical decisions, edge-case domain knowledge, or judgments that require context your rubric does not capture.
Block the deploy when a score drops
Set a threshold per metric. Connect Orq to your CI pipeline. When any metric regresses below your threshold on the incoming branch, Orq blocks the merge automatically, before the change reaches users.
- Per-metric thresholds, per-project rules
- Blocks merge via CI status check API
- Full audit log of every gate decision
- Override with justification (logged)
Note: the release gate works on score regression relative to your defined thresholds. It cannot detect semantic drift that stays within numerical bounds. If your threshold is set too loosely, regressions that do not cross it will pass the gate.
See it working in your agent stack
Join our early-access program and get a trace, an eval suite, and a release gate set up in one afternoon.