How to instrument a LangChain agent for production evals

What this walkthrough covers

This guide walks through adding Orq tracing to a LangChain multi-step agent and connecting eval scores to a CI pipeline gate. By the end, every agent invocation will produce a trace with scores on your configured eval metrics, and a deploy that regresses those scores will be blocked automatically.

The agent in this example is a document Q&A agent: it retrieves relevant context from a vector store, passes it to an LLM with a structured prompt, and returns an answer. The pattern generalizes to any LangChain agent that uses chains, tools, or the LCEL composition API.

Step 1: Install and configure the Orq SDK

Start with the SDK and set your API key in your environment:

pip install orq-sdk

# in your .env or environment config
ORQ_API_KEY=your_api_key_here

Initialize the Orq client near the top of your agent module. The client reads the API key from the environment automatically if ORQ_API_KEY is set:

from orq_sdk import OrqClient

orq = OrqClient()  # reads ORQ_API_KEY from env

Step 2: Attach the tracing callback to your LangChain chain

LangChain's callback system fires hooks at each step: chain start/end, LLM start/end, tool start/end, retriever start/end. The Orq callback wraps these into a structured trace with parent-child spans that reflect the agent's execution structure.

from langchain.callbacks.base import BaseCallbackHandler
from orq_sdk.langchain import OrqCallbackHandler

orq_callback = OrqCallbackHandler(client=orq)

# attach to your chain, LCEL pipe, or agent executor
chain = (
    retriever
    | format_docs
    | prompt
    | llm
    | StrOutputParser()
).with_config(callbacks=[orq_callback])

The callback handler captures inputs, outputs, latency in milliseconds, and token counts at each step. For retrieval steps it captures the query and the documents returned. For LLM steps it captures the full prompt and completion. Nothing is sampled; every invocation produces a trace.

Step 3: Verify the first traces appear

Run a few test invocations of your agent. In the Orq dashboard, you should see traces appearing in real time. Each trace shows the full call sequence: retriever span, prompt formatting span, LLM call span, and the output. Latency is shown at each level and for the end-to-end invocation.

If traces aren't appearing, check that the callback is attached to the outermost chain object. A common mistake is attaching it to a sub-component rather than the top-level chain, which means some spans are captured but the parent trace is missing.

Step 4: Define eval metrics for this agent

For a document Q&A agent, the two metrics that cover the highest-value failure modes are grounding and answer completeness. Define them as rubrics in the Orq dashboard or via the API.

Grounding rubric (summarized): "The answer must only include claims that can be traced to the retrieved documents. Any claim in the answer that cannot be found in the retrieved context is a grounding failure. Reasonable inferences directly implied by the documents are acceptable. Invented facts are not."

Answer completeness rubric (summarized): "If the question can be answered from the retrieved documents, the answer must address all required elements of the question. An answer that is technically accurate but omits a material part of what was asked scores below threshold."

Create these in the Orq dashboard as eval metrics with pass thresholds. A starting threshold of 3.5 on a 1-5 scale is reasonable for both, to be calibrated against your actual agent outputs after you've collected 20-30 traces.

Step 5: Run evals against your trace corpus

With a corpus of traces collected, trigger an eval run from the Orq API against your baseline:

eval_run = orq.evals.run(
    project_id="your_project_id",
    eval_metric_ids=["grounding_metric_id", "completeness_metric_id"],
    trace_ids=your_baseline_trace_ids,  # or use a named trace set
)

print(f"Run status: {eval_run.status}")
print(f"Grounding mean: {eval_run.scores['grounding'].mean:.2f}")
print(f"Completeness mean: {eval_run.scores['completeness'].mean:.2f}")

Review the per-trace scores in the dashboard. Look at the distribution: what percentage of your current traces pass each metric at the threshold you've set? If your baseline only achieves 60% pass rate on grounding, your threshold is probably too strict for the current state of the agent. Adjust the threshold to reflect your current quality level, then tighten it as you improve the agent.

Step 6: Connect the eval gate to your CI pipeline

The gate runs as a step in your CI pipeline after the agent is deployed to your staging environment and before it can be promoted to production. Add an eval run step that fails the pipeline if scores regress:

# in your CI configuration (GitHub Actions, CircleCI, etc.)
# run after staging deploy step

python scripts/run_eval_gate.py \
  --project-id $ORQ_PROJECT_ID \
  --env staging \
  --thresholds grounding:3.5,completeness:3.5

The run_eval_gate.py script (which you write once and maintain) calls the Orq API to run evals against the corpus on the current staging traces, compares the mean and percentile scores to your configured thresholds, and exits 0 if all pass or exits 1 with a detailed report if any threshold is missed. A non-zero exit blocks the pipeline step, which blocks promotion to production.

Calibrating thresholds before going to enforcement mode

The most important setup step that teams skip is calibrating thresholds before activating the gate in blocking mode. If you activate the gate immediately with thresholds set to your first guess, it will likely fire on the first few deploys due to normal score variance, and your team will start overriding it reflexively.

Instead, run the gate in warn-only mode for two to four weeks. Watch when it would have fired and why. Check manually whether those would-have-blocked deploys actually contained regressions. Adjust thresholds based on what you observe. Only then switch to blocking mode.

Once the gate is in blocking mode, treat a gate failure as a real signal. If the gate fires and your scores are showing 3.1 on grounding, that warrants an investigation of what changed in the prompt or model configuration for that deploy, not an immediate override.

What the gate doesn't catch

This setup catches regressions introduced by code or configuration changes you deploy. It does not catch quality drift that happens between deploys: model provider updates, retrieval index staleness, or input distribution shifts as your user base grows. For those, you need continuous monitoring on production traces, which is a separate concern from the deploy gate.

The gate also only covers the eval dimensions you've defined. A regression in a quality dimension you haven't written a rubric for will pass the gate. This is the coverage problem that every eval suite has, and the solution is to add metrics when you encounter failure modes your current suite missed, not to wait until you've thought of every possible failure mode before you deploy the gate at all.

A gate that covers two or three important metrics and runs on every deploy is more valuable than a comprehensive suite that lives in a spreadsheet and gets run manually every quarter. If you're starting from zero, this walkthrough gives you a working gate. Build from here.