Skip to content

Eval Metrics

Orq ships a library of built-in metrics. You can use them individually or combine them into a named suite.

Correctness

Measures whether the response accurately answers the question given the context. Scored 0.0 to 1.0 using an LLM judge with a structured rubric. Threshold default: 0.75.

Faithfulness

Checks whether every factual claim in the response is grounded in the provided context. Useful for RAG pipelines. Scores are 0.0 to 1.0; a score below 0.8 typically indicates hallucination.

Relevance

Scores how closely the response addresses the user intent derived from the input. High relevance with low correctness usually means the model understood the question but answered it wrong.

Toxicity

Binary metric. Uses a classifier to detect harmful, offensive, or inappropriate content. Any positive flag blocks the trace from passing the gate by default.

Latency

Milliseconds from first token request to last token received. Configurable percentile threshold (default p95 < 4000ms).

Custom LLM judge

Define your own rubric as a YAML file and register it with the SDK. The platform runs it as a normal metric.

YAML
name: my-custom-metric
judge_model: gpt-4o-mini
prompt_template: |
  Score the following response 0-1.
  Response: {response}
  ...
threshold: 0.7

Next steps