Evaluations

Automatically analyze completed voice agent sessions. Evaluations score agent performance, check compliance, and extract structured data using LLM judges, structured-data extractors, or deterministic predicates over the session's derived facts and events.

When to Use#

• Quality monitoring — automatically score a percentage of live sessions
• Compliance checking — verify agents follow required scripts or collect required information
• Data extraction — pull structured data from conversations (customer name, intent, outcome)
• Regression testing — re-evaluate historical sessions after changing evaluator criteria

Evaluator Types#

An evaluator is a reusable template that defines how to analyze a session. There are three types:

Judge Evaluators#

A judge evaluator scores a session as pass or fail based on criteria you define. It reads the session transcript or events, sends them to an LLM with your prompt, and checks the response against success criteria.

Example: Did the agent collect the customer's name?

Success Criteria Operators

Deterministic assertions and preconditions support all of the above, plus $contains, $icontains, and $any for working with strings and event payloads. See Deterministic Evaluators below.

Extraction Evaluators#

An extraction evaluator pulls structured data from a session — no pass/fail, just data extraction.

Example: Extract call summary

Deterministic Evaluators#

A deterministic evaluator asserts on the derived session view — no LLM call, no token spend, fully repeatable. Use these for checks that are purely factual: was a tool called, did the agent transfer, did the call end with a goodbye, was the duration within bounds. The assertion runs as a JSON predicate against a flat view of the session.

Example: Did the caller mention cancellation?

Session view fields the assertion can read:

Operators. Assertions support the Success Criteria Operators table above plus three more for payload work:

Dotted field paths walk nested objects (e.g. data.content). Mixing operators and field names at the same level is rejected at evaluation time.

Bare primitive vs array input. When a predicate's value is a primitive and the input field is an array, the engine does membership matching (MongoDB-style). This lets tags: "billing" work without $inc, and lets environment: "production" match against the resolved [id, name] regardless of whether you wrote the name or the ID. Scalar-vs-scalar equality is unchanged.

Example: did the caller say "refund"?

assertion:
  events:
    $any:
      name: "transcript_part"
      data.role: "user"
      data.content: { $icontains: "refund" }

Example: production-tagged inbound calls that ended cleanly:

assertion:
  direction: "inbound"
  environment: "production"     # matches name OR ID via bare-primitive membership
  tags: "customer-vip"          # tag presence — no $inc needed

When the assertion matches, the row records success: true with details: { matched: true }. When it fails, success: false with details: { matched: false, failedPath: "...", reason: "..." } — the failing field path is captured so reviewers can see exactly which clause didn't hold.

Preconditions#

Any evaluator type can declare an optional precondition predicate that gates whether it runs. If the predicate doesn't match the session, the evaluator is skipped instead of executed — no LLM call, no token spend — and a skipped row is recorded with the reason. This stops you from paying for "did the agent handle the objection well?" evals on 1-turn hangups, while keeping the skip auditable.

Preconditions use the same predicate language and session-view fields as deterministic assertions.

Example: Skip the eval unless the call ran long enough to score

When a session has fewer than 3 turns or shorter than 30 seconds, the evaluation row is written with status="skipped" and a skipReason like precondition not met at "turn_count": $gte 3 failed for 1.

To audit which sessions were skipped:

vr evaluation list <agent> --status skipped

Or filter by status in the web dashboard. Skipped rows never incur token cost.

Target Formats#

Judge and extraction evaluators choose what they send to the LLM:

• Transcript — the human-readable conversation turns (recommended for most use cases)
• Events — the full structured JSON event log (useful when you need to inspect timing, tool calls, or internal events)

Deterministic evaluators ignore this setting — they always run against the derived session view (see the field table above).

Creating an Evaluator#

Evaluators are configured as code in .voicerun/templates/ and shipped with a release. The web dashboard no longer creates evaluator definitions or assignment records; the released manifest is the source of truth.

Create a kind: Evaluator document alongside your other declarative resources:

apiVersion: voicerun/v1
kind: Evaluator
metadata:
  name: resolution-judge
spec:
  title: Resolution Judge
  evalType: judge
  targetFormat: transcript
  systemPrompt: |
    Did the agent resolve the caller's request? Respond with JSON.
  responseSchema:
    type: object
    properties:
      resolved: { type: boolean }
      reasoning: { type: string }
    required: [resolved, reasoning]
  successCriteria:
    resolved: { $eq: true }
  apiProvider: google
  model: gemini-3.5-flash

Then render and release the manifest:

vr render
vr release <environment>

See Declarative Resources: Evaluator for the full field reference and additional examples.

Running Evaluators#

Evaluator resources in the active release run automatically after each completed, non-debugger session. There is no dashboard assignment step for manifest-defined evaluators, and manifest evaluators currently run for every eligible session. Use precondition to skip sessions that should not be scored, such as short calls or sessions without a relevant event.

Debug sessions (vr debug) are not evaluated automatically. Simulated sessions and live sessions are eligible when they use a release that contains kind: Evaluator resources.

Viewing Results#

Web Dashboard#

Go to your agent's Evaluation tab, Evaluations sub-tab. You can filter by:

• Status (pending, complete, error, skipped)
• Type (judge, extraction, deterministic)
• Trigger (automatic, batch)
• Success (pass/fail — applies to judge and deterministic)
• Session origin, environment, date range

Skipped rows render with a neutral gray "Skipped" pill and a Skip-reason panel naming the failing precondition field. Deterministic rows render with their assertion JSON and a structured details payload — no model / provider / token block, since no LLM call was made.

Click any evaluation to see the full ruling or extracted data, the model used, and token costs.

CLI#

# List evaluations for an agent
vr evaluation list <agent>

# List evaluations for a specific session
vr evaluation list <agent> --session <session-id>

# Filter by status or type
vr evaluation list <agent> --status complete --type judge

# Audit which sessions were skipped (precondition not met)
vr evaluation list <agent> --status skipped

# Audit cheap deterministic evals
vr evaluation list <agent> --type deterministic

# View evaluation details
vr evaluation info <evaluation-id>

Recommended Workflow#

1. Define evaluators in .voicerun/templates/ for the behaviors you care about (greeting, data collection, issue resolution, tone)
2. Add preconditions to avoid scoring sessions that are too short or not relevant
3. Release the manifest to the target environment
4. Make calls or run simulations against that release — evaluators run automatically after eligible sessions complete
5. Review results in the Evaluation tab or with vr evaluation list, then adjust the evaluator resource and release again

Note: Evaluators do not run automatically on debug sessions (vr debug). To evaluate debug or historical sessions, run an explicit evaluation from the CLI/API.