Development

The VoiceRun CLI ships tools for interactive debugging, automated testing, simulated calls, post-session evaluations, custom metrics, and A/B experiments — covering the full feedback loop on a voice agent.

Debugging#

Launch the Pipeline Debugger — a visual Electron app for real-time testing — against an existing, completed (released) release:

vr debug
vr debug --release <RELEASE_ID>

Without --release, the CLI opens an interactive picker containing only completed releases. Use --environment to narrow the picker. Inside a VoiceRun project, passing --release always selects the release-based flow. The release must belong to the agent in the selected lock file: .voicerun/agent.lock by default, or .voicerun/tenants/<name>/agent.lock with --tenant or $VOICERUN_TENANT. Scripted and other non-interactive runs require an explicit release. If release discovery fails, vr debug exits without pushing local code.

Options#

Interactive Controls#

• Enter — Send a text message to the agent
• Ctrl+C — End the session

The GUI shows Gate, Input, and Output lanes with expanded lanes and event pills for readability; Marks, Metrics, and Pipeline events remain in session exports while their lanes are currently hidden. Visible timing labels truncate to whole milliseconds. Each text_to_speech event expands across its browser playback duration, derived from 24 kHz PCM media and confirmed by the playback mark. If a clear interrupts queued audio, the TTS bar is truncated in red and its details show played and discarded durations. Session exports preserve the playback details on the TTS event without embedding raw audio frames.

Outbound call testing#

Use a phone entrypoint configured for outbound traffic:

vr outbound call <ENTRYPOINT> --to +15551234567 --wait

Use --release <RELEASE_ID> to pin a specific release while testing.

Headless Debugging (CI / Coding Agents)#

--headless runs the debugger without a window, streams JSONL events to stdout, and reads text input from stdin (one message per line). Combine it with --script to drive a fully scripted test run:

vr debug --headless --release <RELEASE_ID>
vr debug --headless --release <RELEASE_ID> --output session.json
vr debug --headless --release <RELEASE_ID> \
  --script test-messages.json -o out.json

The script file is a JSON array of strings. Entries are normally paced by the debugger's response-completion turn_end event rather than turn starts or input-gate changes: the first waits for the greeting boundary, and later entries wait for the preceding response boundary. Compatibility quiet and stall timeouts keep scripts moving if no completion event arrives. For relay sessions, turn_end follows TTS completion, audio forwarding, and playback-mark acknowledgement. The GUI acknowledges the mark after queued audio plays; headless mode discards audio and acknowledges it immediately.

Testing#

Run pytest against your voice agent project:

vr test

vr test validates the project, installs dependencies, and runs pytest.

Options#

Running Specific Tests#

vr test tests/test_handler.py
vr test tests/test_handler.py --verbose --coverage

Passing Arguments to pytest#

Use -- to pass additional arguments directly to pytest:

vr test -- -k "test_greeting" --tb=short

LLM Completions in Tests#

Your handler's LLM completion calls — configure_provider and generate_chat_completion from primfunctions.completions — work under vr test just as they do in a released agent. When you exercise your handler with primfunctions' TestRunner, vr test wires the completions client up automatically.

Availability: completion support in vr test is currently available to enterprise organizations. If it is not enabled for your organization, vr test still runs your suite — completion calls just won't succeed.

Bring your own API key

In vr test, completions use your own provider API key — voicerun_managed=True is not available. Your handler reads the key from context.variables, exactly as in a released agent:

from primfunctions.completions import configure_provider, generate_chat_completion

# In your handler:
configure_provider("openai", api_key=context.variables.get("OPENAI_API_KEY"))

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-4.1-mini",
    "messages": [{"role": "user", "content": "Say hello"}],
})

Export the key before running the test:

OPENAI_API_KEY=sk-... vr test

Then pass it to the runner as variables, so it reaches context.variables:

import os
from primfunctions.test_runner import create_test_runner

runner = create_test_runner(
    handler,
    variables={"OPENAI_API_KEY": os.environ["OPENAI_API_KEY"]},
)

If your handler configures a provider with voicerun_managed=True, those completions won't run under vr test. To exercise that code path locally, have the handler use a key from context.variables when one is present and fall back to voicerun_managed=True otherwise.

Notes

• Sign in with vr signin before running vr test.
• Because tests use your own API key, completion calls in vr test count against that provider account's usage.

Simulating an Agent#

vr simulate runs a Simulation resource (defined in .voicerun/templates/) against the active release for an agent and environment. The API pre-creates N agent sessions (origin: simulation) and drives each one against the live /ws/entrypoint route.

Simulations can be inbound (the default), where the simulator plays a caller dialing the agent, or outbound, where the simulator plays the person who answers a call the agent placed. Outbound simulations use the same context.input_data path as real outbound calls, so they are useful for testing task-driven callback, reminder, or follow-up agents before placing live calls.

vr simulate <ENVIRONMENT>                          # List available simulations
vr simulate <ENVIRONMENT> --name happy-path        # Run a single simulation
vr simulate my-agent production --name happy-path  # Explicit agent

Options#

The active release's manifest is authoritative for spec.systemPrompt and spec.numberOfSimulations — local edits not yet released are previewed but not used at run time.

Example Simulation resource#

apiVersion: voicerun/v1
kind: Simulation
metadata:
  name: happy-path
spec:
  systemPrompt: |
    You are a customer calling support. Ask to reset your password.
  numberOfSimulations: 5
  voice: Aoede       # optional Gemini-Live prebuilt voice
  varietySeed: 12345 # optional reproducible delivery variation

Use varietySeed when you want the same simulated caller delivery — greeting, phrasing, mood, and verbosity — across repeated runs. Omit it for fresh variation every run.

Outbound simulation#

Set spec.direction: outbound when the agent is the caller and the simulated persona is the callee. Use spec.inputData for the dynamic task payload the handler reads from context.input_data.

apiVersion: voicerun/v1
kind: Simulation
metadata:
  name: appointment-reminder
spec:
  direction: outbound
  systemPrompt: |
    You are Dana Lee. You just answered your phone and are willing to talk briefly.
  inputData:
    customerName: Dana Lee
    appointmentTime: Tuesday at 3pm
    reason: confirm upcoming appointment
  numberOfSimulations: 3

Pre-pickup phases are inbound-only. Outbound simulations model the callee answering immediately, so manifests with both direction: outbound and phases are rejected during validation.

After submitting, the CLI prints the run ID and the spawned session IDs. Inspect with:

• vr session info <id> — full session detail
• vr session transcript <id> — conversation transcript
• vr metrics session <id> — custom metrics recorded during the run

Evaluations#

Evaluations are produced by Evaluator resources defined in .voicerun/templates/. Evaluators run server-side after each session completes; the CLI is read-only — define new evaluators by adding a kind: Evaluator document and shipping it via vr release.

vr evaluation list                                  # List evaluations for the project agent
vr evaluation list my-agent                         # Explicit agent
vr evaluation list --session SESSION_ID             # Only evaluations for one session
vr evaluation list --status error --type judge      # Filter
vr evaluation info <EVALUATION_ID>                  # Full detail for one evaluation

List Options#

Example Evaluator resource#

apiVersion: voicerun/v1
kind: Evaluator
metadata:
  name: resolution-judge
spec:
  evalType: judge            # judge | extraction | deterministic | script
  targetFormat: transcript   # events | transcript
  systemPrompt: |
    Score the session 1-5 on whether the agent resolved the caller's request.
  responseSchema: {}
  successCriteria: {}
  apiProvider: google
  model: gemini-3.5-flash

vr evaluation info renders deterministic evaluation details, including the assertion predicate and structured match/failure details. Skipped evaluations show their skipReason so precondition failures are auditable without an LLM call.

Custom Metrics#

vr metrics queries custom metrics emitted by agents at runtime. Time-series queries use type-appropriate aggregation server-side.

vr metrics names                                                    # List metric names
vr metrics names my-agent                                           # Scoped to one agent
vr metrics tags                                                     # Discover tag keys/values
vr metrics tags --metric call_duration                              # Tags for one metric
vr metrics timeseries call_duration \
    --start 2026-05-15T00:00:00Z --end 2026-05-16T00:00:00Z --step 1h
vr metrics session <SESSION_ID>                                     # All metrics for one session

Most metric subcommands accept --agent/-a to scope results to a single agent (or fall back to agent.lock), and support --json / --table output.

A/B Experiments#

vr experiments reads experiment definitions and results. The agent is resolved from --agent (or positional argument) first, falling back to agent.lock.

vr experiments list                                # List experiments for the project agent
vr experiments describe greeting_style             # Variants, conversions, significance
vr experiments funnel greeting_style               # Per-variant funnel with lift
vr experiments timeseries greeting_style \
    --metric booking_completed \
    --start 2026-05-01T00:00:00Z --end 2026-05-15T00:00:00Z --step 1d

vr experiments describe shows session count, conversion metrics, stop conditions (iteration and confidence thresholds), variant performance, and statistical significance (confidence, p-value).

Development Workflow#

A typical development cycle looks like this:

1. Create a new project with vr init
2. Write your agent handler in handler.py and declare runtime config in .voicerun/templates/deployment.yaml
3. Render templates with vr render to confirm the manifest looks right
4. Validate with vr validate (or vr validate -e production)
5. Unit-test locally with vr test
6. Push your code with vr push
7. Release to a non-production environment with vr release staging
8. Debug interactively with vr debug --environment staging, or run a script with vr debug --headless --release <RELEASE_ID> --script test-messages.json
9. Simulate regression scenarios with vr simulate staging --name happy-path
10. Cut over by releasing to production and pointing an entrypoint: vr release production --entrypoint support-line
11. Observe in production with vr session list, vr metrics, vr experiments, and vr evaluation list