Advanced Features

Provider-specific kwargs#

Each provider exposes tuning knobs beyond the common set (temperature, max_tokens, etc.). Pass them via provider_kwargs, keyed by provider name. The proxy applies only the entry matching the currently-executing provider; entries for other providers are ignored, which lets you configure kwargs for every member of a fallback chain in one request.

provider_kwargs = {
    "openai":            {...},  # applied when provider == "openai"
    "anthropic":         {...},  # applied when provider == "anthropic"
    "google":            {...},  # applied when provider == "google"
    "google_vertex":     {...},  # applied when provider == "google_vertex"
    "anthropic_vertex":  {...},  # applied when provider == "anthropic_vertex"
    "anthropic_bedrock": {...},  # applied when provider == "anthropic_bedrock"
    "openai_bedrock":    {...},  # applied when provider == "openai_bedrock"
    "alibaba":           {...},  # applied when provider == "alibaba"
    "deepseek":          {...},  # applied when provider == "deepseek"
}

OpenAI#

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-5-mini",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "openai": {
            "service_tier": "priority",
            "reasoning_effort": "none",
        },
    },
})

See the OpenAI chat/completions reference for the full list.

Anthropic#

response = await generate_chat_completion({
    "provider": "anthropic",
    "model": "claude-sonnet-4-5-20250514",
    "max_tokens": 16000,
    "messages": [{"role": "user", "content": "Think about this..."}],
    "provider_kwargs": {
        "anthropic": {
            "thinking": {"type": "enabled", "budget_tokens": 10000},
        },
    },
})

Extended thinking is disabled by default. Set thinking.type to "enabled" to turn it on.

See the Anthropic messages API reference.

Google#

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-2.5-flash",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "google": {
            "thinking_config": {"thinking_budget": 10000, "include_thoughts": True},
            "safety_settings": [...],
        },
    },
})

Thinking is disabled by default on Google models.

See the Google GenerateContent reference.

Alibaba (Qwen via DashScope)#

response = await generate_chat_completion({
    "provider": "alibaba",
    "model": "qwen3.5-plus",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "alibaba": {
            "enable_search": True,
        },
    },
})

Override the regional endpoint with base_url:

"provider_kwargs": {
    "alibaba": {
        "base_url": "https://dashscope-us.aliyuncs.com/compatible-mode/v1",
    },
}

Available regions:

Singapore (default): https://dashscope-intl.aliyuncs.com/compatible-mode/v1
Virginia (US): https://dashscope-us.aliyuncs.com/compatible-mode/v1
Beijing (CN): https://dashscope.aliyuncs.com/compatible-mode/v1

See the DashScope model reference.

Multi-provider example (with fallback)#

configure_provider("openai", voicerun_managed=True)
configure_provider("anthropic", voicerun_managed=True)

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-5-mini",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "openai":    {"service_tier": "flex"},
        "anthropic": {"thinking": {"type": "disabled"}},
    },
    "fallbacks": [
        {"provider": "anthropic", "model": "claude-haiku-4-5"},
    ],
})

When the primary fails and the fallback fires, the proxy applies the anthropic entry instead of the openai one.

Anthropic cache breakpoints#

Anthropic supports prompt caching. The library exposes it via CacheBreakpoint, which attaches to a tool, system message, assistant message, user message, or tool-result message.

The cache is built in order tools → system → messages. Place large, stable content early and put a breakpoint at the end of each cacheable section. When mixing TTLs, the longer duration ("1h") must appear before the shorter ("5m").

from primfunctions.completions import (
    AssistantMessage,
    CacheBreakpoint,
    SystemMessage,
    UserMessage,
    configure_provider,
    generate_chat_completion,
)

configure_provider("anthropic", voicerun_managed=True)

# 1. Tools — cache the full tool block
tools = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up an order by ID",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
        "cache_breakpoint": {"ttl": "1h"},
    },
]

# 2. System (stable) + 3. Messages (growing)
messages = [
    SystemMessage(
        content="You are a customer support agent.\n\n<2000+ token reference doc>",
        cache_breakpoint=CacheBreakpoint(ttl="1h"),
    ),
    UserMessage(content="Can you look up order #1234?"),
    AssistantMessage(content="Let me look that up for you."),
    UserMessage(content="What's the shipping status?"),
]

# Dynamic breakpoint on the last message for the growing conversation prefix
messages[-1].cache_breakpoint = CacheBreakpoint(ttl="5m")

response = await generate_chat_completion({
    "provider": "anthropic",
    "model": "claude-haiku-4-5",
    "messages": messages,
    "tools": tools,
})

Rules#

At most 4 cache breakpoints per request.
Longer TTLs (1h) must come before shorter ones (5m) in the prefix order.
"5m" — 5-minute ephemeral cache, refreshed on each hit.
"1h" — 1-hour ephemeral cache, higher write cost.
Each model has a minimum cached-block size. Blocks smaller than the minimum are silently ignored (usage reports cache_creation_input_tokens: 0).
- Opus / Sonnet: 1024 tokens
- Haiku 4.5: ~4096 tokens (empirical; higher than older Haikus)

Don't persist breakpoints on stored messages#

Set cache_breakpoint on the last message only right before the call. If you persist it into conversation history and the conversation keeps growing, you end up with stale breakpoints in the middle of the prefix and potentially more than 4 breakpoints — the request will fail. Apply breakpoints dynamically each turn.

Google thought signatures#

Google Gemini maintains context across turns via a thought_signature field on assistant messages and tool calls. The library captures and re-emits it automatically — as long as you keep the same AssistantMessage / ToolCall dataclass around (or round-trip through serialize_conversation / deserialize_conversation), the next turn's request will include the signature.

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-2.5-flash",
    "messages": [...],
})
# response.message.thought_signature is preserved
# on the AssistantMessage dataclass and survives context.set_completion_messages / get.

No handler code is required to propagate it — just reuse the response message objects on the next turn.

OpenAI Responses API (`openai_responses`)#

openai_responses is a provider that routes to OpenAI's Responses API (/v1/responses) instead of chat completions. You select it like any other provider — configure_provider, generate_chat_completion, tools, streaming, structured output, and fallbacks all work unchanged. The proxy keeps it stateless and fallback-safe (the full conversation is sent inline every turn, exactly like openai).

from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("openai_responses", voicerun_managed=True)

response = await generate_chat_completion({
    "provider": "openai_responses",
    "model": "gpt-5.4-mini",
    "messages": [{"role": "user", "content": user_message}],
    # Disable reasoning for low-latency voice responses.
    "provider_kwargs": {"openai_responses": {"reasoning": {"effort": "none"}}},
})

Reasoning persistence#

If you use a reasoning model and want its reasoning to persist across turns, pass include=["reasoning.encrypted_content"] in provider_kwargs (reasoning models only — other models reject it with a 400). When present, the encrypted reasoning is carried on the assistant message's encrypted_reasoning field: keep the AssistantMessage in context (or round-trip through serialize_conversation / deserialize_conversation, which context.set_completion_messages / get_completion_messages do for you) and it replays on the next turn. The field is dropped when a turn falls back to a non-OpenAI provider.

response = await generate_chat_completion({
    "provider": "openai_responses",
    "model": "gpt-5.4-mini",
    "messages": [SystemMessage(content=SYSTEM_PROMPT), *conversation],
    "provider_kwargs": {"openai_responses": {"include": ["reasoning.encrypted_content"]}},
})

Server-side storage (`store`)#

The proxy sends store=false by default, so OpenAI keeps no server-side copy of the response and the provider stays stateless and fallback-safe. Set store via provider_kwargs to have OpenAI retain the response server-side (about 30 days); the full conversation is still sent inline every turn either way, so this only affects OpenAI-side retention.

"provider_kwargs": {"openai_responses": {"store": True}}

Anthropic Vertex#

Anthropic Vertex runs Anthropic models through Google Cloud. It requires explicit service-account credentials passed via provider_kwargs["anthropic_vertex"].

Setup#

Obtain a GCP service-account JSON key with Vertex AI permissions.
Make it available to your agent via context.variables (e.g. GCP_SERVICE_ACCOUNT_JSON).

Usage#

import json
from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("anthropic_vertex", voicerun_managed=True)

sa_info = json.loads(context.variables.get("GCP_SERVICE_ACCOUNT_JSON"))

response = await generate_chat_completion({
    "provider": "anthropic_vertex",
    "model": "claude-haiku-4-5",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "anthropic_vertex": {
            "region": "us-central1",
            "project_id": "your-gcp-project-id",
            "service_account_credentials": sa_info,
            "thinking": {"type": "disabled"},
        },
    },
})

AWS Bedrock#

Run models through your own AWS Bedrock account. VoiceRun mounts no Bedrock credentials — the caller always supplies AWS auth, so usage is tracked but never billed by VoiceRun. Pass credentials once via configure_provider(credentials=...); they're registered for the session and injected for you, so they never appear on individual completion requests.

Two providers cover the two Bedrock auth mechanisms:

Provider	Models	Auth
`anthropic_bedrock`	Claude (e.g. `us.anthropic.claude-haiku-4-5-20251001-v1:0`)	AWS SigV4 — access key + secret (+ optional session token) + region
`openai_bedrock`	OpenAI on Bedrock (e.g. `openai.gpt-oss-20b-1:0`, `openai.gpt-5.5`)	Amazon Bedrock API key (bearer token) + region

Setup#

In the caller's AWS account, enable model access for the model(s) in the target region (Bedrock console → Model access).
Make credentials available to your agent via context.variables:
- For anthropic_bedrock: an IAM access key + secret with bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream. Long-lived IAM keys are recommended; temporary/STS credentials also work (pass the session token).
- For openai_bedrock: an Amazon Bedrock API key. A long-term key is recommended (a short-term key expires in ~12h).

Claude (`anthropic_bedrock`)#

from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("anthropic_bedrock", credentials={
    "aws_access_key": context.variables.get("AWS_ACCESS_KEY"),
    "aws_secret_key": context.variables.get("AWS_SECRET_KEY"),
    "aws_region": "us-east-2",
    # "aws_session_token": context.variables.get("AWS_SESSION_TOKEN"),  # temporary creds only
})

response = await generate_chat_completion({
    "provider": "anthropic_bedrock",
    "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [{"role": "user", "content": "Hello"}],
})

OpenAI (`openai_bedrock`)#

The Bedrock API key goes in api_key; the region goes in credentials.

configure_provider(
    "openai_bedrock",
    api_key=context.variables.get("BEDROCK_API_KEY"),
    credentials={"aws_region": "us-east-2"},
)

response = await generate_chat_completion({
    "provider": "openai_bedrock",
    "model": "openai.gpt-oss-20b-1:0",
    "messages": [{"role": "user", "content": "Hello"}],
})

Model IDs: pass a valid Bedrock model ID. Newer Claude models generally require a cross-region inference-profile ID (a us. / eu. / global. prefix); the bare anthropic.claude-… ID often errors. List the exact IDs available to you with aws bedrock list-inference-profiles.

gpt-oss reasoning: the open-weight gpt-oss models on Bedrock prepend their chain-of-thought to the response, wrapped in <reasoning>…</reasoning>. Strip it before sending to TTS if you don't want it spoken.

Structured output#

Use response_schema to instruct the model to return JSON matching a JSON Schema. The proxy maps it to each provider's native format:

OpenAI — response_format with json_schema mode
Anthropic — output_config with json_schema format
Google — response_mime_type: "application/json" + sanitized response_schema
Alibaba — same shape as OpenAI
Anthropic Vertex — same shape as Anthropic
Anthropic Bedrock — same shape as Anthropic
OpenAI Bedrock — same shape as OpenAI

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-3.5-flash",
    "messages": [
        {"role": "user", "content": "Invent a fictional person with a name, age, and city."},
    ],
    "response_schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age":  {"type": "integer"},
            "city": {"type": "string"},
        },
        "required": ["name", "age", "city"],
        "additionalProperties": False,
    },
})

import json
person = json.loads(response.message.content)
# {"name": "Elara Voss", "age": 34, "city": "Portland"}

response_schema is inherited by fallbacks unless the fallback sets its own, and Google's sanitizer runs automatically when the request lands on the Google provider:

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-4.1-mini",
    "messages": [{"role": "user", "content": "Invent a person."}],
    "response_schema": {
        "type": "object",
        "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
        "required": ["name", "age"],
        "additionalProperties": False,
    },
    "fallbacks": [
        {"provider": "google", "model": "gemini-2.0-flash"},
    ],
})

See JSON Schema support for the cross-provider compatibility matrix.

Next steps#

API Reference — full type documentation
Examples — full handlers putting these features together