Advanced Features

Provider-specific kwargs#

Each provider exposes tuning knobs beyond the common set (temperature, max_tokens, etc.). Pass them via provider_kwargs, keyed by provider name. The proxy applies only the entry matching the currently-executing provider; entries for other providers are ignored, which lets you configure kwargs for every member of a fallback chain in one request.

provider_kwargs = {
    "openai":            {...},  # applied when provider == "openai"
    "openai_responses":  {...},  # applied when provider == "openai_responses"
    "anthropic":         {...},  # applied when provider == "anthropic"
    "google":            {...},  # applied when provider == "google"
    "google_vertex":     {...},  # applied when provider == "google_vertex"
    "anthropic_vertex":  {...},  # applied when provider == "anthropic_vertex"
    "anthropic_bedrock": {...},  # applied when provider == "anthropic_bedrock"
    "openai_bedrock":    {...},  # applied when provider == "openai_bedrock"
    "alibaba":           {...},  # applied when provider == "alibaba"
    "deepseek":          {...},  # applied when provider == "deepseek"
    "groq":              {...},  # applied when provider == "groq"
    "voicerun":          {...},  # applied when provider == "voicerun"
}

OpenAI#

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-5-mini",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "openai": {
            "service_tier": "priority",
            "reasoning_effort": "none",
        },
    },
})

See the OpenAI chat/completions reference for the full list.

Anthropic#

response = await generate_chat_completion({
    "provider": "anthropic",
    "model": "claude-sonnet-4-5",
    "max_tokens": 16000,
    "messages": [{"role": "user", "content": "Think about this..."}],
    "provider_kwargs": {
        "anthropic": {
            "thinking": {"type": "enabled", "budget_tokens": 10000},
        },
    },
})

Extended thinking is disabled by default. Set thinking.type to "enabled" to turn it on.

See the Anthropic messages API reference.

Google#

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-2.5-flash",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "google": {
            "thinking_config": {"thinking_budget": 10000, "include_thoughts": True},
            "safety_settings": [...],
        },
    },
})

Thinking is disabled by default on Google models.

See the Google GenerateContent reference.

Alibaba (Qwen via DashScope)#

response = await generate_chat_completion({
    "provider": "alibaba",
    "model": "qwen3.5-plus",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "alibaba": {
            "enable_search": True,
        },
    },
})

Override the regional endpoint with base_url:

"provider_kwargs": {
    "alibaba": {
        "base_url": "https://dashscope-us.aliyuncs.com/compatible-mode/v1",
    },
}

Available regions:

• Singapore (default): https://dashscope-intl.aliyuncs.com/compatible-mode/v1
• Virginia (US): https://dashscope-us.aliyuncs.com/compatible-mode/v1
• Beijing (CN): https://dashscope.aliyuncs.com/compatible-mode/v1

See the DashScope model reference.

Groq#

Groq serves open-weight models (Llama, Qwen, Kimi, GPT-OSS, …) on its own hardware behind an OpenAI-compatible API. It is bring-your-own-key only: there is no VoiceRun-managed Groq key, so register it with your own key — voicerun_managed=True fails at request time. Usage is tracked but never billed by VoiceRun.

configure_provider("groq", api_key=context.variables.get("GROQ_API_KEY"))

response = await generate_chat_completion({
    "provider": "groq",
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "Hello"}],
})

Some Groq ids are vendor-prefixed with a slash — "model": "openai/gpt-oss-120b" is as valid a value as "llama-3.3-70b-versatile". Both are just examples: the proxy applies no id validation to Groq models and forwards the string verbatim. The billable-model check described in Overview never applies here, since every Groq attempt carries your own key.

The endpoint is fixed to https://api.groq.com/openai/v1; a provider_kwargs["groq"]["base_url"] naming any other host is rejected. See the Groq API reference for available models.

Multi-provider example (with fallback)#

configure_provider("openai", voicerun_managed=True)
configure_provider("anthropic", voicerun_managed=True)

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-5-mini",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "openai":    {"service_tier": "flex"},
        "anthropic": {"thinking": {"type": "disabled"}},
    },
    "fallbacks": [
        {"provider": "anthropic", "model": "claude-haiku-4-5"},
    ],
})

When the primary fails and the fallback fires, the proxy applies the anthropic entry instead of the openai one.

Anthropic cache breakpoints#

Anthropic supports prompt caching. The library exposes it via CacheBreakpoint, which attaches to a tool, system message, assistant message, user message, or tool-result message.

The cache is built in order tools → system → messages. Place large, stable content early and put a breakpoint at the end of each cacheable section. When mixing TTLs, the longer duration ("1h") must appear before the shorter ("5m").

from primfunctions.completions import (
    AssistantMessage,
    CacheBreakpoint,
    SystemMessage,
    UserMessage,
    configure_provider,
    generate_chat_completion,
)

configure_provider("anthropic", voicerun_managed=True)

# 1. Tools — cache the full tool block
tools = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up an order by ID",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
        "cache_breakpoint": {"ttl": "1h"},
    },
]

# 2. System (stable) + 3. Messages (growing)
messages = [
    SystemMessage(
        content="You are a customer support agent.\n\n<2000+ token reference doc>",
        cache_breakpoint=CacheBreakpoint(ttl="1h"),
    ),
    UserMessage(content="Can you look up order #1234?"),
    AssistantMessage(content="Let me look that up for you."),
    UserMessage(content="What's the shipping status?"),
]

# Dynamic breakpoint on the last message for the growing conversation prefix
messages[-1].cache_breakpoint = CacheBreakpoint(ttl="5m")

response = await generate_chat_completion({
    "provider": "anthropic",
    "model": "claude-haiku-4-5",
    "messages": messages,
    "tools": tools,
})

Rules#

• At most 4 cache breakpoints per request.
• Longer TTLs (1h) must come before shorter ones (5m) in the prefix order.
• "5m" — 5-minute ephemeral cache, refreshed on each hit.
• "1h" — 1-hour ephemeral cache, higher write cost.
• Each model has a minimum cached-block size. Blocks smaller than the minimum are silently ignored (usage reports cache_creation_input_tokens: 0).
  • Opus / Sonnet: 1024 tokens
  • Haiku 4.5: ~4096 tokens (empirical; higher than older Haikus)

Don't persist breakpoints on stored messages#

Set cache_breakpoint on the last message only right before the call. If you persist it into conversation history and the conversation keeps growing, you end up with stale breakpoints in the middle of the prefix and potentially more than 4 breakpoints — the request will fail. Apply breakpoints dynamically each turn.

Google thought signatures#

Google Gemini maintains context across turns via a thought_signature field on assistant messages and tool calls. The library captures and re-emits it automatically — as long as you keep the same AssistantMessage / ToolCall dataclass around (or round-trip through serialize_conversation / deserialize_conversation), the next turn's request will include the signature.

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-2.5-flash",
    "messages": [...],
})
# response.message.thought_signature is preserved
# on the AssistantMessage dataclass and survives context.set_completion_messages / get.

No handler code is required to propagate it — just reuse the response message objects on the next turn.

OpenAI Responses API (openai_responses)#

openai_responses is a provider that routes to OpenAI's Responses API (/v1/responses) instead of chat completions. You select it like any other provider — configure_provider, generate_chat_completion, tools, streaming, structured output, and fallbacks all work unchanged. The proxy keeps it stateless and fallback-safe (the full conversation is sent inline every turn, exactly like openai).

from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("openai_responses", voicerun_managed=True)

response = await generate_chat_completion({
    "provider": "openai_responses",
    "model": "gpt-5.4-mini",
    "messages": [{"role": "user", "content": user_message}],
    # Disable reasoning for low-latency voice responses.
    "provider_kwargs": {"openai_responses": {"reasoning": {"effort": "none"}}},
})

Reasoning persistence#

If you use a reasoning model and want its reasoning to persist across turns, pass include=["reasoning.encrypted_content"] in provider_kwargs (reasoning models only — other models reject it with a 400). When present, the encrypted reasoning is carried on the assistant message's encrypted_reasoning field: keep the AssistantMessage in context (or round-trip through serialize_conversation / deserialize_conversation, which context.set_completion_messages / get_completion_messages do for you) and it replays on the next turn. The field is dropped when a turn falls back to a non-OpenAI provider.

response = await generate_chat_completion({
    "provider": "openai_responses",
    "model": "gpt-5.4-mini",
    "messages": [SystemMessage(content=SYSTEM_PROMPT), *conversation],
    "provider_kwargs": {"openai_responses": {"include": ["reasoning.encrypted_content"]}},
})

Server-side storage (store)#

The proxy sends store=false by default, so OpenAI keeps no server-side copy of the response and the provider stays stateless and fallback-safe. Set store via provider_kwargs to have OpenAI retain the response server-side (about 30 days); the full conversation is still sent inline every turn either way, so this only affects OpenAI-side retention.

"provider_kwargs": {"openai_responses": {"store": True}}

Google Vertex#

google_vertex runs Gemini through the caller's own Google Cloud Vertex AI project. VoiceRun mounts no Vertex credentials, so usage is tracked but never billed by VoiceRun — and configure_provider("google_vertex", voicerun_managed=True) fails at request time. Register it with credentials= instead.

Three fields are required — service_account_credentials (a parsed GCP service-account JSON dict), project_id, and location (region is accepted as an alias). A request missing any of them raises CompletionsProxyError with error_type == "ConfigurationError".

import json
from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("google_vertex", credentials={
    "service_account_credentials": json.loads(context.variables.get("GCP_SERVICE_ACCOUNT_JSON")),
    "project_id": "your-gcp-project-id",
    "location": "us-central1",
})

response = await generate_chat_completion({
    "provider": "google_vertex",
    "model": "gemini-3.5-flash",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "google_vertex": {
            "thinking_config": {"thinking_budget": 0},
        },
    },
})

Apart from client construction, google_vertex reuses the base google provider end to end — the same request denormalization, schema sanitization, response normalization, and stream handling — so thinking config, tools, and response_schema behave identically to google.

Anthropic Vertex#

Anthropic Vertex runs Anthropic models through Google Cloud. It requires explicit service-account credentials passed via provider_kwargs["anthropic_vertex"].

Setup#

1. Obtain a GCP service-account JSON key with Vertex AI permissions.
2. Make it available to your agent via context.variables (e.g. GCP_SERVICE_ACCOUNT_JSON).

Usage#

import json
from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("anthropic_vertex", voicerun_managed=True)

sa_info = json.loads(context.variables.get("GCP_SERVICE_ACCOUNT_JSON"))

response = await generate_chat_completion({
    "provider": "anthropic_vertex",
    "model": "claude-haiku-4-5",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider_kwargs": {
        "anthropic_vertex": {
            "region": "us-central1",
            "project_id": "your-gcp-project-id",
            "service_account_credentials": sa_info,
            "thinking": {"type": "disabled"},
        },
    },
})

AWS Bedrock#

Run models through your own AWS Bedrock account. VoiceRun mounts no Bedrock credentials — the caller always supplies AWS auth, so usage is tracked but never billed by VoiceRun. Pass credentials once via configure_provider(credentials=...); they're registered for the session and injected for you, so they never appear on individual completion requests.

Two providers cover the two Bedrock auth mechanisms:

Setup#

1. In the caller's AWS account, enable model access for the model(s) in the target region (Bedrock console → Model access).
2. Make credentials available to your agent via context.variables:
   • For anthropic_bedrock: an IAM access key + secret with bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream. Long-lived IAM keys are recommended; temporary/STS credentials also work (pass the session token).
   • For openai_bedrock: an Amazon Bedrock API key. A long-term key is recommended (a short-term key expires in ~12h).

Claude (anthropic_bedrock)#

from primfunctions.completions import configure_provider, generate_chat_completion

configure_provider("anthropic_bedrock", credentials={
    "aws_access_key": context.variables.get("AWS_ACCESS_KEY"),
    "aws_secret_key": context.variables.get("AWS_SECRET_KEY"),
    "aws_region": "us-east-2",
    # "aws_session_token": context.variables.get("AWS_SESSION_TOKEN"),  # temporary creds only
})

response = await generate_chat_completion({
    "provider": "anthropic_bedrock",
    "model": "us.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [{"role": "user", "content": "Hello"}],
})

OpenAI (openai_bedrock)#

The Bedrock API key goes in api_key; the region goes in credentials.

configure_provider(
    "openai_bedrock",
    api_key=context.variables.get("BEDROCK_API_KEY"),
    credentials={"aws_region": "us-east-2"},
)

response = await generate_chat_completion({
    "provider": "openai_bedrock",
    "model": "openai.gpt-oss-20b-1:0",
    "messages": [{"role": "user", "content": "Hello"}],
})

Model IDs: pass a valid Bedrock model ID. Newer Claude models generally require a cross-region inference-profile ID (a us. / eu. / global. prefix); the bare anthropic.claude-… ID often errors. List the exact IDs available to you with aws bedrock list-inference-profiles.

gpt-oss reasoning: the open-weight gpt-oss models on Bedrock prepend their chain-of-thought to the response, wrapped in <reasoning>…</reasoning>. Strip it before sending to TTS if you don't want it spoken.

Structured output#

Use response_schema to instruct the model to return JSON matching a JSON Schema. Every provider supports it — the proxy maps it to each provider's native format:

• OpenAI (openai) — response_format with json_schema mode, strict: True
• OpenAI Responses (openai_responses) — text.format with type: "json_schema", strict: True
• Anthropic (anthropic) — output_config with json_schema format
• Google (google) — response_mime_type: "application/json" + sanitized response_schema
• Google Vertex (google_vertex) — same shape as Google, including the sanitizer
• Anthropic Vertex (anthropic_vertex) — same shape as Anthropic
• Anthropic Bedrock (anthropic_bedrock) — same shape as Anthropic
• OpenAI Bedrock (openai_bedrock) — same shape as OpenAI
• Alibaba (alibaba) — same shape as OpenAI
• DeepSeek (deepseek) — same shape as OpenAI
• Groq (groq) — same shape as OpenAI
• VoiceRun self-hosted (voicerun) — same shape as OpenAI

Every OpenAI-shaped provider in that list emits strict: True uniformly; the Anthropic-shaped providers send the schema without a strict flag, and the Google-shaped ones run the schema through the Google sanitizer first.

response = await generate_chat_completion({
    "provider": "google",
    "model": "gemini-3.5-flash",
    "messages": [
        {"role": "user", "content": "Invent a fictional person with a name, age, and city."},
    ],
    "response_schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age":  {"type": "integer"},
            "city": {"type": "string"},
        },
        "required": ["name", "age", "city"],
        "additionalProperties": False,
    },
})

import json
person = json.loads(response.message.content)
# {"name": "Elara Voss", "age": 34, "city": "Portland"}

response_schema is inherited by fallbacks unless the fallback sets its own, and Google's sanitizer runs automatically when the request lands on the Google provider:

response = await generate_chat_completion({
    "provider": "openai",
    "model": "gpt-4.1-mini",
    "messages": [{"role": "user", "content": "Invent a person."}],
    "response_schema": {
        "type": "object",
        "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
        "required": ["name", "age"],
        "additionalProperties": False,
    },
    "fallbacks": [
        {"provider": "google", "model": "gemini-2.0-flash"},
    ],
})

See JSON Schema support for the cross-provider compatibility matrix.

Next steps#

• API Reference — full type documentation
• Examples — full handlers putting these features together