Realtime voice and transcription

Realtime voice and realtime transcription use GET /v1/realtime with a WebSocket upgrade. Send your SpicyAPI key in the Authorization: Bearer header. This is a server-side connection: browsers must connect through your application server, which keeps the API key private.

Choose the connection type

Read GET /api/v1/models?includeSchema=true from the authenticated live model catalog and choose an enabled, available endpoint with the appropriate value in tasks. Use the exact public model identifier returned by that catalog. The examples use MODEL_ID_FROM_CATALOG as a placeholder; replace it before connecting.

Read the selected model's inputSchema and parameter notes before selecting voices, reasoning effort, input encodings or language hints. A voice model's image input capability does not imply image generation. This endpoint supports server WebSockets; WebRTC, SIP and the separate Live API are not provided by this route.

Connect with the official OpenAI JavaScript SDK

Install openai and its ws peer dependency. Set SPICY_API_KEY in your application's secret manager and set SPICY_MODEL to a public realtime model from the catalog. The example requires a voice model whose schema accepts minimal reasoning. It requests one short text response; subsequent responses can use audio output and your own audio player.

import OpenAI from 'openai';
import { OpenAIRealtimeWS } from 'openai/realtime/ws';

const model = process.env.SPICY_MODEL;
if (!model || !process.env.SPICY_API_KEY) throw new Error('Configure the model and API key');
const client = new OpenAI({
  apiKey: process.env.SPICY_API_KEY,
  baseURL: 'https://api.spicyapi.ai/v1',
  maxRetries: 0,
});
const url = new URL('wss://api.spicyapi.ai/v1/realtime');
url.searchParams.set('model', model);
url.searchParams.set('max_tokens', '512');
url.searchParams.set('max_context_tokens', '512');
const connection = new OpenAIRealtimeWS({
  model,
  buildRealtimeURL: () => url,
  options: { headers: { 'Idempotency-Key': crypto.randomUUID() } },
}, client);

let phase = 'initial';
connection.on('session.updated', () => {
  if (phase === 'initial') {
    phase = 'configuring';
    connection.send({
      type: 'session.update',
      session: { type: 'realtime', reasoning: { effort: 'minimal' } },
    });
  } else if (phase === 'configuring') {
    phase = 'responding';
    connection.send({
      type: 'conversation.item.create',
      item: {
        type: 'message', role: 'user',
        content: [{ type: 'input_text', text: 'Say hello in one short sentence.' }],
      },
    });
    connection.send({ type: 'response.create', response: { output_modalities: ['text'] } });
  }
});
connection.on('event', event => {
  if (event.spicy_task_id) console.log('Task:', event.spicy_task_id);
  if (event.type === 'response.output_text.delta') process.stdout.write(event.delta);
  if (event.type === 'response.done') {
    console.log('\nResponse status:', event.response.status);
    connection.close();
  }
});
connection.on('error', () => {
  console.error('Realtime request failed. Check the task and request identifiers.');
  connection.close();
});

Only send settings accepted by the selected model. For audio output, choose output_modalities: ['audio'] and handle response.output_audio.delta chunks in a player that matches the configured encoding. The application controls playback and must stop or flush it when the listener interrupts. Function calls are client-owned: execute the requested function in your application, return a function_call_output item with its call_id, then request the next response.

Transcribe committed audio segments

Choose a catalog endpoint with task realtime-transcription. Build the URL with intent=transcription, its public model, and an integer max_audio_seconds between 1 and 60; the default is 15. Use intent: 'transcription' instead of model in the SDK constructor and pass the URL through buildRealtimeURL as above. The query still requires the public model identifier.

After the initial session.updated, send base64-encoded raw audio in input_audio_buffer.append events, followed by input_audio_buffer.commit. PCM must be 24 kHz, mono, signed 16-bit audio; G.711 µ-law and A-law use 8 kHz. Do not include a WAV header in these events. Input encoding must match the session configuration. Wait for conversation.item.input_audio_transcription.completed before beginning the next segment.

The connection uses explicit commits, including for a model that emits transcript deltas before commit. Automatic VAD transcription and input_audio_buffer.clear are not supported here. Session settings can change between segments and require acknowledgement before sending the next audio. Audio beyond the chosen segment duration is not accepted. A segment reaching its limit is committed automatically.

This connection is distinct from uploading a file to an audio-transcription HTTP API. A short WAV can also be submitted through the ordinary task API when the selected catalog model's schema offers audio_url; follow that schema and obtain a confirmed quote.

Budgets and settlement

Each voice response or committed transcription segment becomes an ordinary task with its own funds admission, hold and final settlement. An idle connection does not reserve the next turn. Voice responses can use the retained conversation history, but the next turn can still fail if its budget, key scope or account funds no longer permit it.

For voice, max_tokens bounds output including reasoning. max_context_tokens bounds retained conversation history; system instructions and tools have a separate enforced allowance. Voice pricing lists text, audio and image input, their cached portions, and text/audio output separately. Text-token rates alone do not describe an audio conversation's total price. Reasoning is already included in text output and is not counted twice.

Transcription uses the final audio-duration usage reported for that model. Delivered byte length is not substituted for measured billing usage. Read the selected model's price and quote rather than estimating a file's bill from its text length.

Events such as response.created and transcription commit/completion carry spicy_task_id. Save that identifier and read GET /api/v1/jobs/recordInfo?taskId=... with the same key to reconcile the task. settled: true distinguishes a final charge from a remaining hold. Missing final usage keeps settlement pending for investigation; it is not an estimate silently charged as actual usage.

Disconnects and connection limits

One connection permits a single active response, up to 256 turns and 30 minutes. JSON frames are limited to 1 MiB; an input audio buffer is limited to 32 MiB, with the selected duration cap also enforced. End the active response with response.cancel when appropriate and continue reading its final status and usage. Closing the socket does not guarantee a free request: already processed work may settle from the recovered final usage.

Choose one Idempotency-Key per logical connection attempt and save the task identifiers immediately. The server derives a distinct key for each turn. Reusing the connection key prevents duplicate admission; it does not recreate a lost live stream. Do not automatically reconnect and resend a billable conversation after an uncertain disconnect. First reconcile the accepted tasks.