# Speech

Hosted **VUI** — expressive, low-latency text-to-speech. Send text, get back audio in a natural voice, with support for non-verbal cues like `[sigh]` and `[laugh]`.

Two ways to render text:

- **HTTP** (`POST /v1/tts`) — one request, one render. Simplest to integrate.
- **WebSocket** (`/v1/tts/ws`) — keep a warm socket open across renders so each one skips the TLS/TCP handshake and reaches first audio sooner. Use this for interactive UIs.

**Pricing:** $9 per 1M characters (≈ $0.48 per hour of audio). See [pricing](/pricing).

## Base URL

Speech is served through the unified Fluxions API gateway under the `/vui` namespace:

```
https://api.fluxions.ai/vui
```

## Authentication

All speech endpoints require your API key in the `Authorization` header — see [Overview](#overview) for API key setup.

Over WebSocket the key rides on the **connection**: send it as the `Authorization` header on the handshake, or as `?token=YOUR_API_KEY` on the URL from a browser.

## GET /voices — List Voices

List the built-in voices.

### Request

**Bash (.sh)**
```bash
curl "https://api.fluxions.ai/vui/voices" \
  -H "Authorization: YOUR_API_KEY"
```

**Python (.py)**
```python
import requests

voices = requests.get(
    'https://api.fluxions.ai/vui/voices',
    headers={'Authorization': 'YOUR_API_KEY'},
).json()['voices']
for v in voices:
    print(v['voice_id'], '—', v['preview_text'][:50])
```

**JavaScript (.js)**
```javascript
const { voices } = await fetch('https://api.fluxions.ai/vui/voices', {
  headers: { Authorization: 'YOUR_API_KEY' },
}).then(r => r.json());
voices.forEach(v => console.log(v.voice_id, '—', v.preview_text.slice(0, 50)));
```

### Response

```json
{
  "voices": [
    { "voice_id": "maeve.h8ff7e07da", "preview_text": "Hi there. Here's a quick sample of my voice — nice to meet you." },
    { "voice_id": "abraham.h8ff7e07da", "preview_text": "Hi there. Here's a quick sample of my voice — nice to meet you." },
    { "voice_id": "harry.h8ff7e07da", "preview_text": "Hi there. Here's a quick sample of my voice — nice to meet you." }
  ]
}
```

Pass any `voice_id` as the `voice` field when rendering. `preview_text` is what that voice's sample says: `GET /vui/voices/<voice_id>/sample.wav`.

## POST /v1/tts — Render (HTTP)

Synthesize speech from text. Returns a complete WAV by default, or streams audio chunk-by-chunk when `stream=1`.

### Parameters

JSON body:

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `voice` | string | *(required)* | A `voice_id` from `GET /voices` |
| `input` | string | *(required)* | Text to speak. Supports non-verbal cues (see below) |
| `response_format` | string | `"wav"` | `"wav"` (complete file) or `"pcm"` (raw s16le @ 24 kHz) |
| `stream` | boolean | `false` | Stream audio as it's generated instead of buffering the whole file |
| `max_secs` | float | *(auto)* | Hard ceiling on output length. Auto-estimated from text length when omitted |
| `verify_chunks` | boolean | `false` | Re-checks each rendered chunk with a fast speech-to-text pass and re-renders any that misread the text. Improves reliability at the cost of latency; leave it off for the lowest-latency stream (see [Streaming](#streaming)) |

### Request

**Bash (.sh)**
```bash
curl -X POST "https://api.fluxions.ai/vui/v1/tts" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"voice": "maeve.h8ff7e07da", "input": "[sigh] fine, I will say it one more time."}' \
  --output speech.wav
```

**Python (.py)**
```python
import requests

r = requests.post(
    'https://api.fluxions.ai/vui/v1/tts',
    headers={'Authorization': 'YOUR_API_KEY'},
    json={'voice': 'maeve.h8ff7e07da', 'input': '[sigh] fine, I will say it one more time.'}
)
with open('speech.wav', 'wb') as f:
    f.write(r.content)
```

**JavaScript (.js)**
```javascript
const r = await fetch('https://api.fluxions.ai/vui/v1/tts', {
  method: 'POST',
  headers: { Authorization: 'YOUR_API_KEY', 'Content-Type': 'application/json' },
  body: JSON.stringify({ voice: 'maeve.h8ff7e07da', input: '[sigh] fine, I will say it one more time.' })
});
const wav = await r.blob();
const url = URL.createObjectURL(wav);
new Audio(url).play();
```

### Response

`200 OK` with the audio bytes. `Content-Type` is `audio/wav` (or `audio/L16` when `response_format` is `"pcm"`).

### Streaming

Add `stream=1` (query param or body field) to receive audio as it's generated, delivered as chunked transfer encoding.

By default each chunk streams the instant the model produces it, so first bytes land within ~80 ms. Set `verify_chunks: true` to have each chunk checked — and re-rendered if it misreads the text — *before* it streams: first audio then lands once the first chunk is rendered and verified (~0.5–1 s for a typical sentence), and later chunks are checked while the next one renders.

**Bash (.sh)**
```bash
curl -X POST "https://api.fluxions.ai/vui/v1/tts?stream=1" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"voice": "maeve.h8ff7e07da", "input": "Streaming starts playing almost immediately."}' \
  --output speech.wav
```

**Python (.py)**
```python
import requests

with requests.post(
    'https://api.fluxions.ai/vui/v1/tts?stream=1',
    headers={'Authorization': 'YOUR_API_KEY'},
    json={'voice': 'maeve.h8ff7e07da', 'input': 'Streaming starts playing almost immediately.'},
    stream=True,
) as r, open('speech.wav', 'wb') as f:
    for chunk in r.iter_content(chunk_size=8192):
        f.write(chunk)
```

**JavaScript (.js)**
```javascript
const r = await fetch('https://api.fluxions.ai/vui/v1/tts?stream=1', {
  method: 'POST',
  headers: { Authorization: 'YOUR_API_KEY', 'Content-Type': 'application/json' },
  body: JSON.stringify({ voice: 'maeve.h8ff7e07da', input: 'Streaming starts playing almost immediately.' }),
});
const reader = r.body.getReader();
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  // `value` is a Uint8Array chunk of the streaming WAV — append or play as it arrives
}
```

## WebSocket /v1/tts/ws — Render (warm socket)

Identical render logic to `POST /v1/tts`, but the socket stays open between renders. Hold it open and the TLS/TCP/tunnel handshake is paid once — each subsequent `speak` goes straight to synthesis. Ideal for typing UIs or back-to-back lines.

Audio is delivered as **binary frames of s16le PCM, mono, 24 kHz** (no WAV header — assemble it yourself if you need a file).

### Protocol

**Client → server** (text JSON):

```json
{ "type": "speak", "voice": "<id>", "input": "<text>", "max_secs": 0, "verify_chunks": false }
{ "type": "session.close" }
```

`max_secs` and `verify_chunks` are optional. `verify_chunks` defaults to `false` (lowest latency); set it `true` to check each chunk before it plays (see [Streaming](#streaming)).

**Authentication.** The key is checked once, when the socket opens — send `Authorization: YOUR_API_KEY` on the handshake. Browsers can't set headers on a WebSocket, so from browser JS pass it on the URL instead: `wss://api.fluxions.ai/vui/v1/tts/ws?token=YOUR_API_KEY`. Every `speak` on that socket is then authenticated, including renders with your own cloned voices.

**Server → client:**

| Message | Meaning |
|---------|---------|
| `{"type": "start"}` | The worker stream opened — audio frames follow |
| *(binary frame)* | A chunk of s16le PCM @ 24 kHz |
| `{"type": "done"}` | Current render finished — socket stays open for the next `speak` |
| `{"type": "error", "message": "..."}` | Render failed (socket stays open) |

One render = one `speak` → `start` → binary PCM* → `done`. Send another `speak` on the same socket to render again.

### Request

**JavaScript (.js)**
```javascript
const ws = new WebSocket('wss://api.fluxions.ai/vui/v1/tts/ws?token=YOUR_API_KEY');
ws.binaryType = 'arraybuffer';

const chunks = [];
ws.addEventListener('open', () => {
  ws.send(JSON.stringify({ type: 'speak', voice: 'maeve.h8ff7e07da', input: '[laugh] oh, you are serious?' }));
});
ws.addEventListener('message', (ev) => {
  if (typeof ev.data === 'string') {
    const m = JSON.parse(ev.data);
    if (m.type === 'done') {
      // chunks now hold the full s16le PCM @ 24 kHz — feed to WebAudio or wrap in a WAV
      ws.send(JSON.stringify({ type: 'session.close' }));
    }
    return;
  }
  chunks.push(new Int16Array(ev.data)); // raw PCM frame
});
```

**Python (.py)**
```python
import asyncio, json, websockets

async def render(text, voice='maeve.h8ff7e07da'):
    pcm = bytearray()
    async with websockets.connect(
        'wss://api.fluxions.ai/vui/v1/tts/ws',
        additional_headers={'Authorization': 'YOUR_API_KEY'},
    ) as ws:
        await ws.send(json.dumps({'type': 'speak', 'voice': voice, 'input': text}))
        async for msg in ws:
            if isinstance(msg, bytes):
                pcm += msg                      # s16le PCM @ 24 kHz
            elif json.loads(msg)['type'] == 'done':
                break
    return bytes(pcm)

audio = asyncio.run(render('[sigh] so you want to force me to say things.'))
```

### Conversation context

By default every `speak` renders from the voice alone. For a voice agent, open the socket in **conversation mode** instead: the voice then hears the conversation so far — what the caller said and what it said itself — and replies in context, the way a person carries tone and pacing from one turn to the next.

Send `session.update` as the first message. The voice is fixed for the life of the socket.

```json
{ "type": "session.update", "voice": "<id>", "context": true, "language": "fr" }
```

`language` is optional. The server answers `{"type": "session.updated", "voice": "<id>", "context": true, "caller_audio": true}` — `caller_audio` says whether this server uses caller audio (if `false`, send text only; binary frames are ignored). Then:

| Client → server | Meaning |
|-----------------|---------|
| *(binary frames)* | Optional: the caller's audio, s16le PCM, mono, **16 kHz**, streamed while they speak — it's encoded as it arrives, so the reply doesn't wait for it. Send only the caller's speech (after your VAD), not an open mic. |
| `{"type": "user", "text": "..."}` | Ends the caller's turn: the text (your transcript), plus any audio streamed since the voice's last reply started. Up to the last 10 s of a turn's audio is used. |
| `{"type": "speak", "input": "..."}` | Your reply. Several `speak`s in a row (e.g. one per sentence from your LLM) form one reply until the next `user`. |
| `{"type": "interrupt", "played_ms": 1200}` | Barge-in: stop, and keep only the part of the reply the caller actually heard. `played_ms` counts the reply's audio as played, across its `speak`s. |
| `{"type": "reset"}` | Forget the conversation and start again from the voice. |
| `{"type": "session.close"}` | Close. |

The server replies to each `speak` with `start`, binary PCM and `done`, as above. An `interrupt` is answered with `{"type": "interrupted", "kept_ms": ...}`: what stays in context is cut on a phrase boundary, so `kept_ms` is at most `played_ms`. A `reset` is answered with `{"type": "reset.done"}`. Errors are `{"type": "error", "code": "...", "message": "..."}`. `bad_request` (a malformed message) and `rate_limited` / `insufficient_credits` (that `speak` was not rendered) leave the socket open; `no_slot` means there was no capacity to open the conversation, and the socket closes — retry shortly.

Messages apply in the order you send them: queue a `user` and three `speak`s back to back and they render in that order. `interrupt` and `reset` also drop any `speak` still waiting its turn.

```python
async with websockets.connect('wss://api.fluxions.ai/vui/v1/tts/ws',
                              additional_headers={'Authorization': 'YOUR_API_KEY'}) as ws:
    await ws.send(json.dumps({'type': 'session.update', 'voice': 'maeve.h8ff7e07da', 'context': True}))
    await ws.send(json.dumps({'type': 'user', 'text': "Hi, I'd like to check my balance."}))
    await ws.send(json.dumps({'type': 'speak', 'input': 'Of course, I can help with that.'}))
    await ws.send(json.dumps({'type': 'speak', 'input': 'Could you give me the name on the account?'}))
    # ... read start / PCM / done for each speak, then the next user turn
```

Audio the caller speaks over a reply (barge-in) belongs to their next turn: stream it, send `interrupt`, then `user` when they finish.

**Caller audio or text only?** Text is enough and is the safer default. Sending the caller's audio as well lets the voice react to how something was said, but a caller's voice can occasionally colour the reply.

## Non-Verbal Cues

Wrap a cue in square brackets inside `input` and the model renders it as an expressive sound rather than reading the word aloud:

| Cue | Effect |
|-----|--------|
| `[sigh]` | Audible sigh |
| `[laugh]` | Laughter |
| `[gasp]` | Sharp intake of breath |
| `[sniff]` | Sniffle |
| `[cough]` | Cough |
| `[hesitate]` | Filler / thinking sound |

**Example**: `"[gasp] you did NOT just put pineapple on that pizza! [laugh] okay, okay."`

## Voices

Render with any built-in voice from `GET /voices`, or **clone** a custom voice from a short reference clip. Cloned voices are private to your account — render them exactly like a built-in one, with your API key on the request.

### POST /v1/voices — Clone a Voice

Upload a reference clip plus its transcript; the model encodes a private voice you can render with. Sent as `multipart/form-data`.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `audio` | file | yes | Reference clip (`wav`/`opus`/etc.). A few clean seconds is enough. Max 25 MB |
| `text` | string | no | Exact transcript of the reference clip. **Omit it and we transcribe the clip for you** before cloning |
| `name` | string | no | Display label (defaults to the filename) |

> Leave `text` out and the server runs your clip through transcription automatically — so the simplest clone is just an `audio` file. Pass `text` yourself when you want exact control over the transcript.

**Bash (.sh)**
```bash
curl -X POST "https://api.fluxions.ai/vui/v1/voices" \
  -H "Authorization: YOUR_API_KEY" \
  -F "audio=@reference.wav" \
  -F "text=This is exactly what the reference clip says." \
  -F "name=My Voice"
```

**Python (.py)**
```python
import requests

r = requests.post(
    'https://api.fluxions.ai/vui/v1/voices',
    headers={'Authorization': 'YOUR_API_KEY'},
    data={'text': 'This is exactly what the reference clip says.', 'name': 'My Voice'},
    files={'audio': ('reference.wav', open('reference.wav', 'rb'), 'audio/wav')},
)
voice_id = r.json()['voice_id']
```

**JavaScript (.js)**
```javascript
const fd = new FormData();
fd.append('audio', fileInput.files[0]);
fd.append('text', 'This is exactly what the reference clip says.');
fd.append('name', 'My Voice');

const { voice_id } = await fetch('https://api.fluxions.ai/vui/v1/voices', {
  method: 'POST',
  headers: { Authorization: 'YOUR_API_KEY' },
  body: fd,
}).then(r => r.json());
```

Response: `{ "voice_id": "u-<user>-<hash>", "name": "My Voice", "frames": 173, "seconds": 13.8 }`. Pass the returned `voice_id` as `voice` in any render call (with your token).

### GET /v1/voices/mine — List Your Cloned Voices

**Bash (.sh)**
```bash
curl "https://api.fluxions.ai/vui/v1/voices/mine" \
  -H "Authorization: YOUR_API_KEY"
```

**Python (.py)**
```python
import requests

voices = requests.get(
    'https://api.fluxions.ai/vui/v1/voices/mine',
    headers={'Authorization': 'YOUR_API_KEY'},
).json()['voices']
```

**JavaScript (.js)**
```javascript
const { voices } = await fetch('https://api.fluxions.ai/vui/v1/voices/mine', {
  headers: { Authorization: 'YOUR_API_KEY' },
}).then(r => r.json());
```

Returns `{ "voices": [ { "voice_id": "u-...-ab12cd34", "name": "My Voice" } ] }`.

### POST /v1/voices/delete — Remove a Cloned Voice

**Bash (.sh)**
```bash
curl -X POST "https://api.fluxions.ai/vui/v1/voices/delete" \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"voice_id": "u-...-ab12cd34"}'
```

**Python (.py)**
```python
import requests

requests.post(
    'https://api.fluxions.ai/vui/v1/voices/delete',
    headers={'Authorization': 'YOUR_API_KEY'},
    json={'voice_id': 'u-...-ab12cd34'},
)
```

**JavaScript (.js)**
```javascript
await fetch('https://api.fluxions.ai/vui/v1/voices/delete', {
  method: 'POST',
  headers: { Authorization: 'YOUR_API_KEY', 'Content-Type': 'application/json' },
  body: JSON.stringify({ voice_id: 'u-...-ab12cd34' }),
});
```

> **Note:** cloned voices currently live in the running server's memory, not a database — they're tied to your account but are not guaranteed to survive a server restart. Re-upload if a `voice_id` stops resolving.

## Output Format

- **Sample rate**: 24,000 Hz
- **Channels**: mono
- **Sample format**: signed 16-bit little-endian PCM
- **HTTP `wav`**: PCM wrapped in a standard WAV container
- **HTTP `pcm`** / **WebSocket binary frames**: raw s16le PCM (no header)