# Voice and audio

> ElevenLabs conversational voice mode, OpenAI realtime transcription, TTS streaming IPC, microphone permissions on macOS, and optional API keys for audio providers.

- Repository: Parcha-ai/build
- GitHub: https://github.com/Parcha-ai/build
- Human docs: https://grok-wiki.com/public/docs/parcha-ai-build-bea5702b371b
- Complete Markdown: https://grok-wiki.com/public/docs/parcha-ai-build-bea5702b371b/llms-full.txt

## Source Files

- `src/main/services/audio.service.ts`
- `src/main/services/realtime.service.ts`
- `src/main/services/elevenlabs-voice.service.ts`
- `src/main/ipc/audio.ipc.ts`
- `src/main/ipc/voice.ipc.ts`
- `src/renderer/components/chat/VoiceMode.tsx`

---

---
title: "Voice and audio"
description: "ElevenLabs conversational voice mode, OpenAI realtime transcription, TTS streaming IPC, microphone permissions on macOS, and optional API keys for audio providers."
---

Build routes voice and audio through three main-process services (`AudioService`, `RealtimeService`, `ElevenLabsVoiceService`), registered IPC handlers in `audio.ipc.ts`, `realtime.ipc.ts`, and `voice.ipc.ts`, and renderer hooks/components that call `window.electronAPI.audio`, `.realtime`, and `.voice`. Chat voice mode today uses the ElevenLabs Conversational AI SDK over WebRTC; OpenAI powers batch Whisper transcription and a separate Realtime WebSocket path for streaming STT; ElevenLabs also powers message-level TTS streaming over IPC push events.

## Architecture

```mermaid
flowchart TB
  subgraph renderer["Renderer (React)"]
    MB[MicrophoneButton / VoiceMode]
    SDK[useVoiceConversationSDK @elevenlabs/client]
    SB[SpeakerButton]
    AS[audio.store Zustand]
    MB --> SDK
    SB --> AS
    AS --> SB
  end

  subgraph preload["preload.ts → electronAPI"]
    EA[audio.*]
    ER[realtime.*]
    EV[voice.*]
  end

  subgraph main["Main process"]
    AI[audio.ipc.ts]
    RI[realtime.ipc.ts]
    VI[voice.ipc.ts]
    Aud[AudioService]
    RT[RealtimeService]
    EL[ElevenLabsVoiceService]
    AI --> Aud
    RI --> RT
    VI --> EL
  end

  subgraph external["External APIs"]
    OAI_RT[OpenAI Realtime wss]
    OAI_WH[OpenAI Whisper]
    EL_CONV[ElevenLabs ConvAI]
    EL_TTS[ElevenLabs TTS]
  end

  SDK --> EV
  MB --> AS
  SB --> EA
  EA --> AI
  ER --> RI
  EV --> VI
  Aud --> OAI_WH
  Aud --> EL_TTS
  RT --> OAI_RT
  EL --> EL_CONV
  SDK -.WebRTC.-> EL_CONV
```

| Layer | Responsibility |
| --- | --- |
| `AudioService` | Whisper batch STT, ElevenLabs TTS streaming, voice catalog, `audioSettings` in `claudette-settings` |
| `RealtimeService` | OpenAI Realtime WebSocket (`gpt-4o-realtime-preview`), PCM16 input, server VAD, transcription events |
| `ElevenLabsVoiceService` | ElevenLabs ConvAI WebSocket (signed URL), agent prompt PATCH, tool results, status TTS via `[STATUS]` prefix |
| Renderer SDK path | `@elevenlabs/client` `Conversation.startSession` with `connectionType: 'webrtc'` and token from main |

<Note>
Neither Realtime nor ElevenLabs voice services auto-reconnect after disconnect. Reconnection requires an explicit user action (for example toggling the mic), which prevents voice mode from reactivating unintentionally.
</Note>

## API keys and BYOC

Keys live in the `claudette-settings` electron-store. User-provided keys override compile-time embedded keys from `EMBEDDED_KEYS` (injected via Webpack from `.env.production`).

| Store key | Used by | Purpose |
| --- | --- | --- |
| `openAiApiKey` | `AudioService`, `RealtimeService` | Whisper (`whisper-1`) and Realtime transcription |
| `elevenLabsApiKey` | `AudioService`, `ElevenLabsVoiceService` | TTS, ConvAI WebSocket, agent PATCH, conversation token |
| `audioSettings` | Renderer + `AudioService.getAudioSettings()` | Voice ID, agent ID, toggles, trigger word, language |

<ParamField body="elevenLabsAgentId" type="string">
ElevenLabs Conversational AI agent ID (`agent_...`). Required for voice mode; stored inside `audioSettings`.
</ParamField>

<ParamField body="voiceModeEnabled" type="boolean" default="true">
Feature toggle in Settings. Default in `DEFAULT_AUDIO_SETTINGS` is `true`.
</ParamField>

Settings UI (`SettingsDialog`) saves OpenAI and ElevenLabs keys through `window.electronAPI.audio.setOpenAiKey` / `setElevenLabsKey`, and merges other audio fields via `audio.setSettings` → `AUDIO_SETTINGS_SET`.

## ElevenLabs conversational voice mode

Primary chat integration: `MicrophoneButton` in `InputArea` (wrapped in `VoiceModeErrorBoundary`) uses `useVoiceConversationSDK`.

<Steps>
<Step title="Configure agent and keys">
Set **ElevenLabs API Key** and **ElevenLabs Agent ID** in Settings. Enable **Voice mode** if disabled.
</Step>
<Step title="Connect">
Click the phone/mic control. On macOS, Build checks `audio.checkMicrophonePermission` and may call `audio.requestMicrophonePermission` before connecting.
</Step>
<Step title="Run WebRTC session">
The hook calls `voice.getConversationToken({ agentId })`, then `Conversation.startSession({ conversationToken, connectionType: 'webrtc' })`. The SDK owns mic capture and hardware echo cancellation.
</Step>
<Step title="Finalize transcript into chat">
Final user transcripts invoke `onTranscriptionComplete` in `InputArea`, which can auto-submit when the utterance ends with the configured trigger word (default `please`).
</Step>
</Steps>

While connected, `MicrophoneButton` sets per-session `audioModeActive` so TTS auto-play does not compete with ConvAI audio (`triggerAutoPlayTTS` skips when voice mode is connected). It also sends `voice.sendContextUpdate`, `voice.sendUserActivity`, and `voice.sendText` for Build progress, thinking narration (`[THINKING]` in the system prompt), and permission announcements.

### Legacy main-process WebSocket path

`ElevenLabsVoiceService` + `voice.ipc.ts` still expose full ConvAI control: `VOICE_CONNECT`, `VOICE_SEND_AUDIO` (PCM16 `Int16Array` as number[]), `VOICE_USER_TRANSCRIPT`, `VOICE_AUDIO_CHUNK`, `VOICE_TOOL_CALL`, etc. The older `useVoiceConversation` hook uses this path with renderer-side mic streaming. The SDK/WebRTC path is what `MicrophoneButton` and `VoiceMode.tsx` use today.

`clearAudioBuffer()` on the ElevenLabs service is a documented no-op; echo prevention relies on client-side mic muting during agent playback.

## OpenAI Realtime transcription

`RealtimeService` opens `wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview` with headers `Authorization: Bearer <key>` and `OpenAI-Beta: realtime=v1`. After connect it sends `session.update` with:

- `modalities: ['text']` (no model audio output)
- `input_audio_format: 'pcm16'`
- `input_audio_transcription.model: 'whisper-1'`
- `turn_detection`: server VAD (`threshold: 0.3`, `silence_duration_ms: 800`)

Main relays events to the renderer:

| Push channel | Event |
| --- | --- |
| `realtime:transcription-delta` | Partial transcript text |
| `realtime:transcription-completed` | Final segment |
| `realtime:speech-started` / `realtime:speech-stopped` | VAD boundaries |
| `realtime:error` | API error message |
| `realtime:disconnected` | Socket closed |

Invoke handlers: `realtime:connect`, `realtime:disconnect`, `realtime:send-audio` (Int16 LE buffer), `realtime:commit-audio`, `realtime:clear-audio`.

`useAudioRecorder` implements the renderer side (24 kHz `AudioContext`, `ScriptProcessorNode`, resampling to PCM16). It is not wired into current chat UI components; it remains the reference integration for Realtime STT.

## AudioService: Whisper STT and ElevenLabs TTS

### Batch transcription (Whisper)

`AUDIO_TRANSCRIBE` accepts an `ArrayBuffer` of `audio/webm`, calls `openai.audio.transcriptions.create` with `model: 'whisper-1'` and optional `language` (defaults to `en`). Returns `{ success, result: { text, partial: false } }` or `{ success: false, error }`. No renderer caller is present in the current tree; the IPC surface is available for future or custom UI.

### TTS streaming

`AUDIO_TTS_STREAM` is invoked with a `TTSRequest`:

| Field | Type | Notes |
| --- | --- | --- |
| `text` | string | Source text |
| `messageId` | string | Stream and cancel key |
| `voiceId` | string | ElevenLabs voice; falls back to `audioSettings.selectedVoice` |
| `modelId` | string | Default `eleven_turbo_v2_5` in service |

The handler async-iterates `generateTTSStream`, pushing:

- `audio:tts-chunk` — `{ messageId, chunk: number[] }` (raw bytes as JSON array)
- `audio:tts-complete` — `{ messageId }`
- `audio:tts-error` — `{ messageId, error }`

`AUDIO_TTS_CANCEL` aborts via per-`messageId` `AbortController`. `SpeakerButton` decodes accumulated chunks in a shared `AudioContext` when `progress === 100`. `session.store` calls `triggerAutoPlayTTS` after assistant messages when `audioModeActive[sessionId]` is true and voice mode is not connected.

Default voice in `DEFAULT_AUDIO_SETTINGS`: `EXAVITQu4vr4xnSDxMaL` (Rachel).

## IPC and preload surface

Handlers register in `src/main/index.ts` via `registerAudioHandlers`, `registerRealtimeHandlers`, and `registerVoiceHandlers`.

### Audio (`window.electronAPI.audio`)

| Invoke | Channel | Returns / behavior |
| --- | --- | --- |
| `transcribe` | `audio:transcribe` | Whisper result |
| `streamTTS` | `audio:tts-stream` | Starts stream; chunks via `onTTSChunk` |
| `cancelTTS` | `audio:tts-cancel` | Aborts stream |
| `getVoices` | `audio:get-voices` | ElevenLabs voice list |
| `getSettings` / `setSettings` | `audio:settings-get/set` | `AudioSettings` object |
| `getElevenLabsKey` / `setElevenLabsKey` | `audio:get/set-elevenlabs-key` | Raw key string |
| `getOpenAiKey` / `setOpenAiKey` | `audio:get/set-openai-key` | Raw key string |
| `checkMicrophonePermission` | `audio:check-microphone-permission` | macOS `systemPreferences` status |
| `requestMicrophonePermission` | `audio:request-microphone-permission` | `askForMediaAccess('microphone')` |

Subscribe helpers: `onTTSChunk`, `onTTSComplete`, `onTTSError`.

### Realtime (`window.electronAPI.realtime`)

Invoke: `connect`, `disconnect`, `sendAudio`, `commitAudio`, `clearAudio`. Subscribe: `onConnected`, `onDisconnected`, `onTranscriptionDelta`, `onTranscriptionCompleted`, `onSpeechStarted`, `onSpeechStopped`, `onError`.

### Voice (`window.electronAPI.voice`)

Invoke includes `connect`, `disconnect`, `sendAudio`, `sendText`, `endInput`, `clearAudioBuffer`, `sendContextUpdate`, `sendToolResult`, `updateAgentPrompt`, `sendUserActivity`, `getSignedUrl`, `getConversationToken`. Subscribe: `onConnected`, `onDisconnected`, `onUserTranscript`, `onAgentResponse`, `onAudioChunk`, `onInterruption`, `onError`, `onToolCall`, `onReconnecting`.

For a full channel listing, see the IPC channels reference page.

## Renderer state (`audio.store`)

Zustand store tracks:

- `recordingStates` — per-session mic/recording (Realtime hook)
- `ttsStates` — per-message chunk playback
- `voiceModeStates` — per-session ConvAI UI state
- `audioModeActive` — enables auto TTS after assistant replies

`initializeTTSListeners()` registers global `onTTSChunk` / `onTTSComplete` / `onTTSError` once (HMR-safe via `window.__ttsListenerCleanups`).

## macOS microphone permissions

On `darwin`, `audio.ipc.ts` uses Electron `systemPreferences`:

| Status | `checkMicrophonePermission` | `requestMicrophonePermission` |
| --- | --- | --- |
| `granted` | `granted: true` | Returns success immediately |
| `not-determined` | `canRequest: true` | Calls `askForMediaAccess('microphone')` |
| `denied` / `restricted` | `canRequest: false` | Error directing user to System Settings → Privacy & Security → Microphone |

Non-macOS platforms return `{ status: 'granted', granted: true }`; renderer still uses `navigator.mediaDevices.getUserMedia` where applicable.

Electron `session.defaultSession` permission handlers allow `media` (microphone/camera) for the main window. Packaged macOS builds can include `com.apple.security.device.audio-input` in `entitlements.mac.plist`; distribution signing in `forge.config.ts` currently references `entitlements.plist` (network/JIT only). If mic access fails only in signed builds, verify the active entitlements file matches production needs.

## `AudioSettings` reference

Stored under `audioSettings` in `claudette-settings` (see `src/shared/types/audio.ts`):

| Field | Default | Role |
| --- | --- | --- |
| `selectedVoice` | Rachel voice ID | ElevenLabs TTS voice |
| `voiceSettings` | stability 0.5, similarity 0.75, etc. | Passed to `textToSpeech.convertAsStream` |
| `autoPlayResponses` | `false` | Setting exists; auto-play is gated by `audioModeActive` in practice |
| `transcriptionLanguage` | `en` | Whisper language |
| `voiceTriggerWord` | `please` | Auto-submit when final transcript matches trailing pattern |
| `elevenLabsAgentId` | `undefined` | ConvAI agent |
| `voiceModeEnabled` | `true` | Settings toggle |
| `ralphLoopEnabled` | `false` | Build It loop (orthogonal but stored with audio settings) |
| `computerUseEnabled` | `false` | Computer Use toggle |
| `maxComputerUseIterations` | `20` | Computer Use cap |

## Troubleshooting

<Warning>
**OpenAI API key not configured** — Realtime connect and Whisper transcribe throw if neither user nor `EMBEDDED_OPENAI_API_KEY` is set.
</Warning>

<Warning>
**ElevenLabs API key not configured** — TTS, voice list, ConvAI connect/token, and agent prompt update require `elevenLabsApiKey` or embedded key.
</Warning>

| Symptom | Likely cause | What to check |
| --- | --- | --- |
| Voice button disabled | Missing `elevenLabsAgentId` | Settings → ElevenLabs Agent ID |
| Mic works in dev, not production | macOS TCC / entitlements | System Settings microphone list; signing entitlements |
| No auto-read of replies | `audioModeActive` false or voice connected | Connect voice mode or enable audio mode path in chat |
| Duplicate TTS + agent speech | Both ConvAI and `streamTTS` active | Voice connected skips `triggerAutoPlayTTS` by design |
| Realtime never used in UI | Hook not mounted | `useAudioRecorder` exists; chat uses ElevenLabs SDK |
| Quota errors in voice mode | ElevenLabs billing | WebSocket close reason or SDK `handleErrorEvent` patch messages |

## Related pages

<CardGroup>
<Card title="Configure API keys and providers" href="/configure-providers">
Store OpenAI and ElevenLabs keys alongside other provider credentials.
</Card>
<Card title="IPC and preload bridge" href="/ipc-bridge">
How `electronAPI` maps invoke handlers and push events.
</Card>
<Card title="IPC channels reference" href="/ipc-channels-reference">
Full `audio:*`, `realtime:*`, and `voice:*` channel catalog.
</Card>
<Card title="Settings reference" href="/settings-reference">
`claudette-settings` keys including `audioSettings`.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
PATH, ports, and production log locations when audio features fail.
</Card>
</CardGroup>