Voice conversations: control listening, reasoning, and speech

First read the whole diagram. Expand “Read at full size” to inspect the labels and scroll horizontally, then trace the failure path.

Voice changes the meaning of time

With text, users can read after waiting. With voice, silence and overlapping speech become part of the conversation. A design needs turn ownership, end-of-speech decisions, and interruption handling alongside audio transport. This chapter presents a conceptual architecture, not a provider-specific API contract.

A chained pipeline turns speech into text with STT, generates an answer with an LLM, and synthesizes speech with TTS. A speech-to-speech design centers a model that accepts and emits audio. Neither removes application responsibilities for tools, permissions, and conversation state.

Read at full size (scroll horizontally)

The diagram compares logical responsibilities; it does not claim undocumented model internals. A chain offers observable, replaceable stages but can accumulate recognition errors and waiting. Direct audio processing may preserve vocal information, while requiring a separate decision about what transcripts mean for auditing.

Separate media from control

WebRTC provides mechanisms for transporting media. Signaling exchanges connection information through a separately designed channel, such as HTTP or WebSocket. SDP describes media conditions; ICE gathers and checks candidate network paths. STUN helps discover an externally visible address, while TURN may relay traffic when direct connectivity fails. The WebRTC peer connection guide explains connection and signaling responsibilities.

A browser-to-voice-server WebRTC connection still uses servers. Measure relay usage, bandwidth, packet loss, and microphone availability. The backend verifies conversation access and, where supported by the destination, issues narrowly scoped short-lived credentials. Never place long-lived service secrets in browser code.

Model turns as a state machine

Voice activity detection, or VAD, estimates whether speech is present. It does not guarantee semantic completion. A short silence threshold can split a thought; a long threshold delays a response. Define how silence, explicit user actions, and model estimates establish the end of a turn.

Read at full size (scroll horizontally)

Start with listening → end_detected → generating → speaking, with failure and cancellation transitions. Assign a generation ID. On interruption, advance the generation, request cancellation of the old one, and clear the device playback queue. Reject late audio tagged with the old generation. Server cancellation alone cannot stop audio already waiting on the device.

Cancellation does not undo committed effects

Read at full size (scroll horizontally)

Stopping speech cannot reverse a completed transfer or reservation. Give external operations their own state, deduplication key, and confirmation boundary. Do not treat a conversational interruption as a rollback of every action. The tool execution boundary remains necessary in a voice interface.

After connection loss, authenticate again and query conversation and run state. Decide whether replaying old audio is useful. One option is to show committed text and ask whether to continue. Never repeat external operations merely because transport failed. If the microphone is unavailable, explain the failure and provide a text-input path.

Measure latency by stage

“Response time is one second” cannot locate a bottleneck. Measure from user speech completion to first audible response, then break the path into endpoint detection, upload, recognition, generation, synthesis, delivery, and playback waiting. A hypothetical chained budget of 200, 50, 150, 250, 150, 50, and 50 milliseconds totals 900 milliseconds. This is arithmetic, not a benchmark. Overlapping stages change elapsed time, so do not blindly add their durations in a streaming implementation.

Measure p95 as well as median, accidental interruptions, clipped speech, and time to stop playback across device conditions. Distinguish recognized text, model-generated text, and what was actually played. They need not match. Specify which record is authoritative for each purpose, and design recording consent, retention, and deletion.

Worked design and exercise

Begin with observable chained stages and a coordinator owning turn state and generation IDs. Keep credentials and tool authorization on the backend, separate from the media path. A later change to direct audio processing should preserve turn and external-effect contracts.

Exercise and answer

Old audio arrives after an interruption. Is sending cancellation again sufficient? No: compare its generation ID, discard stale frames, and clear queued playback. If a reservation completed just before the interruption, query the operation record and communicate the committed result rather than pretending cancellation erased it.

These are conceptual reference architectures for learning and design discussions. They do not describe the deployed Kumyu infrastructure.

← Browse all chapters