What I built

A real-time voice loop, and the instruments to measure it. One agent process joined a room. It heard speech, turned it into text, asked a language model for a reply, and spoke the reply back. Around that loop I put a collector that graded every stage against a target, and a small manager that noticed when the conversation was over.

Deployment was a single Docker container on a Hetzner server. The agent ran as one process: audio in, a model call, audio out, with the measurements collected in the same process.

Use cases

• A voice interface that answers and acts. The classic case, and still the best one.
• Anything where waiting is the whole problem. A voice loop is judged on how fast it starts speaking, not on how clever the answer is.
• A test bench for real-time voice stacks. This was the real use. I wanted to know where the milliseconds went, so I built something that printed them.

Architecture

The system had two paths, not one.

The audio path was the product: room, speech gate, transcript, model, voice, and back to the room. The metrics path ran beside it. Every stage reported its timings as events. The collector read those events and decided what to keep. The framework produced the numbers. My code only stored them and graded them.

Three details made the loop work in real time.

• The gate came before the transcript. Silero VAD decided whether sound was speech before anything downstream ran, so the expensive stages never saw silence.
• The transcript streamed. The text was not complete when the model started. It began on the first stable words and finished while the person was still finishing their sentence.
• The reply spoke as it was written. Text to speech started on the first phrase of the answer, not on the last. The end of the model's output arrived while the voice was already talking.

That overlapping was the whole trick. In a serial pipeline the wait times stack: hear, wait, read, wait, think, wait, speak. Here the stages ran on top of each other, so the total wait was close to the slowest stage, not the sum of all of them. The diagram above is the data flow. The timing was this:

The stages

• LiveKit — rooms and audio transport. It carried the audio and held the session state.
• Silero VAD — voice activity detection. It decided when speech started and when it stopped. Without it the assistant transcribed silence and cut people off mid-sentence.
• Azure Speech — speech to text, streaming, so the transcript was ready while the person was still talking.
• GPT-4o mini — the reply, called directly from OpenAI.
• Text to speech — the voice. I used Azure Neural first, then switched to ElevenLabs for the better sound, at a higher price and higher latency.

Two voices

I started with Azure Neural TTS. It shared credentials with the speech service, it was already in the same region as the transcript, and it cost almost nothing per minute. Then I switched to ElevenLabs, because the voice was clearly better.

That switch was a straight trade. The better voice cost roughly $0.18 for every minute of spoken audio, and the extra network hop added to the time before the first audio byte arrived. The saving was that people liked hearing it. Neither option dominated the other. It depended on whether the call was a test or a product.

The price range for one minute of talking was therefore about $0.01 to $0.18, and the speech stage was the largest line in the cost of the whole system. Both figures are approximate list prices from the time, not invoices.

The measurement harness

It measured how long the speech gate took, how long the pause was after someone stopped talking, transcription time, model time, and the number that mattered most: the time to the first audio byte from the speech service.

It graded itself against three targets. End-of-utterance delay under 500 ms. First audio byte under 300 ms. The whole turn under 1200 ms. It printed a session summary and a row for every turn, then wrote both to markdown files.

Two honest notes.

The total was not real end-to-end latency. It added stages that already overlapped in time, so it was an upper bound. And the logs were written next to the process, so they are gone now. There are no numbers on this page.

One thing I learned

Real-time voice is not one latency. It is five, and they hide in different places. Measuring each stage turned "it feels slow" into a list of suspects. The framework already produced the timings; all the value came from deciding to look at them.