Real-Time Voice AI with LiveKit, WebRTC and FastAPI

A voice agent feels intelligent only when it feels fast. Latency, not model quality, is usually what makes a demo feel robotic. This article covers the architecture I use to build real-time voice AI systems with LiveKit, WebRTC and a FastAPI backend.

Why WebRTC and LiveKit

HTTP is the wrong transport for voice. WebRTC is built for real-time audio: it handles jitter, packet loss and echo cancellation, and it keeps a persistent, low-latency connection. LiveKit gives you a managed SFU and clean server SDKs so you do not have to write the signalling layer yourself.

Sub-second audio transport instead of request/response round trips.
Built-in echo cancellation, noise suppression and device handling.
A room model that scales from one caller to many participants.
Server-side SDKs in Python so the agent logic stays in one place.

The latency budget

Human conversation tolerates roughly 300 to 500 milliseconds of turn-taking delay before it feels unnatural. That budget is split across speech-to-text, the LLM and text-to-speech, so every stage has to stream.

Capture: browser or telephony sends audio to the room.
STT: stream partial transcripts as the user speaks.
LLM: start generating as soon as a final transcript arrives.
TTS: synthesise the first sentence while the model is still writing.
Playback: push audio back into the room immediately.

Turn detection and interruptions

Fixed silence timers feel clumsy. A better approach combines voice activity detection with a semantic check: is the user actually done, or just pausing? Crucially, when a user speaks over the agent, you must cancel the in-flight generation and stop playback. Barge-in is what makes a voice agent feel like a conversation rather than a recording.

Connecting a LiveKit agent to FastAPI

Keep the real-time loop close to LiveKit and put business logic behind a FastAPI service. The agent streams audio and transcripts; FastAPI handles retrieval, tool calls and persistence.

from livekit.agents import JobContext, WorkerOptions, cli
from livekit.agents.voice import Agent, AgentSession
from livekit.plugins import deepgram, openai, silero

async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    session = AgentSession(
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-2"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=openai.TTS(voice="alloy"),
    )

    agent = Agent(instructions="You are a helpful support agent. Be concise.")
    await session.start(room=ctx.room, agent=agent)

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

The FastAPI side stays a normal, testable service. The agent calls it for anything stateful — looking up an order, booking an appointment or fetching knowledge for a RAG step.

import httpx

async def lookup_order(order_id: str) -> dict:
    async with httpx.AsyncClient(timeout=2.0) as client:
        response = await client.get(f"https://api.example.com/orders/{order_id}")
        response.raise_for_status()
        return response.json()

What breaks in production

Network jitter: always degrade gracefully rather than freezing the call.
Cold starts: keep workers warm, because a slow first token ruins the first impression.
Accents and background noise: test STT on real audio, not studio samples.
Tool latency: give the agent something to say while a backend call is in flight.
Cost: streaming STT and TTS are billed per second, so measure both.