Building an AI Calling Platform with Python, LiveKit and Telephony

An AI calling platform is more than a voice agent with a phone number attached. It is telephony, media transport, agent logic and call state working together — plus the compliance rules that come with handling real phone calls. This is the architecture I use to build them.

The five layers

Telephony: a phone number and a SIP trunk from a carrier.
Media transport: a WebRTC room that carries the audio.
Agent: speech-to-text, the LLM and text-to-speech in a loop.
Call state: who called, when, why and what happened.
Compliance: consent, recording disclosure, retention and opt-out.

Telephony: getting calls in and out

You need a carrier that can route a phone number to a SIP endpoint. Common choices are Twilio, Telnyx and Plivo. The carrier does not run your agent; it hands the audio to your media layer over SIP and gets audio back.

Inbound: the carrier forwards calls to your SIP endpoint.
Outbound: your platform originates a call through the carrier API.
Always buy numbers in the regions you actually serve.
Test DTMF, call transfers and voicemail detection early.

Wiring telephony into LiveKit

LiveKit provides a SIP bridge so that a phone call becomes a room participant. That means your agent code does not need to know whether the caller is in a browser or on a phone — it just joins a room and speaks.

Originating a call is a normal API call to your carrier or your own orchestration service.

import httpx

async def start_outbound_call(to_number: str, agent_id: str) -> str:
    payload = {"to": to_number, "from": OUTBOUND_NUMBER, "agent": agent_id}
    async with httpx.AsyncClient(timeout=5.0) as client:
        response = await client.post(
            "https://api.carrier.example/v1/calls",
            json=payload,
            headers={"Authorization": f"Bearer {CARRIER_TOKEN}"},
        )
        response.raise_for_status()
        return response.json()["call_id"]

Call state is the hard part

Voice agents fail in messy ways: the caller hangs up mid-sentence, the network drops, or two webhooks arrive for the same call. Model call state explicitly so the system can recover.

Store one row per call with a status: queued, ringing, in_progress, completed, failed.
Treat webhooks as at-least-once: make handlers idempotent with a unique event ID.
Persist transcripts and outcomes against the call ID.
Expire stale calls with a scheduled job instead of leaving rows open forever.

Compliance is not optional

Automated calls are regulated. Depending on where your callers are, you may need prior consent, a recording disclosure, an opt-out mechanism and a defined retention period. In Pakistan, telemarketing and recording are governed by PTA rules; in the EU and UK, GDPR applies to the personal data in every call.

Disclose automated calling and recording at the start of the call.
Maintain a do-not-call list and honour opt-outs immediately.
Encrypt recordings and set a deletion schedule.
Log consent with a timestamp and the exact wording shown.

Costs and scaling

Voice is billed by the second across several layers: telephony minutes, STT, TTS and LLM tokens. Measure the cost per call as a first-class metric, because latency and cost both scale linearly with talk time. Keep workers warm to avoid cold-start silence, and cap concurrent calls per worker so one busy period cannot starve the rest.