Why Voice AI Audio Gets Choppy: A Guide to RTP Playout Pacing

One of the hardest problems teams face when they run phone calls on real-time voice models is the metallic breakup and clipped syllables that appear even when network measurements look clean. Many teams, out of VoIP habit, assume it is network jitter or packet loss and look for the fix in the wrong layer.

Measurements on live calls by the Yanitly voice team showed that choppy audio in real-time AI calls usually comes not from network jitter but from the gap between how fast the voice model sends audio and how fast the phone line consumes it. This article explains the cause, the pacing approach that closes the gap, and the design choices that speed up call setup, at a conceptual level. This infrastructure runs underneath our AI call center solution.

In real-time voice AI, dropping audible speech to shrink a buffer clips syllables; corrections should only ever happen during silence.

Diagnosis: Why Choppy Audio Is Usually Not Jitter

In a classic phone call, the devices at both ends produce small audio packets at regular intervals, about every 20 milliseconds, and the other side plays them back at the same pace. Real-time voice models instead produce audio in computation chunks rather than on a fixed clock:

• Audio arrives in chunks: The model sends audio in larger groups instead of small, regular packets.
• Slightly slower than real time: During long sentences, audio can arrive a little more slowly than the phone line consumes it.
• Silence when the queue is empty: The phone side expects one packet per time slot; if the queue is empty at that moment, no packet goes out.

Every packet that cannot be sent is heard as a short gap or break on the caller's phone. The problem is not the network but two different rhythms meeting.

Why a Classic Jitter Buffer Is Not Enough

Jitter buffers in traditional phones are designed to smooth out packets whose average rate is correct but whose arrival times fluctuate. When the audio source is on average slower than real time, the problem is not packet order but that there is less audio than is being consumed. A passive buffer will sooner or later run dry. What is needed is an application layer that understands speech, silent moments, and feedback about how full the phone side is.

How the Pacer Works

Instead of pushing the model's audio straight onto the line, Yanitly passes it through a pacer. The pacer applies a few principles together on every turn:

1. 1.A small, adaptive pre-buffer: When a new reply starts, playback begins with a very short pre-buffer. If starvation is seen during the call, this buffer grows in a controlled way.
2. 2.Lead packets that feed the line: At the start of the stream, a few packets are handed to the phone side in advance so the first time slots never meet an empty queue.
3. 3.Bridging short gaps: If the next chunk from the model is late, a limited number of silent bridge packets keep the line's rhythm intact.
4. 4.Corrections only during silence: When the buffer builds up more than needed, latency is reduced only through silent packets; audible speech is never dropped.
5. 5.Small corrections from feedback: Limited adjustments are made based on how full the phone side reports it is, and audio that arrives before the call is answered is not counted.

Audible Speech Is Never Cut

The most important design rule is that packets containing speech are never thrown away to reduce latency. Clipped syllables are far more annoying to a listener than a little delay, so corrections happen in the silent moments between sentences.

Waiting Audio Is Cleared When You Interrupt

When the caller starts speaking while the assistant is talking, the audio waiting in the queue has to be cleared at once. The pacer drops the waiting audio and makes a clean start for the next reply, so the assistant can quickly switch to listening.

Balancing Latency and Choppiness

Buffer design always involves a trade-off. A large buffer reduces choppiness but delays the moment the assistant starts to answer; after a short silence, the caller tries to speak again and the two voices overlap. A very small buffer reduces delay but stays exposed to gaps when the audio source slows down.

Yanitly's approach is to start small and grow only when needed instead of using a fixed large buffer. Calls without problems keep a natural conversational pace, while calls where the audio source slows down are protected against choppiness first. The naturalness of a phone conversation comes from this adaptive behavior rather than from a single setting.

Starting Calls Without Waiting

In real-time voice, the opening moments of the call matter as much as the pacing of the stream. If the system waits for the recording system to be ready before answering, a momentary slowdown in storage delays the opening.

Recording and Connection Start in Parallel

On lines where recording is enabled, Yanitly starts recording preparation and the audio connection in parallel. It waits only a short, bounded time for the recording decision. If the recording system is not ready in that time, the late recording is marked and skipped, and the customer's conversation starts without waiting. The live conversation always comes first.

Processing Audio on a Nearby Node

In a voice infrastructure that runs on more than one server, the distance between the edge node that answers the call and the media worker that processes the audio also affects latency. Yanitly therefore prefers the local media workers on the node where the call arrived. If the local node is accepting calls and its load is below a set limit, the audio is processed on the same server; when the limit is exceeded or the node is in maintenance, the call is routed to the least-loaded node. This decision is recorded for every call. Under rising load, calls are balanced across nodes; when a node is drained for planned maintenance, new calls are sent to the other node while ongoing calls are allowed to finish naturally. Capacity is managed without turning into an interruption the caller would notice.

You Cannot Improve What You Do Not Measure

To track audio quality objectively, a short quality summary is kept at the end of every call. The summary contains no free text or personal data; it carries numeric indicators such as buffer adjustments, bridge packets, and media node selection. A new audio setting is shipped as a release, and if needed the previous behavior is restored with another release, so it is always clear which setting is running in production.

A Practical Checklist for Call Audio Quality

When you evaluate audio quality on your own lines, these steps help:

1. 1.Make test calls: Place a few test calls from different carriers and at different times of day and listen carefully.
2. 2.Tell the symptoms apart: Constant delay, occasional lost syllables, and complete drops point to different causes; note your observations with that distinction.
3. 3.Test the line and transfers together: Try after-hours routing and callback scenarios for inbound calls and the Web Call Desk and mobile outbound calling flow for team calls, and share the results with us.