Dedicated Inference Endpoints for Real-Time Voice AI: Latency, Jitter, and Infrastructure Requirements

Voice AI has the strictest latency requirements of any production AI workload. A text chat interface where the first token appears after 800 milliseconds feels responsive. A voice conversation where the AI takes 800 milliseconds to begin speaking feels broken. Human conversational turn-taking operates on a 200 to 300 millisecond gap, and this constraint cascades through every layer of the stack: speech-to-text must complete fast enough to leave budget for the LLM, the LLM must produce its first token fast enough to leave budget for text-to-speech, and every layer's latency adds to a total that must land under 800 milliseconds for the interaction to feel natural.

• The voice AI latency budget is approximately 800 milliseconds end-to-end, split across speech-to-text (100 to 200ms), LLM inference TTFT (150 to 300ms), text-to-speech first chunk (100 to 200ms), and network round trips (50 to 150ms). Exceeding this budget produces perceptible awkwardness in conversational turn-taking.
• Jitter is more damaging than average latency in voice applications. A system that consistently responds in 600 milliseconds feels more natural than one that averages 500 milliseconds but occasionally takes 2 seconds. Human conversational expectations are built around predictable timing, and variance breaks the interaction pattern in ways that a slightly higher constant latency does not.
• GMI Prime Inference provides the dedicated single-tenant GPU capacity that voice AI requires: reserved GPUs with model weights pre-loaded and warm at all times (no cold start), per-model runtime tuning for sub-200ms TTFT, region-pinned endpoints for minimum network round-trip time, and a 99.9 percent uptime SLA.
• Cold starts are catastrophic in voice applications, not merely inconvenient. A 15-second cold start on a text chat interface produces an annoyed user. The same cold start in a voice conversation produces 15 seconds of silence during which the user assumes the call dropped. Voice AI cannot tolerate any cold start, which structurally rules out serverless inference for the primary conversational path.
• Shared-tier inference cannot deliver bounded p99 latency, and voice AI is judged on p99 rather than p50. One conversational turn in fifty that takes 2 seconds is one broken moment per conversation, which users remember more than the 49 turns that worked correctly.
• Persistent WebSocket connections and session affinity are infrastructure requirements, not optimizations. Voice sessions maintain a continuous audio stream. Session affinity keeps conversation context in the same serving instance's KV cache, eliminating per-turn context rebuild that would otherwise add 200 to 400ms to every turn.

The Voice AI Latency Budget

Voice AI systems chain multiple models in sequence. Understanding where the latency budget is spent identifies which layers to optimize and which infrastructure decisions matter most.

Stage 1: Voice activity detection and audio buffering (50 to 150ms)

The system must detect that the user has stopped speaking before it can begin processing. Voice activity detection (VAD) analyzes the audio stream for silence patterns. Aggressive VAD (short silence threshold) reduces latency but risks cutting off users mid-sentence. Conservative VAD (longer silence threshold) avoids interruption but adds latency to every turn.

Most production voice systems use a 300 to 500ms silence threshold, which means the system waits 300 to 500ms after the user stops speaking before processing. This wait is part of the perceived latency budget from the user's perspective, though it is not typically counted in technical latency measurements.

Stage 2: Speech-to-text transcription (100 to 200ms)

Streaming STT models (Whisper streaming variants, Deepgram, AssemblyAI streaming) transcribe audio incrementally as it arrives. By the time the user stops speaking, most of the transcription is already complete. The remaining latency is finalizing the last few words and returning the complete transcript.

Non-streaming STT models wait for the complete audio segment before transcribing, adding 300 to 800ms depending on audio length. For voice AI, streaming STT is a requirement.

Stage 3: LLM inference time to first token (150 to 300ms)

This is the stage where infrastructure decisions have the largest impact. The LLM receives the transcript, processes the conversation context, and begins generating the response. TTFT is the metric that matters: as soon as the first token is available, TTS can begin synthesizing audio.

TTFT is the sum of network transit to the inference endpoint plus model prefill time. For a voice application, both components must be minimized:

Network transit: minimized by deploying inference endpoints geographically close to users. A user in Tokyo connecting to a US West endpoint adds 200 to 260ms round-trip, which alone consumes a third of the total latency budget.

Model prefill: minimized by model selection (smaller models prefill faster), context length management (shorter contexts prefill faster), and KV cache reuse (cached conversation history skips prefill entirely).

Stage 4: Text-to-speech first audio chunk (100 to 200ms)

Streaming TTS models begin generating audio as soon as the first few words of text are available. The latency to first audio chunk determines when playback can begin. Fast TTS models (ElevenLabs Turbo, Cartesia Sonic, PlayHT) achieve 100 to 150ms time to first audio byte.

Non-streaming TTS waits for the complete LLM response before synthesizing, which adds the entire LLM generation time to the perceived latency. For voice AI, streaming TTS chained to streaming LLM output is required.

Stage 5: Network delivery and audio playback (50 to 100ms)

The generated audio must reach the user's device and begin playing. WebRTC and WebSocket transport add 50 to 100ms depending on network conditions and geographic distance.

The 800ms target sits in the middle of this range. Achieving it consistently requires optimizing every stage, with the LLM inference stage offering the largest infrastructure-controllable improvement. The gap between a working prototype and a production voice system is largely a latency engineering problem, which is a specific instance of the broader pattern described in GMI Cloud's guide to production AI inference: the distance between the first inference attempt and a reliable production deployment is where most AI projects spend their engineering time.

Why Serverless Inference Fails for Voice AI

Serverless inference is the wrong architecture for the primary conversational path in voice AI, for three specific reasons.

Reason 1: Cold start is a call-dropping event, not a slow response.

A 15 to 30 second cold start on a 70B parameter model produces 15 to 30 seconds of complete silence in a voice conversation. Users do not wait 15 seconds in silence assuming the system is thinking. They assume the call dropped, hang up, and try again, which triggers another cold start if the previous instance was released.

Text applications degrade gracefully under cold start: the user sees a loading indicator and waits. Voice applications do not have a loading indicator. The medium is audio, and silence in audio means failure.

Reason 2: Shared-tier p99 latency is unbounded.

Voice AI is judged on p99 latency, not p50. A conversation with 40 turns where 39 turns respond in 600ms and one turn takes 2.5 seconds contains one moment where the interaction visibly breaks. Users remember that moment.

Shared multi-tenant serverless infrastructure cannot bound p99 latency because the latency depends on aggregate platform load, not on your traffic. Another tenant's batch job saturating the shared GPU pool produces p99 latency spikes for your voice sessions, and there is no configuration you can apply to prevent it. GMI Cloud's analysis of high-concurrency inference workloads describes how latency compounds and queues explode under concurrency when execution is sequential rather than parallelized, which is precisely the failure pattern that breaks voice AI during peak load.

Reason 3: Session context rebuild on every turn.

Voice conversations accumulate context across turns. Without session affinity, each turn routes to whatever serving instance has capacity, forcing a complete KV cache rebuild for the accumulated conversation history. For a 10-turn conversation with 2,000 tokens of history, this rebuild adds 200 to 400ms to every turn, consuming half the LLM latency budget on recomputation that a warm session with cached context would skip entirely.

Dedicated Endpoint Requirements for Voice AI

Voice AI requires five specific infrastructure properties that dedicated inference endpoints provide and shared infrastructure does not.

Requirement 1: Always-warm model weights with zero cold start

The model must be resident in GPU VRAM continuously, so that every request receives the same TTFT regardless of when the previous request arrived. On GMI Prime Inference, reserved GPUs maintain weights in VRAM permanently. There is no cold state and therefore no cold start.

This is the non-negotiable requirement for voice AI. Any infrastructure that can evict model weights during idle periods will produce a cold start on some conversation, and that conversation will be a failed user interaction.

Requirement 2: Single-tenant capacity for bounded p99 latency

The GPU must serve only your workload. Single-tenant allocation means p99 latency is determined by your traffic patterns and model performance characteristics, not by aggregate platform load. This is what makes a p99 latency SLA contractually supportable, which matters both for internal quality targets and for enterprise customer commitments.

Requirement 3: Region-pinned endpoints for minimum network latency

Network round-trip time is a fixed cost determined by geographic distance. A user 200ms away from the inference endpoint pays 200ms of the 800ms budget on network transit alone. Deploying inference endpoints in the regions where users are located reduces this to 10 to 40ms.

GMI Prime Inference provides regional deployment across APAC (Tokyo, Singapore, Taiwan), North America (US West, East, Central, South), and Europe. For voice AI serving users in multiple geographies, regional endpoint deployment is the single largest latency improvement available.

Requirement 4: Session affinity with persistent KV cache

Voice sessions must route to the same serving instance across all turns to preserve the KV cache for accumulated conversation history. Session affinity is implemented through consistent hashing on the session identifier at the load balancer layer.

The benefit compounds with conversation length. At turn 20 with 4,000 tokens of accumulated context, session affinity saves 400 to 800ms of prefill time per turn compared to full context rebuild. This saving is the difference between a conversation that stays natural at turn 20 and one that becomes progressively slower.

Requirement 5: Per-model runtime tuning for TTFT optimization

Generic serving stacks optimize for aggregate throughput. Voice AI requires optimization for TTFT specifically, which is a different tuning target. Lower batch sizes reduce queuing delay before generation begins, at the cost of aggregate throughput. Chunked prefill prevents long-context requests from blocking short-context requests. Speculative decoding reduces inter-token latency for streaming output.

GMI Prime Inference's per-model runtime tuning configures these parameters per model and per GPU class. For voice AI workloads, the tuning target is minimum TTFT and consistent inter-token latency rather than maximum aggregate throughput.

Model Selection for Voice AI

The LLM choice affects the latency budget more than most teams expect, because prefill time and inter-token latency both scale with model size.

Smaller models are dramatically faster for TTFT. A 32B parameter model prefills in roughly half the time of a 70B model at equivalent context length. For voice AI where TTFT budget is 150 to 300ms, this difference is significant. Qwen3-32B at FP8 on a single H100 achieves TTFT in the 80 to 150ms range for typical conversation contexts, well within the voice budget. Llama 3.3 70B at FP8 achieves 150 to 250ms, which fits the budget but leaves less headroom.

Context length management matters more than in text applications. Voice conversations accumulate context, and prefill time scales with context length. A conversation with 8,000 tokens of history prefills significantly slower than one with 2,000 tokens. Voice AI systems should implement conversation summarization at defined thresholds, condensing older turns into a summary that preserves relevant context while reducing token count.

Inter-token latency determines speech naturalness. After the first token, the rate at which subsequent tokens generate determines whether the TTS can maintain continuous speech output. If token generation is slower than speech playback rate, the TTS runs out of text and produces audible gaps. Token generation must sustain at least 15 to 20 tokens per second to keep pace with natural speech rate, which all modern models on H100 or H200 hardware comfortably exceed at reasonable batch sizes.

Quality versus latency tradeoff for voice. Voice interactions are conversational rather than analytical. The quality bar for a conversational response is different from the quality bar for a code generation task or a document analysis. A 32B model that responds in 100ms produces a better voice experience than a 70B model that responds in 250ms, even if the 70B model's response is marginally more sophisticated. For voice AI, latency optimization typically outweighs marginal quality improvement.

Teams evaluating model options for voice workloads benefit from testing multiple model sizes before committing. GMI Cloud's on-demand infrastructure provides both managed API access to 100-plus models and bare metal GPU access with hourly billing, making it practical to benchmark TTFT across candidate models on the same hardware before selecting the production configuration.

Handling Interruptions and Barge-In

Voice conversations include interruptions. A user may begin speaking while the AI is still talking, which requires the system to stop TTS playback, cancel the in-progress LLM generation, and process the new user input.

Cancellation latency matters. When a user interrupts, the system must stop the current response quickly. If cancellation takes 500ms, the AI continues talking over the user for half a second, which feels rude and unnatural.

Cancellation requires the ability to abort an in-progress inference request. Streaming inference APIs typically support request cancellation, but the actual GPU-side cancellation latency varies by serving framework. vLLM and SGLang both support request abort with cancellation latency in the 10 to 50ms range on dedicated infrastructure.

Partial generation state management. When an interruption occurs mid-generation, the conversation state must record what the AI said before being interrupted, not the complete response it would have generated. The KV cache for the partial response must be either preserved (if the conversation continues from the interruption point) or discarded (if the new user input replaces the interrupted turn).

Dedicated infrastructure advantage for cancellation. On shared serverless infrastructure, request cancellation may not free GPU capacity immediately, and the cancellation itself competes with other tenants' requests for scheduling. On dedicated infrastructure, cancellation is immediate because the GPU is exclusively serving your workload.

Measuring Voice AI Latency in Production

Voice AI latency measurement requires instrumentation at each stage boundary, not just end-to-end timing.

Stage-level timing. Log timestamps at each stage transition: user speech end detected, transcript available, LLM request sent, first LLM token received, first TTS audio chunk generated, first audio chunk delivered to client. The deltas between these timestamps identify which stage is consuming the latency budget.

p50, p95, and p99 for each stage. Voice AI quality depends on p99, not average. Track percentile distributions for each stage separately. A stage with p50 of 150ms and p99 of 900ms is the source of the broken conversational moments, even if the average looks acceptable.

Turn-level and session-level aggregation. Measure latency per conversational turn, then aggregate per session. A session where turn 1 responds in 400ms and turn 15 responds in 900ms indicates context growth is degrading latency, pointing to a context management problem rather than an infrastructure problem.

Interruption cancellation latency. Measure the time from user speech detection to TTS playback stop. This is a separate latency metric from response latency and requires separate instrumentation.

Client-side measurement for real network conditions. Server-side latency measurement misses network variability, mobile network conditions, and client-side audio processing. Client-side instrumentation captures the latency the user actually experiences, including all network and device factors.

GMI Prime Inference for Voice AI Workloads

GMI Prime Inference is designed for the workload categories where shared inference infrastructure falls short, and real-time voice is the clearest example of that category.

Warm endpoints eliminate cold start. Reserved GPUs with pre-loaded weights mean no conversation ever encounters a cold start. This is the foundational requirement for voice AI and the reason serverless inference cannot serve the primary conversational path.

Single-tenant isolation bounds p99 latency. Voice quality depends on p99 latency, and only single-tenant capacity makes p99 bounded by your own traffic rather than platform aggregate load.

Region-pinned endpoints minimize network round trips. Regional deployment across APAC, North America, and Europe reduces the network component of the latency budget from 200 to 260ms (cross-continental) to 10 to 40ms (in-region).

Per-model runtime tuning targets TTFT. For voice AI workloads, the tuning target is minimum TTFT and consistent inter-token latency rather than maximum aggregate throughput. GMI's inference engineering team configures serving parameters per model and per GPU class to achieve this.

Persistent WebSocket sessions with dedicated warm capacity. Voice AI requires continuous session connections. Dedicated capacity with session affinity keeps conversation context in the same instance's KV cache, eliminating the per-turn context rebuild that shared infrastructure forces.

Elastic burst for traffic peaks. Voice AI applications experience traffic peaks (business hours, campaign activations, event-driven spikes). Burst capacity absorbs these peaks without queuing, maintaining consistent latency during high-demand periods when reserved capacity alone would be insufficient.

Conclusion

Voice AI is the production workload with the least latency headroom and the least tolerance for variance. The 800 millisecond end-to-end budget leaves 150 to 300 milliseconds for LLM inference, which requires warm model weights, single-tenant GPU capacity, region-proximate deployment, session affinity for KV cache continuity, and serving stack tuning optimized for TTFT rather than throughput.

Serverless inference cannot deliver these properties. Cold start produces silent failures in a medium where silence means the call dropped. Shared-tier p99 latency is unbounded by aggregate platform load. Session context rebuild on every turn consumes half the LLM latency budget on recomputation.

GMI Prime Inference provides the dedicated single-tenant infrastructure that real-time voice AI requires: always-warm reserved GPUs, per-model TTFT tuning, region-pinned endpoints across three global regions, and a 99.9 percent uptime SLA that supports the latency commitments voice AI products make to their users.

FAQs

What is the latency budget for real-time voice AI and how is it distributed? The total end-to-end budget is approximately 800 milliseconds, matching the human conversational turn-taking gap of 200 to 300 milliseconds plus acceptable tolerance. It distributes across five stages: voice activity detection and buffering (50 to 150ms), speech-to-text transcription (100 to 200ms with streaming STT), LLM inference TTFT (150 to 300ms), text-to-speech first audio chunk (100 to 200ms with streaming TTS), and network delivery plus playback (50 to 100ms). The LLM inference stage offers the largest infrastructure-controllable improvement through warm dedicated endpoints, regional deployment, and KV cache reuse.

Why can't serverless inference be used for voice AI? Three structural reasons. Cold start produces 15 to 30 seconds of silence for 70B class models, which in a voice conversation reads as a dropped call rather than a slow response. Shared-tier p99 latency is unbounded because it depends on aggregate platform load rather than your traffic, and voice AI is judged on p99 rather than p50. Session context rebuild on every turn (without session affinity) adds 200 to 400ms per turn for a 10-turn conversation, consuming half the LLM latency budget on recomputation. Dedicated endpoints with always-warm weights, single-tenant capacity, and session affinity address all three.

How much does session affinity actually improve voice AI latency? Session affinity keeps all turns from a conversation on the same serving instance, preserving the KV cache for accumulated conversation history. At turn 10 with 2,000 tokens of history, this saves 200 to 400ms of prefill time per turn. At turn 20 with 4,000 tokens, it saves 400 to 800ms per turn. Without affinity, each turn rebuilds the full context from scratch, and the latency degradation compounds as the conversation grows. Session affinity is implemented through consistent hashing on the session identifier at the load balancer layer.

Should voice AI use smaller models for lower latency, and what is the quality tradeoff? For voice AI, latency optimization typically outweighs marginal quality improvement. A 32B parameter model achieving 80 to 150ms TTFT produces a better voice experience than a 70B model at 150 to 250ms TTFT, because the conversational quality bar is different from analytical task requirements. Voice interactions are conversational rather than analytical, and the perceived quality of a voice interaction depends heavily on timing naturalness. Qwen3-32B at FP8 on a single H100 is a strong default for voice AI. Larger models are appropriate when the voice application requires complex reasoning or domain expertise that smaller models cannot deliver.

How should interruption handling be implemented for natural voice conversations? Interruption handling requires three components. First, fast cancellation: aborting the in-progress inference request when user speech is detected, with cancellation latency in the 10 to 50ms range on dedicated infrastructure using vLLM or SGLang request abort. Second, TTS playback stop: halting audio output immediately rather than completing the current audio chunk. Third, partial state management: recording what the AI actually said before interruption (not the full response it would have generated) in the conversation history, so subsequent turns have accurate context. On shared serverless infrastructure, cancellation may not free GPU capacity immediately; dedicated infrastructure provides immediate cancellation because the GPU serves only your workload.