KV Cache Optimization for LLM Inference: How Cache-Aware Serving Reduces Cost and Latency

The KV cache is the largest variable cost in LLM inference that most teams do not explicitly manage. For every request, the model computes key-value pairs for each attention head across each layer of the network and stores them in GPU VRAM for the duration of generation. These cached values allow the model to generate each new token without recomputing attention over the entire prompt on every forward pass. Without the KV cache, LLM inference would be orders of magnitude slower. With it, KV cache memory management becomes one of the primary drivers of GPU utilization, batch size limits, and effective cost per token.

At production scale, poor KV cache management means three things: lower maximum batch sizes (because KV cache fills VRAM before the GPU is compute-saturated), higher cost per request (because repeated prompt prefixes are recomputed from scratch on every request), and degraded tail latency (because requests queue waiting for KV cache memory to become available rather than waiting for compute).

• KV cache size scales linearly with context length. For Llama 3.3 70B, each token in the context window requires approximately 0.5 MB of KV cache storage. A single session with 128K tokens of context consumes 64 GB of KV cache, which is the entire VRAM of an H100 80GB. Managing how KV cache memory is allocated, shared, and reclaimed is the primary GPU memory management problem in production LLM serving.
• Prefix caching eliminates recomputation for shared context. When multiple requests share a common prefix (the same system prompt, the same retrieved documents, the same accumulated conversation history), prefix caching stores the KV cache for that shared prefix once and reuses it across all requests. For RAG pipelines and agent systems, this reduces effective input cost by 60 to 90 percent for the shared portion.
• GMI Prime Inference includes per-model KV cache tuning as part of its runtime optimization. vLLM with PagedAttention, SGLang with RadixAttention, and TensorRT-LLM with KV cache quantization are pre-configured per GPU class (H100, H200, B200) to maximize cache efficiency for each deployed model.
• PagedAttention eliminates KV cache fragmentation by managing cache memory in fixed-size blocks rather than contiguous regions. Contiguous KV cache allocation wastes 20 to 40 percent of VRAM through fragmentation across concurrent requests with different sequence lengths. PagedAttention recovers this wasted memory, effectively increasing the number of concurrent requests a GPU can serve.
• KV cache quantization (INT8, INT4) reduces cache memory by 2 to 4 times with minimal quality impact for most generation tasks. An H200 serving Llama 3.3 70B can support twice as many concurrent long-context sessions with INT8 KV cache as with FP16 KV cache, at a quality tradeoff that is typically acceptable for inference.
• Provider-side prompt caching (Anthropic, OpenAI, Kimi, and others) exposes KV cache reuse as a billing feature. Cache-hit input tokens cost 80 to 90 percent less than cache-miss input tokens. For workloads with large repeated context, choosing a provider with effective prompt caching reduces inference cost more than choosing a provider with a lower base per-token rate.

What the KV Cache Actually Is

In transformer attention, each token in the input sequence is represented as three vectors: a query (Q), a key (K), and a value (V). During the attention computation, the model computes how much attention each token should pay to every other token by taking the dot product of queries with keys, then using those attention weights to combine value vectors.

During autoregressive generation (producing tokens one at a time), each new token needs to attend to all previous tokens. Without the KV cache, the model would recompute the K and V vectors for every previous token on every forward pass. For a 1,000-token context generating a 500-token response, this means recomputing all 1,000 K and V vectors 500 times -- a waste of 500 times the necessary compute.

The KV cache eliminates this redundancy by storing K and V vectors in GPU VRAM as they are computed and retrieving them on subsequent forward passes rather than recomputing them. This converts generation from O(n²) compute (where n is context length) to O(n) compute for decode, at the cost of O(n) memory.

KV cache size calculation:

KV cache size per token = 2 (K and V) × n_layers × n_heads × head_dim × bytes_per_element

For Llama 3.3 70B at FP16:

• n_layers = 80, n_kv_heads = 8 (GQA), head_dim = 128, FP16 = 2 bytes
• KV cache per token = 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.32 MB per token

For a 128K context: 128,000 × 0.32 MB = 40 GB just for the KV cache of one session.

For Llama 3.3 70B at FP8 on a single H200 (141 GB VRAM):

• Model weights: 70 GB (FP8)
• KV cache at FP16 for one 128K session: 40 GB
• Available VRAM after weights: 71 GB
• Maximum concurrent 128K sessions: approximately 1.7 (effectively 1)

This is why long-context serving either requires KV cache quantization or careful batch size management. At 4K average context (more representative of production chat workloads):

• KV cache per session: 1.28 GB
• Available VRAM for KV cache (71 GB): approximately 55 concurrent sessions

The Three Core KV Cache Optimization Techniques

Technique 1: PagedAttention

PagedAttention, introduced by vLLM, manages KV cache memory using a paging system analogous to virtual memory in operating systems. Standard KV cache allocation reserves a contiguous block of VRAM for each request at the start of the sequence. Because different requests have different lengths, these contiguous blocks leave gaps between them as requests complete, fragmenting available VRAM.

Fragmentation is the central problem. With contiguous allocation, a GPU with 40 GB of available KV cache VRAM might only be able to serve 15 concurrent requests even though the combined KV cache size of those requests is only 30 GB, because fragmentation prevents using the remaining 10 GB efficiently.

PagedAttention divides KV cache into fixed-size blocks of 16 to 32 tokens each. Blocks are allocated and freed as needed, regardless of where they sit in VRAM. A request's KV cache may be scattered across non-contiguous VRAM blocks, with a page table tracking the mapping. This eliminates fragmentation almost entirely.

Practical impact: PagedAttention enables 2 to 4 times more concurrent requests on the same GPU hardware compared to naive contiguous KV cache allocation, without any change to model quality. For production inference with variable-length requests, this directly translates to 2 to 4 times higher throughput per GPU-hour.

Technique 2: Prefix Caching and RadixAttention

Prefix caching stores KV cache for shared prompt prefixes and reuses that cache across multiple requests. If 1,000 requests all share the same 2,000-token system prompt, standard serving computes the KV cache for those 2,000 tokens 1,000 times. With prefix caching, the KV cache is computed once and shared across all 1,000 requests.

vLLM implements basic prefix caching through its automatic prefix caching feature. SGLang extends this with RadixAttention, a trie-based prefix sharing system that handles more complex sharing patterns. RadixAttention organizes cached prefixes in a radix tree structure, enabling efficient lookup of the longest matching cached prefix for any incoming request. When two requests share a prefix that is not the full system prompt (for example, the same system prompt plus the first three turns of a conversation), RadixAttention finds and reuses the longest available cached prefix rather than only reusing exact matches.

For production workloads with shared context:

RAG pipelines: the retrieved documents represent a shared prefix across all queries that retrieve the same documents. With prefix caching, the KV cache for retrieved documents is computed once per document cache lifetime rather than once per query.

Agentic systems: accumulated conversation history grows with each tool call. Across multi-turn sessions where the beginning of the context is stable, RadixAttention reuses the cached prefix of prior turns, paying only the incremental compute for new turns.

Multi-tenant systems: when many users share the same system prompt, prefix caching amortizes the system prompt KV cache across all users.

Measured impact: SGLang reports up to 6.4 times throughput improvement for prefix-heavy workloads through RadixAttention, compared to serving without prefix caching. For workloads where 70 percent of input tokens are shared across requests, effective token compute costs drop proportionally.

Technique 3: KV Cache Quantization

KV cache is stored in GPU VRAM at the same precision as the activations being cached. For FP16 activations, KV cache occupies 2 bytes per value. Quantizing KV cache to INT8 reduces storage to 1 byte per value (2x reduction) and INT4 to 0.5 bytes (4x reduction).

The quality impact of KV cache quantization is typically small for standard generation tasks because the KV values participate in attention score computation rather than model weight arithmetic. The attention computation using quantized KV values introduces rounding error, but this error is small relative to the model's generation quality on most tasks. Evaluations show less than 1 percent quality degradation on most benchmarks for INT8 KV cache, and 1 to 3 percent for INT4 on complex reasoning tasks.

The practical benefit: a single H200 serving Llama 3.3 70B at FP16 KV cache can hold approximately 55 concurrent 4K-context sessions. The same H200 with INT8 KV cache holds approximately 110 concurrent sessions. At INT4, approximately 220 concurrent sessions. The throughput increase is proportional to the concurrency increase for long-context workloads where KV cache rather than compute is the VRAM bottleneck.

vLLM supports INT8 and INT4 KV cache quantization with the --kv-cache-dtype parameter. SGLang supports FP8 KV cache. TensorRT-LLM supports INT8 and FP8 KV cache with calibration-based quantization.

Production Patterns Where KV Cache Optimization Matters Most

Four production patterns benefit most from cache-aware serving. Understanding which pattern describes your workload determines which optimization techniques to prioritize.

Pattern 1: RAG with repeated document context

RAG pipelines retrieve document chunks and prepend them to the user query. When many queries retrieve the same documents (a common pattern when the document corpus is small or when popular documents are frequently retrieved), the KV cache for those documents is recomputed on every query without prefix caching.

Optimization priority: prefix caching with a document-level cache key. When retrieved documents are deterministic (the same query always retrieves the same documents), the cache hit rate for the document prefix can approach 100 percent after the first query.

Measured benefit: for a RAG system where retrieved documents represent 80 percent of input tokens and the same documents appear in 50 percent of queries, prefix caching reduces effective input compute by approximately 40 percent and reduces TTFT for cached queries by 60 to 80 percent (since the KV prefill for cached tokens is skipped).

Pattern 2: Agentic systems with accumulated history

Agentic systems accumulate context across tool call cycles. After 20 tool calls, the context may contain the initial instruction, 20 tool call-result pairs, and the model's reasoning across all of them. Each subsequent model call reprocesses this accumulated history.

Optimization priority: session-level KV cache management with checkpoint storage. The KV cache for the session prefix (all context up to the most recent tool call) should be preserved across tool call cycles rather than rebuilt from scratch. This is the core optimization that reduces per-token cost for long agentic workflows.

Measured benefit: for an agentic session with 50 tool calls where the prefix grows by an average of 500 tokens per cycle, the total prefix compute saved by caching grows quadratically with session length. By the 50th tool call, the cached prefix is 25,000 tokens; recomputing it from scratch costs as much as generating the entire prefix from scratch.

Pattern 3: Multi-tenant systems with shared system prompts

Applications that serve many users through the same system prompt (a customer service agent, a product assistant, a coding assistant) send the same system prompt token-for-token with every user request. Without prefix caching, this system prompt is fully recomputed for every request.

Optimization priority: system prompt prefix caching. This is the highest-leverage single optimization for multi-tenant applications because the cache hit rate for the system prompt prefix is effectively 100 percent across all users.

For a system prompt of 1,000 tokens serving 10,000 daily requests, prefix caching eliminates 10 million tokens of daily recompute. At $0.10 per million input tokens (GMI Cloud Qwen3-32B FP8 rate), this saves $1.00 per day, or approximately $365 per year, for this single application's system prompt alone. At frontier model rates ($3.00/M input), the same optimization saves $10.95 per day.

Pattern 4: Multi-turn conversations

Conversational applications maintain conversation history across turns. Each new turn reprocesses all prior turns in the context window. As conversations grow longer, the per-turn compute cost increases linearly with history length.

Optimization priority: turn-level KV cache storage with session affinity. Routing all turns from the same conversation session to the same serving instance preserves the KV cache across turns. Load balancers that break session affinity send each turn to a different serving instance, forcing KV cache rebuild on every turn.

Session affinity configuration in production: use consistent hashing on session identifiers for routing, with overflow routing to secondary instances that accept the KV cache miss cost rather than breaking the session.

Provider-Side Prompt Caching: How to Use It Effectively

Major managed inference providers expose KV cache reuse as a billing feature. Cache-hit tokens cost 80 to 90 percent less than cache-miss tokens. Using this effectively requires understanding how provider caching works and structuring prompts to maximize cache hits.

Cache hit requirements. Provider caching typically requires the cached prefix to be byte-identical to prior requests, above a minimum length threshold (usually 1,024 tokens), and recently accessed enough to still be resident in the provider's cache. Anthropic's prompt caching requires explicit cache breakpoint markers. OpenAI's automatic prompt caching does not require markers but applies only to prefixes above 1,024 tokens.

Maximizing cache hit rates:

• Place stable content first in the prompt: system prompt, then retrieved documents, then conversation history, then the current user message. Content that changes with every request (the current message) belongs at the end, after the cacheable content.
• Keep the stable prefix byte-identical across requests: even minor variations (different whitespace, slightly different phrasing of the system prompt) produce cache misses on providers that require exact match.
• Batch requests with shared context to the same endpoint during the cache's valid window.

Cache-hit pricing reference:

For providers where cache hits have explicit pricing, workloads with 70 percent or more of input tokens in a stable prefix achieve effective input rates 60 to 80 percent below the standard rate.

How to Measure KV Cache Efficiency

Three metrics quantify KV cache efficiency in production and indicate where optimization effort should focus.

Cache hit rate. The fraction of input tokens that are served from cached KV values rather than recomputed. Tracked separately for prefix cache hits (tokens matching a cached prefix) and KV block cache hits (tokens matching any previously cached block). A cache hit rate below 30 percent for a workload with significant shared context indicates that prefix caching is not configured or is not working correctly.

KV cache utilization. The fraction of allocated KV cache VRAM that is actively used by in-flight requests versus held for potential future use. Very low utilization (below 40 percent) indicates that the cache allocation is too large or that cache eviction is happening too aggressively. Very high utilization (above 90 percent) indicates that the cache is undersized and requests are waiting for cache memory rather than compute.

Prefill latency for cached versus uncached requests. Compare TTFT between requests where the full prefix was cached versus requests that required full prefix recomputation. A large TTFT gap (more than 3x) confirms that prefix caching is working and quantifies the latency benefit. A small gap (less than 1.5x) indicates that prefix caching is not activating correctly for requests that should be cache hits.

GPU VRAM breakdown. Periodically log the split between model weight VRAM, active KV cache VRAM, and free VRAM. If active KV cache exceeds 50 percent of total VRAM during typical production load, KV cache quantization is likely to improve throughput. If active KV cache is below 20 percent, the serving instance may be under-utilized and batch size can be increased.

GMI Prime Inference and KV Cache Optimization

GMI Prime Inference's per-model runtime tuning includes KV cache configuration as part of the optimization applied to each deployed model on each GPU class.

KV cache sizing per model. The KV cache size allocation is tuned per model based on its actual context length distribution in production rather than the theoretical maximum context window. A model deployed for customer service applications with typical contexts of 2,000 to 4,000 tokens receives a different KV cache allocation than the same model deployed for document analysis with contexts of 20,000 to 50,000 tokens. Over-allocating for the maximum reduces available space for concurrent requests; under-allocating causes unnecessary cache evictions.

Prefix caching enabled by default. GMI Prime Inference enables vLLM's automatic prefix caching and SGLang's RadixAttention by default for models where prefix sharing is common. The cache configuration is adjusted per model based on typical request patterns rather than applying generic defaults.

KV cache quantization per GPU class. H100 deployments default to FP8 KV cache for models where the quality tradeoff is acceptable, recovering additional VRAM for concurrent requests. H200 deployments with larger VRAM headroom can run FP16 KV cache for models where quality precision matters. B200 deployments support FP4 KV cache through Blackwell's native FP4 operations.

Warm endpoints preserve KV cache across requests. Because Prime Inference maintains warm GPU instances with pre-loaded model weights, the KV cache state persists between requests from the same session. For agentic and multi-turn workloads with session affinity configured, prior-turn KV cache is available without reloading, delivering the per-turn latency reduction of 60 to 80 percent for cached prior context.

Conclusion

KV cache management is the difference between a production LLM deployment that uses GPU VRAM efficiently and one that leaves 30 to 60 percent of available throughput on the table through fragmentation, redundant recomputation, and poor concurrency management.

PagedAttention eliminates VRAM fragmentation and recovers 2 to 4 times more concurrent capacity on existing hardware. Prefix caching eliminates recomputation of shared context, reducing effective input costs by 60 to 90 percent for workloads with stable prefixes. KV cache quantization doubles or quadruples the number of concurrent long-context sessions a single GPU can serve. Together, these techniques transform GPU utilization and effective cost per token for production deployments.

GMI Prime Inference applies these optimizations through per-model runtime tuning on each GPU class, with prefix caching, quantization, and sizing configurations calibrated to the specific model and workload rather than applied as generic defaults.

FAQs

What is the KV cache and why does it consume so much GPU memory? The KV cache stores key and value vectors from the attention computation for all tokens in the current context. Without it, generating each new token would require recomputing attention over every prior token, making generation O(n²) expensive. With it, generation is O(n) for decode, but O(n) memory must be allocated per token per session. For Llama 3.3 70B at FP16, each token requires approximately 0.32 MB of KV cache. A single session at 128K context requires 40 GB of KV cache storage, equivalent to the entire VRAM of an H100 80GB.

How much does prefix caching actually save in production? The savings depend on what fraction of input tokens are in a stable shared prefix. For a RAG pipeline where retrieved documents represent 80 percent of input tokens and the same documents appear in 60 percent of queries, prefix caching reduces effective input compute by approximately 48 percent and reduces TTFT for cache-hit queries by 60 to 80 percent. For multi-tenant applications with a shared system prompt of 1,000 tokens serving 10,000 daily requests at $3.00/M input, prefix caching saves approximately $10.95 per day from system prompt recompute alone.

What is the quality impact of KV cache quantization? INT8 KV cache quantization produces less than 1 percent quality degradation on most standard benchmarks compared to FP16. INT4 introduces 1 to 3 percent degradation on complex reasoning tasks. For most production inference workloads (conversational AI, code generation, document summarization), INT8 KV cache quantization is the correct default: it doubles available concurrent capacity at negligible quality cost. INT4 is appropriate for workloads where throughput maximization matters more than marginal quality preservation.

How does session affinity affect KV cache efficiency for multi-turn conversations? Session affinity routes all requests from the same conversation session to the same serving instance, preserving the KV cache for prior turns. Without affinity, each request routes to whatever instance has capacity, forcing a full KV cache rebuild on every turn. For a conversation with 10 prior turns averaging 200 tokens each (2,000 tokens of cached history), affinity reduces the per-turn prefill computation by 2,000 tokens and TTFT by 60 to 80 percent for the cached portion. Load balancers should use consistent hashing on session identifiers rather than round-robin routing for conversational AI workloads.

What is the difference between PagedAttention and RadixAttention? PagedAttention solves memory fragmentation by managing KV cache in fixed-size blocks rather than contiguous regions, enabling 2 to 4 times more concurrent requests on the same VRAM. RadixAttention (SGLang) solves prefix recomputation by organizing cached prefixes in a trie structure that enables efficient lookup and reuse of the longest matching cached prefix for any incoming request. PagedAttention is about memory management; RadixAttention is about computation reuse. Production deployments benefit most from combining both: PagedAttention for memory efficiency and RadixAttention for prefix sharing.