Prompt Caching in Production: How to Cut Inference Costs by 60 to 90 Percent for Repeated Contexts

Most AI applications send the same content with every request. A customer service agent sends the same system prompt and product documentation to the model with every user message. A RAG pipeline sends the same retrieved documents with every query against the same document. A coding assistant sends the same codebase context with every follow-up question. Without prompt caching, each of these requests recomputes the KV cache for the repeated content from scratch, paying the full input token rate every time.

Prompt caching is the mechanism that eliminates this redundant computation. When a request shares a prefix with a prior request, the provider retrieves the cached KV values for that prefix rather than recomputing them, and charges the cache-hit rate rather than the standard input rate. Cache-hit rates across major providers are 80 to 90 percent lower than standard input rates. For workloads where 70 percent or more of input tokens are in a stable repeated prefix, prompt caching reduces total inference cost by 56 to 63 percent without any change to model quality or output behavior.

• Prompt caching is a billing feature built on top of KV cache reuse. The underlying mechanism is the same whether implemented by a provider (Anthropic, OpenAI, Kimi) or by a self-hosted serving framework (vLLM prefix caching, SGLang RadixAttention). The difference is that provider-side caching is exposed through API parameters or automatic detection, while self-hosted caching is configured at the infrastructure layer.
• Anthropic's cache_control parameter provides the most explicit control. You mark specific parts of the prompt as cacheable with "cache_control": {"type": "ephemeral"} markers. Cache TTL is a minimum of 5 minutes, extendable to 1 hour with explicit refresh. Cache hits are charged at $0.30/M tokens versus $3.00/M standard, a 90 percent reduction.
• OpenAI's automatic prompt caching requires no code changes. Any prompt prefix of 1,024 or more tokens that matches a recent request is automatically cached at 50 percent off the standard input rate. No markers, no explicit cache management. The tradeoff is less control: you cannot guarantee what is cached or for how long.
• GMI Cloud includes KV cache reuse in its managed inference through vLLM prefix caching and SGLang RadixAttention on Prime Inference dedicated endpoints. For self-hosted models, cache efficiency is part of the per-model runtime tuning rather than a separate billable feature.
• The single highest-ROI prompt caching implementation is system prompt caching. A 2,000-token system prompt sent with 10,000 daily requests at $3.00/M input costs $60/day without caching and $6/day with 90 percent cache hits. That $54/day saving requires one code change: adding a cache breakpoint marker after the system prompt.
• Prompt structure determines cache hit rate more than cache TTL. A prompt where the user message appears at the beginning and the system prompt appears at the end will never hit the cache, because providers cache from the beginning of the prompt and the prefix changes with every user message.

How Prompt Caching Works

Prompt caching is the provider-facing interface to the same KV cache reuse that the previous article covered at the infrastructure level. When a model processes a prompt, it computes key-value pairs for every input token and stores them in GPU VRAM. These KV values represent the model's "understanding" of the input at that point in the sequence.

For subsequent requests that begin with the same sequence of tokens, the provider can retrieve the pre-computed KV values from storage rather than recomputing them. The model starts generation from the point where the cached prefix ends, as if it had just processed those tokens. The output is identical to what would have been generated with full recomputation.

What makes a prefix cacheable:

• Byte-for-byte identical to a prior request's prefix (no variation in whitespace, punctuation, or ordering)
• Long enough to cross the provider's minimum threshold (typically 1,024 tokens)
• Recent enough that the cache entry has not expired
• Within the provider's supported cache structure (some providers require explicit markers; others detect automatically)

What breaks the cache:

• Any change to the content in the cached prefix, even a single character
• Reordering content that was previously stable
• Variable content (timestamps, session IDs, user names) appearing before the stable content
• Exceeding the cache's TTL without a refresh request

Provider-by-Provider Implementation Guide

Anthropic (Claude models)

Anthropic's prompt caching requires explicit cache breakpoint markers in the API request. You add "cache_control": {"type": "ephemeral"} to the last content block that should be cached.

response = anthropic.messages.create(

    model="claude-opus-4-6",

    max_tokens=1024,

    system=[

        {

            "type": "text",

            "text": "You are a helpful customer service agent for Acme Corp. [2,000 tokens of product documentation]",

            "cache_control": {"type": "ephemeral"}

        }

    ],

    messages=[{"role": "user", "content": user_message}]

)

Key parameters:

• Minimum cacheable length: 1,024 tokens (for Claude Sonnet and Haiku), 2,048 tokens (for Claude Opus)
• Cache TTL: minimum 5 minutes from last use, extendable
• Maximum breakpoints per request: 4
• Cache write cost: 25 percent premium on standard input rate for tokens being cached for the first time
• Cache hit cost: $0.30/M for Claude Opus (90 percent discount from $3.00/M standard)

Cache write cost means the first request that populates the cache pays slightly more than standard. The break-even point is typically 2 to 3 cache hits before the savings exceed the write cost. For stable system prompts used hundreds of times per day, the break-even is reached within minutes.

Monitoring cache performance with Anthropic: The API response includes cache_creation_input_tokens (tokens written to cache this request) and cache_read_input_tokens (tokens read from cache). Tracking these over time gives the cache hit rate and the effective cost per request.

OpenAI (GPT-4o and GPT-4o-mini)

OpenAI's prompt caching is automatic: no markers, no API parameters. Any prompt where the first 1,024 or more tokens match a recent prior request is automatically cached. Cache hits are charged at 50 percent of the standard input rate.

response = openai.chat.completions.create(

    model="gpt-4o",

    messages=[

        {"role": "system", "content": "System prompt content [stable]"},

        {"role": "user", "content": user_message}

    ]

)

# Check cache usage in response

cached_tokens = response.usage.prompt_tokens_details.cached_tokens

The automatic approach requires no code changes and works for any prompt that naturally starts with stable content. The limitation is that you cannot force a cache write or guarantee TTL. Cache availability depends on server-side state that OpenAI does not expose.

Google Gemini

Gemini's context caching is distinct from Anthropic and OpenAI: you create a named cache object containing the content to be cached, then reference it by name in subsequent requests. The cache is billed per token per hour of storage rather than per cache hit.

This model suits use cases where you cache a very large document or codebase once and then run many queries against it. For smaller system prompts used across many users, the per-hour billing model may be more expensive than Anthropic's or OpenAI's per-hit model.

Self-hosted via vLLM and SGLang

On self-hosted models, prefix caching is a serving configuration rather than an API billing feature. Enable automatic prefix caching in vLLM with --enable-prefix-caching. SGLang's RadixAttention is enabled by default and handles prefix sharing automatically.

For self-hosted models, there is no per-hit charge. The benefit appears as higher throughput: requests that hit the cache complete their prefill phase faster (since cached tokens skip KV computation), freeing GPU compute for generation. The cost saving is indirect: higher throughput on the same hardware reduces effective GPU cost per request.

The Four Prompt Structures That Maximize Cache Hit Rate

Prompt structure is the largest controllable variable in cache efficiency. The same content organized differently can produce a 0 percent or 90 percent cache hit rate.

Structure 1: Static content first, dynamic content last

This is the fundamental rule. Providers cache from the beginning of the prompt. Every token before the first dynamic element must be byte-identical across requests for the prefix to match. Place everything that does not change with every request before everything that does.

Correct structure:

1. System prompt (static, same for all users)
2. Few-shot examples (static, same for all users)
3. Retrieved documents (static for a given retrieval result)
4. Conversation history (grows but prior turns are stable)
5. Current user message (dynamic, changes every request)

Incorrect structure (breaks caching):

1. Current user message (dynamic)
2. System prompt
3. Retrieved documents

With the incorrect structure, the prefix changes on every request because the first element is the dynamic user message. Cache hit rate is 0 percent regardless of how much stable content follows.

Structure 2: Segment by stability tier

Different elements of a prompt have different stability levels. The system prompt never changes. Conversation history grows but prior turns are stable. Retrieved documents vary across sessions but are stable within a session. The current message changes on every request.

Organize the prompt so that content at each stability tier is grouped together:

• Tier 1 (never changes): system prompt, static examples, tool definitions
• Tier 2 (stable within a session): retrieved documents, conversation history
• Tier 3 (changes per request): current user message, dynamic context

For Anthropic with multiple cache breakpoints, place cache markers at the boundary between tiers. The Tier 1 to Tier 2 boundary is the primary cache breakpoint. The Tier 2 to Tier 3 boundary is a secondary cache breakpoint for session-level caching.

Structure 3: Normalize variable content before the stable prefix

Some prompts include variable metadata that appears before the main content: timestamps, session IDs, user identifiers. If this variable metadata appears before the stable system prompt, every request produces a unique prefix and the cache hit rate is 0 percent.

Move variable metadata after the stable content, or eliminate it from the main prompt and pass it as a separate parameter. The model does not need to see a session ID to do its job; session tracking is an infrastructure concern, not a model concern.

Structure 4: Stabilize retrieved content ordering

RAG systems retrieve documents and include them in the prompt. If retrieved documents appear in a different order on different requests (because the retrieval ranking changes slightly), the prefix is not byte-identical and does not hit the cache even if the same documents were retrieved.

Deterministic ordering for cached documents: sort retrieved documents by a stable key (document ID, URL, or hash) rather than by retrieval score. The retrieval score can be passed as metadata within the document's text block without affecting the content order.

Sizing the Benefit: A Calculation Framework

Before implementing prompt caching, calculate the expected cost reduction to prioritize which workloads to tackle first.

Formula: Expected cost reduction = (cached_tokens / total_input_tokens) × (1 - cache_hit_rate_per_request) × standard_input_rate

Wait, that is not quite right. Let me reframe:

For a workload with the following properties:

• Total daily requests: N
• Input tokens per request: T_total = T_stable + T_dynamic
• T_stable: tokens in the stable cacheable prefix
• Cache hit rate: H (fraction of requests where the prefix is already cached)
• Standard input rate: P_standard
• Cache-hit rate: P_cached (e.g., 0.10 × P_standard for 90% discount)

Daily cost without caching: N × T_total × P_standard

Daily cost with caching: First request (cache write): T_total × P_standard × (1 + 0.25) (write penalty) Subsequent cache hits: T_dynamic × P_standard + T_stable × P_cached Remaining cache misses: T_total × P_standard

Practical example:

Customer service agent: 10,000 daily requests, 3,000-token system prompt, 200-token user message, $3.00/M input rate.

Without caching: 10,000 × 3,200 × $3.00/M = $96/day

With 90 percent cache hit rate (Anthropic):

• Cache write requests (10%): 1,000 × 3,200 × ($3.00/M × 1.25) = $12
• Cache hit requests (90%): 9,000 × (200 × $3.00/M + 3,000 × $0.30/M) = $9,000 × ($0.60 + $0.90) / 1,000,000 = $13.50

Daily cost with caching: $25.50 Daily saving: $70.50 (73.4 percent reduction) Annual saving: $25,732

This example represents a single customer service application. Enterprise AI platforms serving hundreds of application configurations with shared system prompts multiply this saving proportionally.

Measuring Cache Effectiveness in Production

Three metrics track whether prompt caching is working as expected.

Cache hit rate by request type. Break down cache hits separately for system prompt hits, document context hits, and conversation history hits. A system prompt hit rate below 90 percent when the same system prompt is sent on every request indicates a structural problem: the prompt is probably placing variable content before the system prompt, preventing the prefix from matching.

Effective input cost per request. Compute (total input cost) / (total input tokens) over a rolling window. Compare against the standard input rate. If effective input cost is within 10 to 15 percent of the standard rate for a workload with significant shared context, caching is not working. A healthy effective rate for a RAG or agentic workload should be 40 to 70 percent below the standard rate.

Cache write to read ratio. A high write-to-read ratio (many cache writes, few reads) indicates the cache TTL is too short relative to request frequency, or that the prompt structure is breaking cache continuity. A TTL problem appears as high write rate during business hours but near-zero read rate after short idle periods. A structure problem appears as consistently high write rate regardless of request frequency.

Common Implementation Mistakes

Putting the user message first. Any variable content before the stable prefix produces a 0 percent cache hit rate. This is the most common mistake and the easiest to fix: move the user message to the end of the prompt.

Not marking cache breakpoints explicitly (Anthropic). Without cache_control markers, Anthropic's API does not cache anything regardless of prompt structure. The automatic caching that OpenAI provides is not available on Anthropic by default.

Caching after the dynamic content. Anthropic's cache breakpoint marks everything from the beginning of the prompt up to the marker's position. If you place the marker after the user message, the cached prefix includes the dynamic user message and produces a 0 percent hit rate.

Varying the stable content unnecessarily. Adding a timestamp or request ID to the system prompt, including the current date, or using slightly different whitespace conventions breaks the byte-exact match required for cache hits. Stable content should be truly static: the exact same bytes on every request.

Ignoring the cache write cost. Anthropic charges a 25 percent premium for cache writes (tokens being cached for the first time). For workloads with high request frequency, this write cost is negligible compared to the savings from cache hits. For workloads with low request frequency (fewer than 5 to 10 requests per cache lifetime), the write cost may exceed the savings, making caching less beneficial.

Over-caching short content. Caching a 200-token system prompt saves $0.72 per 1,000 requests at $3.00/M input with 90 percent cache hits. For a low-volume endpoint with 100 requests per day, this saving is $0.07/day, which is not worth the implementation effort. Focus caching implementation on prompts above 1,000 tokens and workloads above 1,000 daily requests.

Prompt Caching for Specific Use Cases

Coding assistants with codebase context. Load the relevant codebase files into the context as a cached prefix. Cache hit rate approaches 100 percent for all questions about the same codebase. The cache should be refreshed when the codebase changes, using the modified files as the trigger for a cache invalidation and rewrite.

Document Q&A systems. Load the document or document set as a cached prefix. Multiple questions about the same document hit the cache on every request after the first. For large document sets, consider creating separate cached contexts per document rather than a single large context that changes when any document updates.

Few-shot learning prompts. Include examples in the cached prefix. Few-shot examples are typically the same across all requests and represent the highest-stability, highest-value content to cache. A 20-example few-shot prompt of 5,000 tokens cached at Anthropic's rate produces $0.135 per 1,000 requests versus $1.50 without caching.

Agent tool definitions. Tool definitions (the JSON schema describing available functions) are the same across all agent sessions. Include them in the cached prefix before conversation history. For agents with 20 to 30 tool definitions, tool schemas typically add 1,500 to 3,000 tokens that can be cached rather than recomputed on every agentic step.

GMI Cloud and Self-Hosted Prompt Caching

For workloads running on GMI Cloud's dedicated GPU infrastructure or Inference Engine, prompt caching operates at the serving framework level rather than as a separate billable feature.

GMI Inference Engine (serverless). Automatic prefix caching is enabled for the managed model library. Requests that share a stable prefix benefit from faster prefill (KV values are retrieved from cache rather than recomputed) and lower effective GPU cost per request. The prompt structure rules apply equally: stable content must come before dynamic content for the cache to activate.

GMI Prime Inference (dedicated endpoints). SGLang's RadixAttention provides trie-based prefix sharing across all requests to the same endpoint. System prompts, tool definitions, and retrieved documents that appear as shared prefixes across requests are automatically cached and shared. Session affinity configuration ensures that multi-turn conversation history remains on the same serving instance, enabling turn-level KV cache reuse without session rebuild.

Self-hosted models on GMI GPU clusters. Teams deploying their own vLLM or SGLang instances on GMI H100 or H200 clusters configure prefix caching through the serving framework. vLLM's --enable-prefix-caching flag and SGLang's default RadixAttention provide the same prefix sharing benefits without per-token cache billing.

Conclusion

Prompt caching is the highest-ROI optimization available for production LLM applications with repeated context. It requires no change to model quality, no change to output behavior, and in most cases a single structural change to the prompt: moving stable content before dynamic content and adding a cache breakpoint marker where applicable.

For a customer service agent with a 3,000-token system prompt, caching reduces daily input costs by 70 percent or more. For a RAG pipeline with 5,000-token document contexts, caching reduces per-query input costs for repeat document queries by 90 percent. For an agentic system with accumulated tool call history, session-level KV cache retention reduces the per-step cost growth that makes long agentic sessions expensive.

The implementation is a few lines of code. The saving, for any workload with significant shared context, pays back that implementation cost within hours.

FAQs

What is the difference between prompt caching and KV cache optimization? They are the same underlying mechanism accessed at different layers. KV cache optimization (PagedAttention, RadixAttention, cache quantization) is configured at the infrastructure layer by the team operating the serving framework. Prompt caching is the API-level interface providers expose to developers, where stable prompt prefixes are cached server-side and cache hits are charged at a discounted rate. On self-hosted serving, you get the throughput benefit of KV reuse without separate billing. On managed APIs, you get a cost discount on cache-hit tokens.

Which provider offers the best prompt caching for production workloads? Depends on the workload. Anthropic offers the deepest discount (90 percent) and explicit control over what is cached through cache_control markers, making it best for workloads where you want predictable cache hit rates and transparent cache management. OpenAI's automatic caching requires no code changes and is best for teams that want caching with zero implementation overhead, accepting a smaller discount (50 percent) and less control. Kimi K3 offers 90 percent cache discounts with cache-aware pricing built into its flat per-token rate, making it competitive for long-context agentic workloads. For self-hosted models on GMI Cloud, caching is part of the serving stack with no per-token billing.

How do I know if my prompt caching is actually working? Three checks confirm working cache implementation. First, the Anthropic response includes cache_read_input_tokens greater than zero on requests after the first. OpenAI responses include prompt_tokens_details.cached_tokens. Second, the effective input cost per request (total input cost / total input tokens) should be significantly below the standard rate for workloads with large stable prefixes. Third, TTFT for cached requests should be meaningfully lower than for cold requests, because cached tokens skip prefill computation.

Can prompt caching be used with RAG systems? Yes, and RAG is one of the highest-value use cases. Structure the prompt with retrieved documents in the stable prefix position, before the user query. If the same documents are retrieved repeatedly (common when the document corpus is small or when popular documents dominate retrieval), those documents cache at near-100 percent hit rate after the first query. Sort retrieved documents by a stable key (document ID rather than retrieval score) to ensure byte-identical ordering across requests that retrieve the same documents.

What is the minimum prompt length for caching to be worthwhile? The practical minimum for meaningful savings is 1,000 tokens of stable prefix with at least 1,000 daily requests. Below 1,000 tokens, the absolute dollar saving per request is small. Below 1,000 daily requests, the total daily saving is small relative to implementation time. The highest-ROI implementations cache system prompts above 2,000 tokens in applications with 5,000 or more daily requests.