RAG Infrastructure for Production AI: Embedding Models, Vector Search, and GPU Requirements

Retrieval-augmented generation is usually discussed as a prompt engineering pattern: retrieve relevant documents, put them in the context, let the model answer. At production scale it is an infrastructure problem with two distinct inference workloads that have almost nothing in common. Embedding inference processes short inputs, produces small fixed-size outputs, runs at high throughput, and fits on a fraction of a GPU. Generation inference processes long inputs, produces variable-length outputs, is memory-bandwidth-bound, and consumes most of a GPU. Teams that size RAG infrastructure by thinking about the generation model alone underestimate the embedding layer's throughput requirements and overestimate its hardware needs, usually at the same time.

• The embedding layer and the generation layer are different workloads on different hardware profiles. An embedding model with 600 million parameters occupies roughly 1.2 GB at FP16 and can serve thousands of requests per second on a single GPU. A 70B generation model occupies 70 GB at FP8 and serves 10 to 15 requests per second at typical output lengths.
• Embedding dimension is a direct multiplier on storage, memory, and retrieval cost. Retrieval quality flattens after 768 dimensions for most tasks. Going from 1,536 to 3,072 dimensions produces marginal recall gains at roughly six times the storage cost. For most production RAG, 768 or 1,024 dimensions is the practical sweet spot.
• GMI Cloud provides warm dedicated endpoints for both layers. This matters specifically for embedding inference: independent testing found that self-hosted embedding models matched top managed APIs on retrieval quality, but p95 latency roughly doubled without a warm GPU. Cold-start latency on the embedding layer directly inflates query latency for every RAG request.
• MTEB leaderboard position is a shortlist signal, not a selection criterion. MTEB averages across 56-plus tasks including classification and clustering. RAG depends on retrieval, and a model that dominates classification may underperform on retrieval. Use retrieval-specific NDCG@10 scores, then evaluate on your own corpus.
• The retrieval latency budget competes with the generation latency budget. In a RAG pipeline, embedding the query, searching the vector index, optionally reranking, and then generating all happen sequentially before the user sees a token. Every millisecond spent in retrieval is a millisecond unavailable for generation.
• Hosted embedding APIs run $0.02 to $0.15 per million tokens. Self-hosting is not free: the weights are free, and you pay in GPU time and VRAM. The crossover depends on embedding volume, and for most teams the deciding factor is data residency rather than cost.

The Two Inference Workloads in a RAG Pipeline

A RAG pipeline runs inference twice per query, on two models with fundamentally different resource profiles.

Embedding inference: short input, fixed output, high throughput.

The query is converted to a vector. Input length is typically 10 to 100 tokens for a user query, or 200 to 800 tokens for a document chunk during ingestion. Output is a fixed-size vector: 768, 1,024, 1,536, or 3,072 floating point numbers depending on the model.

Resource profile: small model, minimal VRAM, compute-bound rather than memory-bandwidth-bound, extremely high throughput per GPU. An embedding model with 600 million parameters occupies approximately 1.2 GB at FP16. Dozens of concurrent requests fit trivially in the VRAM of any modern datacenter GPU.

Generation inference: long input, variable output, memory-bandwidth-bound.

The retrieved documents plus the user query are sent to the LLM. Input length is typically 2,000 to 20,000 tokens depending on how many chunks are retrieved and how large they are. Output is variable, typically 200 to 1,000 tokens.

Resource profile: large model, substantial VRAM, memory-bandwidth-bound during decode, throughput measured in tokens per second. Llama 3.3 70B at FP8 occupies 70 GB and serves 2,000 to 3,000 output tokens per second in aggregate with continuous batching.

The infrastructure implication.

These two workloads should not share the same GPU configuration in a high-volume deployment. An embedding model deployed alongside a 70B generation model on the same GPU consumes VRAM that the generation model needs for KV cache, while receiving far more compute capability than it requires.

The correct architecture at scale separates them: embedding inference on a smaller, cheaper GPU allocation sized for throughput, generation inference on a GPU sized for the model's memory footprint and KV cache requirements. At lower volume, colocating them on one GPU is simpler and acceptable.

Choosing an Embedding Model

Embedding model selection has more impact on RAG quality than most teams give it credit for, and the MTEB leaderboard is a poor selection tool used alone.

Why MTEB rankings mislead for RAG.

MTEB (Massive Text Embedding Benchmark) covers 56 or more tasks across retrieval, classification, clustering, reranking, and semantic similarity. The headline score is an average across all task types. A model that dominates classification and clustering can pull its average up while underperforming on retrieval, which is the only task type that matters for RAG.

The correction: use retrieval-specific NDCG@10 scores from the retrieval sub-leaderboard rather than the overall MTEB average. Then verify on your own corpus, because benchmark performance on Wikipedia and legal documents does not predict performance on an internal ticketing system or a product catalog.

The score-per-parameter tradeoff.

Larger embedding models score higher and cost more to run. The efficiency spread is substantial:

Qwen3-Embedding-0.6B scores 64.34 on MTEB with 600 million parameters. KaLM-Embedding-Gemma3-12B scores 72.32 with 11.76 billion parameters. The larger model gains 8 points of MTEB score for roughly 20 times the parameter count, which translates directly into GPU cost and latency.

For most production RAG, the smaller model is the correct choice unless retrieval quality is measurably the bottleneck. The test: measure Recall@10 on your own retrieval set with both models. If the smaller model's recall is within 2 to 3 percentage points, the quality difference is unlikely to be visible to users after the generation step.

Model options by requirement.

The self-hosting latency caveat.

Independent testing in mid-2026 found that self-hosted Qwen3-Embedding-8B matched a top managed API on English retrieval quality, but its p95 latency roughly doubled without a warm GPU. This is the same cold-start problem that affects generation inference, and it is more damaging in RAG because the embedding step is on the critical path before generation begins.

An embedding endpoint that occasionally takes 400 milliseconds instead of 40 adds 360 milliseconds to the total RAG query latency for the affected requests. On GMI Prime Inference, reserved capacity keeps the embedding model warm in VRAM continuously, which eliminates the cold-start variance that self-hosted embedding deployments on serverless infrastructure produce.

Embedding Dimension: The Cost Multiplier Nobody Budgets For

Embedding dimension determines the size of every vector in the index. It is a direct multiplier on three production costs.

Storage. Linear in dimension. A corpus of 10 million chunks at 768 dimensions and 4 bytes per float occupies approximately 30 GB. At 3,072 dimensions, the same corpus occupies approximately 123 GB.

Memory footprint for the ANN index. Linear in dimension. HNSW and similar approximate nearest neighbor indices hold vectors in memory for fast search. A 123 GB index requires a machine with 123 GB or more of RAM, versus 30 GB for the 768-dimension version.

Retrieval latency. Sublinear in dimension, because HNSW scales with the logarithm of index size rather than raw dimension count. The latency penalty for higher dimensions is real but smaller than the storage penalty.

Where the quality curve flattens.

Retrieval quality does not scale linearly with dimension. Independent evaluation shows Recall@10 barely moving from 3,072 to 1,024 dimensions, then falling off more sharply below 512. The practical implications:

Going from 256 to 768 dimensions produces meaningful recall gains. Going from 1,536 to 3,072 produces marginal gains at roughly six times the storage cost. For most RAG applications, 768 or 1,024 dimensions is the sweet spot.

Matryoshka representation learning changes the tradeoff.

Models trained with Matryoshka representation learning (MRL) produce embeddings that can be truncated to a shorter dimension while retaining most of their quality. This allows indexing at a lower dimension without changing models: embed at 3,072 dimensions, truncate to 1,024 for the index, and retain most of the retrieval quality at one third of the storage cost.

Gemini Embedding 001 ships 3,072 dimensions with MRL truncation support. Voyage-4 and voyage-4-large support Matryoshka dimensions. For teams that want to reduce vector storage without re-embedding a large corpus, MRL truncation is the mechanism.

Vector Search: Sizing and Selection

The vector database is the layer where retrieval latency and storage cost are determined.

The three architectural options.

Managed vector databases (Pinecone, Weaviate Cloud, Qdrant Cloud) handle index management, scaling, and replication. The tradeoff is per-query cost and data residency: the vectors and their metadata live in the provider's infrastructure.

Self-hosted vector databases (Qdrant, Weaviate, Milvus, pgvector) run on your infrastructure with full data control. The tradeoff is operational responsibility: index tuning, replication, backup, and scaling become your problem.

Vector search in an existing database (pgvector in PostgreSQL, vector types in Oracle, MongoDB Atlas Vector Search) avoids introducing a new system. For corpora below a few million vectors, this is often the correct choice: the operational simplicity of one fewer system outweighs the performance advantage of a purpose-built vector database.

Index type selection.

HNSW (Hierarchical Navigable Small World) is the default for most production RAG. Fast queries, good recall, higher memory usage because the graph structure is held in RAM.

IVF (Inverted File Index) uses less memory and is faster to build, at the cost of lower recall at equivalent search parameters. Appropriate for very large corpora where HNSW memory requirements become prohibitive.

Flat (exhaustive search) computes distance to every vector. Perfect recall, latency linear in corpus size. Appropriate below roughly 100,000 vectors, where exhaustive search completes in single-digit milliseconds and the recall guarantee is worth it.

Sizing the index.

Index memory requirement is approximately: number of vectors × dimension × 4 bytes × 1.5 (HNSW graph overhead).

For 10 million chunks at 1,024 dimensions: 10,000,000 × 1,024 × 4 × 1.5 = approximately 61 GB of RAM.

This is the number that determines the machine size for a self-hosted vector database, and it is frequently underestimated because teams calculate raw vector storage without the graph overhead.

Chunking: The Decision That Constrains Everything Downstream

Chunk size determines how much context each retrieved unit carries, how many chunks fit in the generation model's context window, and how precisely retrieval can target relevant content.

The tradeoff.

Small chunks (100 to 300 tokens) produce precise retrieval: the retrieved chunk is tightly relevant to the query. The cost is fragmentation: a concept explained across three paragraphs may be split across chunks, and retrieving one chunk gives the model incomplete information.

Large chunks (1,000 to 2,000 tokens) preserve context: the retrieved chunk contains the full explanation. The cost is precision and token consumption: more irrelevant content per retrieved chunk, and fewer chunks fit in the generation context window.

Practical defaults.

For question answering over documentation: 400 to 600 tokens per chunk with 50 to 100 token overlap between adjacent chunks. The overlap prevents concepts at chunk boundaries from being split.

For code retrieval: chunk at semantic boundaries (function, class, module) rather than fixed token counts. A function split across chunks is worse than a chunk that is longer than the target size.

For conversational or transcript data: chunk at speaker turns or topic boundaries rather than token counts.

The infrastructure implication of chunk size.

Chunk size determines the total vector count for a given corpus, which determines index size and storage cost. Halving the chunk size doubles the vector count, doubles the index memory, and doubles the embedding inference cost during ingestion.

For a 10 million word corpus: at 500 tokens per chunk, approximately 27,000 chunks. At 250 tokens per chunk, approximately 54,000 chunks. The embedding inference cost, index size, and query-time search cost all double.

The RAG Latency Budget

A RAG query runs four operations sequentially before the user sees the first generated token.

Two observations from this budget.

The retrieval stages consume 15 to 30 percent of the total budget. For an interactive application targeting sub-500ms TTFT, the retrieval stages need to complete in under 100 milliseconds combined, which requires a warm embedding endpoint and a well-tuned index.

Prompt caching on the retrieved chunks is the largest single optimization. The retrieved documents are the bulk of the input tokens in a RAG prompt. When the same documents are retrieved across multiple queries, prefix caching eliminates the prefill cost for those tokens entirely. This requires that the retrieved chunks appear in a stable position in the prompt (before the user query) and in a deterministic order.

The ordering detail matters and is frequently missed: if retrieved chunks are ordered by relevance score, and the score ordering varies slightly between queries that retrieve the same documents, the prefix is not byte-identical and the cache does not hit. Sorting retrieved chunks by a stable key (document ID or content hash) rather than by relevance score preserves cache hits at no cost to retrieval quality, because the model attends to all provided context regardless of order.

Reranking: When the Extra Latency Is Worth It

A reranker is a second-stage model that scores retrieved chunks for relevance to the query more accurately than embedding similarity alone. Retrieve 50 chunks by vector similarity, rerank them, pass the top 5 to the generation model.

When reranking improves RAG quality meaningfully.

Embedding similarity is a coarse relevance signal. It captures semantic proximity but not query-specific relevance nuances. A reranker that processes the query and each candidate chunk together (cross-encoder architecture) evaluates relevance more accurately than comparing independently computed embeddings.

Reranking helps most when: the corpus contains many superficially similar documents where fine distinctions matter, queries are specific and the top-5 by embedding similarity frequently misses the best match, or the generation model's context window limits how many chunks can be passed and precision in chunk selection is therefore critical.

The cost.

Reranking adds a third inference workload with its own latency (30 to 150 milliseconds depending on the reranker size and candidate count) and its own compute cost. For a latency-sensitive application, this may consume the entire remaining budget.

The evaluation that decides it.

Measure end-to-end answer quality with and without reranking on a representative query set. If the quality improvement is small, the latency and cost are not justified. If the improvement is substantial, reranking is the highest-ROI addition to the pipeline.

GPU Sizing for RAG Infrastructure

RAG infrastructure needs GPU capacity for two or three inference workloads with different sizing logic.

Embedding inference sizing.

Throughput-driven rather than memory-driven. An embedding model with 600 million parameters occupies 1.2 GB at FP16, so VRAM is not the constraint. The constraint is request throughput.

Ingestion is the high-throughput phase: embedding a 10 million word corpus at 500 tokens per chunk means 27,000 embedding calls. This is a batch workload that benefits from large batch sizes and can run on spot or on-demand capacity.

Query-time embedding is a low-volume, latency-sensitive workload: one embedding call per user query. A single GPU with the model warm serves thousands of queries per second, so throughput is rarely the constraint. Latency consistency is, which is why a warm dedicated endpoint matters more than raw capacity.

Generation inference sizing.

Memory-driven. Compute the weight footprint at your target precision, add KV cache for your context length distribution and target concurrency. RAG contexts are longer than typical chat contexts because they include retrieved documents, which increases KV cache requirements proportionally.

For Llama 3.3 70B at FP8 (70 GB) serving RAG queries with 8,000-token contexts at 20 concurrent sessions: 70 GB weights plus approximately 51 GB KV cache plus overhead. This exceeds H100's 80 GB and fits H200's 141 GB.

GMI Cloud GPU options for RAG. H100 from $2.00 per GPU-hour, H200 from $2.60, B200 from $4.00, and GB200 from $8.00 as of the March 2026 pricing page. For a typical RAG deployment: the embedding model on a fraction of an H100 (or colocated with other small models), and the generation model on H200 if contexts are long and concurrency is meaningful.

Colocating versus separating.

At low volume (under a few hundred queries per hour), colocating the embedding and generation models on one GPU is simpler and cheaper. The embedding model's 1.2 GB is negligible against a 70B model's 70 GB.

At higher volume, separating them allows independent scaling: the embedding layer scales with query rate, the generation layer scales with token throughput, and these do not grow at the same ratio. As covered in GMI Cloud's analysis of high-concurrency inference workloads, workloads with different scaling characteristics benefit from independent capacity allocation rather than shared provisioning.

The Data Residency Question for RAG

RAG has a specific data governance property that pure generation does not: the corpus is your data, and it goes through the embedding model.

For a company embedding internal documentation, customer records, legal contracts, or proprietary technical content, every chunk of that corpus is sent to the embedding model during ingestion. If the embedding model is a managed API, the entire corpus passes through the provider's infrastructure.

This makes embedding model selection a data governance decision, not only a quality and cost decision. Three paths:

Managed embedding API with acceptable terms. For corpora that are not sensitive, or where the provider's data processing agreement covers the requirement, a managed API is the simplest option at $0.02 to $0.15 per million tokens.

Self-hosted open-weight embedding model. The corpus never leaves your infrastructure. Qwen3-Embedding, BGE-M3, nomic-embed-text, and E5 variants are all open weight and competitive on retrieval quality.

Self-hosted on region-pinned infrastructure. For teams with geographic data residency requirements, the embedding model runs on infrastructure in the required region. GMI Prime Inference's region-pinned endpoints across APAC, North America, and Europe support this configuration.

The generation model has the same consideration for the retrieved chunks, which are also your data. A pipeline that self-hosts the embedding model and sends retrieved chunks to a managed generation API has resolved half the data flow.

Putting It Together: A RAG Infrastructure Reference

For a production RAG deployment serving a 10 million word internal corpus with 1,000 queries per day:

Embedding layer. Qwen3-Embedding-0.6B at 1,024 dimensions (truncated from native if the model supports MRL). Deployed warm on a shared GPU allocation. Ingestion of 27,000 chunks runs as a batch job. Query-time embedding is one call per query at 10 to 40 milliseconds warm.

Vector index. 27,000 vectors at 1,024 dimensions with HNSW. Index memory: approximately 165 MB, which fits comfortably in pgvector inside an existing PostgreSQL instance. No separate vector database needed at this corpus size.

Generation layer. Llama 3.3 70B at FP8 on H200, or routed through a managed API depending on data residency requirements. Context per query: 5 chunks at 500 tokens plus the query, approximately 2,700 tokens. Prompt caching on the retrieved chunks where documents repeat across queries.

Total inference cost. 1,000 queries per day at 2,700 input tokens and 400 output tokens. At GMI Cloud's managed inference rates, this is a small fraction of a single GPU-hour per day for the generation layer, plus negligible embedding cost.

When to scale up. At 100,000 queries per day, the embedding layer needs its own dedicated endpoint for latency consistency, the vector index likely justifies a purpose-built vector database, and the generation layer needs dedicated GPU capacity with prompt caching aggressively configured.

For teams evaluating the model options across both layers, GMI Cloud's inference platform provides embedding and generation models through a unified API, which makes it practical to test model combinations on the same infrastructure before committing.

Conclusion

RAG infrastructure is two inference workloads with opposite resource profiles: a small embedding model that needs throughput and latency consistency, and a large generation model that needs VRAM and memory bandwidth. Sizing them with the same logic produces either an over-provisioned embedding layer or an under-provisioned generation layer.

The decisions that matter most, in order: embedding dimension (768 or 1,024 for most workloads, because quality flattens above that and storage scales linearly), embedding model size (smaller models are usually correct unless retrieval quality is measurably the bottleneck), chunk size (which determines vector count, index size, and ingestion cost), and whether the embedding layer runs warm (because cold-start variance on the embedding step inflates every RAG query's latency).

The data governance dimension is specific to RAG and often overlooked: the corpus passes through the embedding model during ingestion, which makes embedding model selection a data residency decision as much as a quality decision.

FAQs

What embedding dimension should a production RAG system use? 768 or 1,024 dimensions is the practical sweet spot for most workloads. Retrieval quality does not scale linearly with dimension: Recall@10 barely moves from 3,072 to 1,024 dimensions and falls off more sharply below 512. Meanwhile, dimension is a linear multiplier on both vector storage and ANN index memory. A 10 million chunk corpus occupies approximately 30 GB at 768 dimensions and 123 GB at 3,072. Models trained with Matryoshka representation learning allow embedding at high dimension and truncating for the index, which retains most quality at a fraction of the storage cost.

Why is the MTEB leaderboard a poor way to select an embedding model for RAG? MTEB averages across 56 or more tasks including classification, clustering, reranking, and semantic similarity. RAG depends on retrieval specifically. A model that dominates classification can pull its overall average up while underperforming on retrieval. The correction is to use retrieval-specific NDCG@10 scores from the retrieval sub-leaderboard rather than the overall average, then evaluate the shortlist on your own corpus. Benchmark performance on Wikipedia and legal documents does not predict performance on an internal ticketing system or product catalog.

How much GPU capacity does the embedding layer of a RAG pipeline need? Far less than teams expect. An embedding model with 600 million parameters occupies approximately 1.2 GB at FP16, so VRAM is not the constraint. A single GPU with the model warm serves thousands of queries per second, which means throughput is rarely the constraint at query time either. The actual requirement is latency consistency: independent testing found that self-hosted embedding models matched top managed APIs on quality but roughly doubled p95 latency without a warm GPU. Because the embedding step is on the critical path before generation begins, cold-start variance inflates every RAG query's total latency.

Does prompt caching work for RAG, and what breaks it? Yes, and it is the largest single latency and cost optimization available for RAG. The retrieved documents are the bulk of the input tokens, so caching them eliminates prefill cost for repeat retrievals. Two requirements: the retrieved chunks must appear in a stable position in the prompt (before the user query, not after), and they must appear in a deterministic order. The ordering detail is frequently missed: sorting retrieved chunks by relevance score means the order varies slightly between queries that retrieve the same documents, producing a non-identical prefix and a cache miss. Sorting by a stable key such as document ID preserves cache hits at no cost to retrieval quality.

When does a RAG deployment need a dedicated vector database rather than pgvector? Below roughly a few million vectors, vector search inside an existing PostgreSQL instance via pgvector is usually the correct choice: the operational simplicity of one fewer system outweighs the performance advantage of a purpose-built vector database. For a 27,000 chunk corpus at 1,024 dimensions, the HNSW index requires approximately 165 MB of memory, which fits comfortably in an existing database instance. The transition point is when index memory requirements or query latency at your corpus size exceed what the existing database can serve, which for HNSW is approximately number of vectors times dimension times 4 bytes times 1.5 for graph overhead.