Scaling Agent Infrastructure: From 10 to 10,000 Concurrent Agent Sessions

Agent infrastructure fails in a predictable order as concurrency grows, and the failure that appears first is almost never the one teams prepared for. Most capacity planning focuses on GPU throughput: how many inference calls per second can the model serve. That constraint matters, but it is rarely the first thing to break. At 100 concurrent sessions, database connection pools exhaust. At 500, the shared state store becomes an I/O bottleneck. At 1,000, KV cache memory pressure forces evictions that break session continuity. At 5,000, session affinity assumptions collapse under load balancer rebalancing. GPU saturation, the constraint everyone plans for, typically arrives after all of these.

• The scaling bottleneck moves as concurrency grows, and it starts on the CPU side. Orchestration logic, tool call execution, and state management run on CPU. Teams that size infrastructure by GPU count alone hit CPU-bound bottlenecks well before GPU saturation.
• Connection pool exhaustion is the most common first failure. An agent that opens a database connection per tool call, running 500 concurrent sessions with 3 tool calls in flight each, needs 1,500 concurrent connections. Default pool sizes are typically 10 to 50.
• GMI Agentbox provides the model and compute layer with access to 170-plus models through a unified API, and GMI Prime Inference provides the dedicated GPU capacity with session affinity and KV cache persistence that agent workloads require above moderate concurrency.
• KV cache is the GPU-side constraint that binds before compute. Agent sessions accumulate context across steps. At 1,000 concurrent sessions averaging 8,000 tokens of accumulated context, KV cache demand reaches 2.5 TB, far exceeding any single GPU and requiring either aggressive eviction, cache quantization, or distributed cache management.
• Session affinity becomes harder to maintain as the endpoint count grows. At 2 serving instances, consistent hashing keeps sessions pinned reliably. At 20 instances with autoscaling, instance churn breaks affinity for sessions whose target instance is removed, forcing full context rebuild mid-conversation.
• The economics change shape at scale, not just magnitude. At 10 concurrent sessions, serverless inference is clearly cheaper. At 10,000, dedicated capacity with high sustained utilization is clearly cheaper. The crossover is not a single point but a region where the right architecture is a hybrid.

The Scaling Failure Sequence

Agent infrastructure breaks in a consistent order. Knowing the sequence lets teams fix the next bottleneck before it becomes an incident rather than after.

10 to 50 concurrent sessions: nothing breaks.

At this scale, default configurations work. Connection pools are adequate, GPU capacity is ample, state stores handle the load, and session affinity is trivially maintained because there are few serving instances. This is the range where prototypes and early production deployments operate, and where infrastructure problems remain invisible.

50 to 200 sessions: connection pool exhaustion.

The first failure. Agents make tool calls, tool calls open connections to databases and external APIs, and connections are held for the duration of the call. With 3 tool calls in flight per session on average, 200 concurrent sessions require 600 concurrent connections. A default connection pool of 20 to 50 exhausts, and new tool calls queue waiting for a connection.

The symptom is misleading: tool call latency increases, which looks like an external API slowdown. The actual cause is queue time waiting for a pool connection, not the API's response time.

200 to 500 sessions: shared state store contention.

Agents read and write session state: conversation history, accumulated context, tool call results, intermediate reasoning. In multi-agent systems, agents also read each other's outputs through the shared store.

At 500 concurrent sessions each performing several state reads and writes per step, the shared store handles thousands of operations per second. A single Redis instance or a modest database handles this, but a poorly indexed query pattern, a hot key that all sessions read, or write contention on a shared counter produces latency that grows superlinearly with session count.

500 to 1,500 sessions: KV cache pressure.

Agent sessions accumulate context. A session at step 20 with 8,000 tokens of accumulated history requires approximately 2.5 GB of KV cache at FP16 for a 70B model. At 500 concurrent sessions with that context profile, total KV cache demand is 1.25 TB, which exceeds the VRAM of any single GPU and most multi-GPU nodes.

The serving framework responds by evicting KV cache for less recently used sessions. When an evicted session's next step arrives, its context must be fully recomputed. The symptom is per-step latency that varies wildly: some steps complete in 200ms, others in 2 seconds, depending on whether the session's cache survived.

1,500 to 5,000 sessions: GPU saturation.

This is the constraint teams planned for, and it arrives fourth. At this point, request volume exceeds what the available GPUs can serve at acceptable latency, and requests queue at the inference layer.

5,000 to 10,000+ sessions: session affinity breakdown.

At this scale, multiple serving instances are required and autoscaling adjusts instance count with load. Every scaling event redistributes the consistent hash ring, breaking affinity for sessions whose target instance changed. Those sessions lose KV cache continuity mid-conversation and pay full context rebuild on their next step.

The symptom is a correlation between autoscaling events and latency spikes that does not appear in per-instance metrics, because the latency is caused by cache misses following redistribution rather than by any instance being slow.

Bottleneck 1: Connection Pool Sizing

The math.

Required connections = concurrent sessions × average concurrent tool calls per session × connection hold ratio.

For 1,000 concurrent sessions where each session has 2 tool calls in flight on average, and connections are held for the duration of the call:

1,000 × 2 = 2,000 concurrent connections required.

Most default pool configurations are 10 to 50. Even generously configured pools of 200 exhaust at 100 concurrent sessions with this profile.

Three mitigations, in order of effectiveness.

Connection pooling with proper sizing.

Set the pool size based on the calculation above with 20 percent headroom. For 1,000 sessions at 2 concurrent tool calls, size the pool at 2,400. Verify the backing service can accept that many connections: PostgreSQL's default max_connections is 100, which becomes the binding constraint regardless of application-side pool size.

Connection multiplexing.

For databases, use a connection proxy (PgBouncer for PostgreSQL, ProxySQL for MySQL) that multiplexes many application connections onto a smaller number of backend connections. This decouples the application-side pool size from the database's connection limit.

Reducing connection hold time.

Tool calls that hold a connection while performing slow operations (a query that scans a large table, an external API call that takes 2 seconds) tie up pool capacity for the duration. Optimizing the slow operations reduces the required pool size proportionally.

The measurement that identifies this bottleneck: track connection pool wait time separately from tool execution time. If wait time is a meaningful fraction of total tool call latency, the pool is undersized.

‍

Bottleneck 2: Shared State Store Throughput

Agent state operations grow with both session count and steps per session.

The math.

State operations per second = concurrent sessions × steps per second per session × state operations per step.

For 1,000 concurrent sessions, each completing one step every 3 seconds, with 4 state operations per step (read context, read prior results, write step result, update session status):

1,000 × 0.33 × 4 = approximately 1,320 operations per second.

A single Redis instance handles this comfortably. The problems appear in the access pattern rather than the raw volume.

Three access patterns that break at scale.

Hot keys.

A shared configuration object, a global counter, or a shared context that every session reads produces a single key receiving all traffic. Redis handles high read volume on a single key well, but write contention on a hot key serializes and becomes a bottleneck.

Fix: partition hot keys by session or shard, or cache read-heavy shared data in each serving instance's local memory with a TTL rather than reading from the shared store on every access.

Large value reads.

An agent that stores accumulated context in a single key and reads the entire context on every step transfers megabytes per read. At 1,000 sessions reading 2 MB of context every 3 seconds, that is 660 MB/s of network traffic to the state store.

Fix: store context incrementally (append-only step records) and read only the delta since the last read, or keep hot context in the serving instance's local memory with the shared store as the durable backing.

Unindexed queries in multi-agent coordination.

In multi-agent systems where an orchestrator queries "which sub-agents have completed," an unindexed scan across all session records grows linearly with total session count. At 10,000 sessions this becomes prohibitively slow.

Fix: maintain explicit indexes on the query patterns the orchestration layer uses, or use a purpose-built task queue rather than querying a general state store.

Bottleneck 3: KV Cache at Agent Scale

KV cache is the GPU-side constraint that binds before compute for agent workloads specifically, because agents accumulate context in a way that stateless inference does not.

The math.

KV cache per session = accumulated context tokens × per-token cache size.

For Llama 3.3 70B with grouped query attention at FP16 KV cache, per-token cache is approximately 0.32 MB.

A single H200 has 141 GB of VRAM, of which roughly 71 GB is available for KV cache after a 70B model at FP8. That supports approximately 27 sessions at 8,000 tokens of context.

Four mitigations.

KV cache quantization.

INT8 KV cache halves the per-token cache size to 0.16 MB, doubling the sessions per GPU. INT4 quarters it. Quality impact for agent workloads is typically small: KV values participate in attention score computation, and the rounding error introduced by quantization is minor relative to overall generation quality for most tasks.

Context summarization at thresholds.

Rather than accumulating full context indefinitely, summarize accumulated history when it exceeds a threshold (for example, 6,000 tokens), replacing the detailed history with a condensed summary. This caps per-session KV cache at the threshold rather than allowing unbounded growth.

Aggressive cache eviction with checkpoint recovery.

Accept that KV cache will be evicted for idle sessions, and design for fast recovery: checkpoint the session state so that when an evicted session resumes, the context is rebuilt from a compact checkpoint rather than replayed from the full history.

Distributed KV cache across GPUs.

For very large deployments, KV cache can be distributed across a GPU cluster with sessions routed to the GPU holding their cache. This is effectively session affinity applied at the GPU level rather than the serving instance level.

Bottleneck 4: GPU Capacity Planning for Agent Workloads

GPU capacity planning for agents differs from stateless inference planning in one important respect: the relevant metric is steps per second across all sessions, not requests per second from users.

The math.

Required inference throughput = concurrent sessions × steps per second per session.

An agent session that completes one step every 3 seconds, at 1,000 concurrent sessions, generates 333 inference requests per second. This is the load the GPU layer must serve.

A single H100 serving Llama 3.3 70B at FP8 with continuous batching handles roughly 2,000 to 3,000 output tokens per second in aggregate. At 200 output tokens per agent step, that is 10 to 15 steps per second per GPU.

333 steps per second requires 22 to 33 H100s at that throughput profile.

Two factors that change this calculation substantially.

Model size per step.

Not every agent step requires the same model. Classification and routing steps can use a 32B model at significantly higher throughput than reasoning and synthesis steps that need a 70B model. Routing steps to appropriately sized models reduces the GPU requirement for the classification-heavy portion of the workload.

Prefill versus decode balance.

Agent steps typically have long inputs (accumulated context) and short outputs (a tool call or a brief reasoning step). This makes agent workloads more prefill-heavy than conversational workloads. Prefill is compute-bound rather than bandwidth-bound, which changes the optimal serving configuration: chunked prefill becomes important to prevent long prefills from blocking short requests.

Bottleneck 5: Session Affinity Under Autoscaling

Session affinity keeps all steps from a session on the same serving instance, preserving KV cache. It works reliably at small instance counts and degrades as the fleet scales and churns.

Why affinity breaks.

Consistent hashing maps session identifiers to instances. When the instance count changes, the hash ring is redistributed and a fraction of sessions map to a different instance than before. With consistent hashing, adding one instance to a fleet of N redistributes approximately 1/(N+1) of sessions. At a fleet of 20, adding an instance moves 5 percent of sessions.

Those 5 percent lose their KV cache and pay full context rebuild on their next step. If autoscaling is active and instance count changes frequently, this affinity churn is continuous.

Three mitigations.

Drain rather than terminate.

When scaling down, mark the instance as draining (accepting no new sessions) and allow existing sessions to complete before terminating. This eliminates affinity breakage from scale-down events entirely at the cost of slower scale-down.

Sticky session tokens with explicit routing.

Rather than relying on consistent hashing, issue an explicit instance identifier in the session token and route directly to that instance. Affinity is then exact rather than probabilistic, and only breaks when the specific instance becomes unavailable.

Reserved capacity rather than autoscaled capacity for the baseline.

Autoscaling is the source of affinity churn. Reserving capacity for the sustained baseline load and using burst capacity only for peaks reduces the frequency of scaling events that break affinity.

GMI Prime Inference implements the third approach: reserved dedicated capacity holds the baseline with model weights pre-loaded and session state preserved, while elastic burst capacity absorbs peaks without requiring the baseline fleet to scale. This keeps affinity stable for the majority of sessions even during traffic spikes.

The Architecture at Each Scale

10 to 100 concurrent sessions: serverless inference, single state store, default pools.

At this scale, serverless inference is clearly the right economic choice. Idle cost is zero, scaling is automatic, and the concurrency is low enough that shared infrastructure latency variance is tolerable. A single Redis instance handles state. Default connection pool sizes work.

100 to 1,000 sessions: sized pools, connection multiplexing, hybrid inference.

Connection pools must be sized to the calculation above with a multiplexing proxy for databases. The state store access patterns need review for hot keys and large value reads. Inference moves toward a hybrid: dedicated capacity for the latency-critical primary model, serverless for less frequent or batch-tolerant steps.

1,000 to 5,000 sessions: dedicated GPU capacity, KV cache management, session affinity.

Dedicated GPU capacity with session affinity becomes necessary rather than optional. KV cache quantization and context summarization are required to fit the session count into available VRAM. The state store likely needs partitioning or a read replica topology.

5,000 to 10,000+ sessions: multi-region, reserved baseline with burst, distributed state.

At this scale, the architecture is distributed across regions for both capacity and latency. Reserved capacity holds the baseline with elastic burst for peaks, minimizing autoscaling-induced affinity churn. State is partitioned across instances with the shared store handling only cross-partition coordination. Model routing across GPU tiers becomes a meaningful cost lever: classification steps to smaller models, reasoning steps to larger ones.

‍

The Capacity Planning Checklist

Before scaling an agent deployment past its current concurrency, compute these six numbers.

1. Required connections: concurrent sessions × concurrent tool calls per session. Compare against both the application pool size and the backing service's connection limit.

2. State operations per second: concurrent sessions × steps per second per session × state operations per step. Compare against the state store's measured throughput on your access pattern, not its theoretical maximum.

3. Total KV cache demand: concurrent sessions × average accumulated context × per-token cache size. Compare against available VRAM after model weights.

4. Inference throughput required: concurrent sessions × steps per second per session. Compare against measured throughput per GPU for your model and step profile.

5. Affinity churn rate: expected autoscaling events per hour × fraction of sessions redistributed per event. This is the fraction of sessions paying full context rebuild due to affinity breakage.

6. Cost per session at target scale: total infrastructure cost divided by concurrent session capacity. This is the number that determines whether the architecture is economically viable at the target scale.

GMI Cloud Infrastructure for Agent Scaling

Two GMI Cloud products address different layers of the agent scaling problem.

GMI Agentbox provides the model and compute layer with unified access to 170-plus models through an OpenAI-compatible API. For agent pipelines where different steps require different models (a smaller model for classification steps, a larger one for reasoning), the unified API means model selection per step is a parameter rather than a separate integration per provider. Per-session usage tracking and spend attribution provide the measurement foundation for the cost-per-session calculation.

GMI Prime Inference provides the dedicated GPU capacity that agent workloads require above moderate concurrency. Three properties matter specifically for agent scaling:

Reserved capacity with pre-loaded model weights eliminates cold start and provides stable session affinity, because the baseline fleet does not churn under autoscaling.

Per-model runtime tuning includes KV cache configuration calibrated to the workload's actual context distribution, which for agent workloads with accumulating context is a meaningfully different configuration than for stateless chat.

Elastic burst capacity absorbs traffic peaks without requiring the reserved fleet to scale, which keeps affinity stable during exactly the conditions where affinity churn would otherwise be highest.

As covered in GMI Cloud's analysis of high-concurrency inference workloads, the parallel execution patterns that agentic systems create require infrastructure designed for concurrent multi-step execution rather than sequential request handling.

Conclusion

Agent infrastructure scaling is a sequence of bottlenecks that appear in a predictable order, and GPU capacity is fourth on that list rather than first. Connection pools exhaust around 100 to 200 concurrent sessions. State store access patterns degrade around 200 to 500. KV cache pressure binds around 500 to 1,500. GPU saturation arrives around 1,500 to 5,000. Session affinity breaks down above 5,000 when autoscaling churn redistributes the hash ring continuously.

The capacity planning that avoids incidents computes all six numbers before scaling: required connections, state operations per second, total KV cache demand, inference throughput required, affinity churn rate, and cost per session at target scale. Each has a specific mitigation, and each mitigation is significantly cheaper to implement before the bottleneck produces a production incident than after.

GMI Agentbox provides the model and compute layer with unified multi-model access for step-level model selection. GMI Prime Inference provides the reserved dedicated capacity with stable session affinity and KV cache configuration tuned for accumulating agent context.

Deploy scalable agent infrastructure on GMI Cloud

FAQs

What breaks first when scaling agent infrastructure, and why is it not GPU capacity? Connection pool exhaustion is typically the first failure, appearing around 100 to 200 concurrent sessions. Agents make tool calls, tool calls open connections, and connections are held for the call duration. At 200 sessions with 3 tool calls in flight each, 600 concurrent connections are required against default pool sizes of 10 to 50. GPU capacity is the fourth constraint to bind, typically around 1,500 to 5,000 sessions, because orchestration logic, tool execution, and state management all run on CPU and saturate before the GPU layer does.

How do I calculate KV cache requirements for a concurrent agent workload? Multiply concurrent sessions by average accumulated context tokens by per-token cache size. For Llama 3.3 70B with grouped query attention at FP16 KV cache, per-token cache is approximately 0.32 MB. At 1,000 concurrent sessions averaging 8,000 tokens of accumulated context, total demand is 2.56 TB, far exceeding any single GPU. A single H200 with 141 GB VRAM has roughly 71 GB available for KV cache after a 70B model at FP8, supporting approximately 27 sessions at that context length. Mitigations include INT8 KV cache quantization (doubles capacity), context summarization at defined thresholds (caps per-session growth), and checkpoint-based recovery for evicted sessions.

Why does session affinity break down at high concurrency and what fixes it? Session affinity uses consistent hashing to map sessions to serving instances. When the instance count changes through autoscaling, the hash ring redistributes and a fraction of sessions map to a different instance, losing their KV cache and paying full context rebuild. Adding one instance to a fleet of 20 redistributes approximately 5 percent of sessions. With frequent autoscaling, this churn is continuous. Three fixes: drain instances during scale-down rather than terminating them abruptly, use explicit instance identifiers in session tokens rather than probabilistic hashing, or reserve capacity for the baseline load so that autoscaling events are rare.

How much GPU capacity does a given number of concurrent agent sessions require? Required inference throughput equals concurrent sessions times steps per second per session. An agent completing one step every 3 seconds at 1,000 concurrent sessions generates 333 inference requests per second. A single H100 serving Llama 3.3 70B at FP8 with continuous batching handles roughly 10 to 15 steps per second at 200 output tokens per step, so 333 steps per second requires 22 to 33 H100s. Two factors change this substantially: routing classification and simple steps to smaller models increases throughput per GPU, and agent workloads are more prefill-heavy than conversational workloads, which makes chunked prefill configuration important to prevent long prefills from blocking short requests.

At what concurrency should an agent deployment move from serverless to dedicated GPU capacity? Below roughly 100 concurrent sessions, serverless inference is clearly cheaper because idle cost is zero and shared infrastructure latency variance is tolerable. Between 100 and 1,000, a hybrid is usually correct: dedicated capacity for the latency-critical primary model, serverless for less frequent or batch-tolerant steps. Above 1,000 concurrent sessions, dedicated capacity with session affinity becomes necessary rather than optional, because KV cache continuity across steps requires stable session-to-instance mapping that shared serverless infrastructure cannot provide.