Agent Sandbox to Production: A Deployment Readiness Checklist for Agentic Systems

Most agent deployments go live because someone decided they were ready, not because a defined set of criteria was verified. The decision is usually made on the strength of a demo: the agent completes the target task correctly in front of the team, everyone agrees it works, and it ships. Two weeks later the failure modes arrive: a session that costs forty dollars, a tool call that fails in a way the agent cannot recover from, a conversation that hits the context limit mid-task, an authorization gap nobody tested. Every one of those failures is detectable before launch with a defined verification procedure, which is what this checklist provides.

• Deployment readiness is a set of verifications, not a judgment call. Each item below has a specific test, a pass criterion, and a defined owner. The checklist either passes or it identifies what needs fixing before launch.
• Six gates cover the surface area: functional correctness, failure handling, resource bounds, observability, authorization, and operational readiness. Skipping any one of them produces a predictable class of production incident.
• GMI Agentbox provides private deployment as the environment where this verification happens. The agent runs on production infrastructure with real model access and real compute, without any external listing or public access, which is the condition required for meaningful pre-launch testing.
• The most commonly skipped gate is resource bounds. Teams verify that the agent completes tasks correctly and do not verify what happens when it does not terminate. A cost ceiling with automatic session termination is a ten-line implementation that prevents the most expensive category of production incident.
• Load testing at 2x expected peak concurrency is non-negotiable. Agent failure modes at concurrency (connection pool exhaustion, shared state contention, KV cache eviction breaking session continuity) do not appear at the sequential testing that development uses.
• Each gate needs a named owner and a recorded pass. A checklist where items are verified informally and the results live in someone's memory is a checklist that gets partially completed under launch pressure.

Gate 1: Functional Correctness

What this gate verifies: the agent completes its intended tasks correctly across the realistic distribution of inputs, not just the demo case.

Verification procedure.

Build a task set of 20 to 30 tasks drawn from the actual expected usage distribution. Include the typical case that dominates traffic, the edge cases that appear less frequently, tasks at the boundary of the agent's intended scope, and tasks that are just outside scope to verify the agent declines appropriately rather than attempting them badly.

Run each task three times. Agentic systems are non-deterministic: the same task run twice may take different solution paths, call tools in different orders, and produce equally valid but structurally different outputs. A task that succeeds once may fail on the second attempt for reasons that single-run testing does not surface.

Score at the task level, not the step level. Binary task completion is the metric: did the agent accomplish what it was asked to do, verified against objective success criteria defined before the test.

Pass criteria.

Task completion rate above 85 percent across the full task set, with no task type showing a completion rate below 60 percent. A uniform 85 percent is a different production profile than 95 percent on most tasks with total failure on one category, and the second pattern requires either fixing that category or excluding it from the agent's scope.

Common failure at this gate. Testing only the tasks the team wrote as examples. Team-written tasks are phrased the way the team expects the agent to understand them. Real inputs are ambiguous, contain typos, combine multiple requests, and arrive in phrasings the team did not anticipate.

Gate 2: Failure Handling

What this gate verifies: the agent behaves correctly when things go wrong, which is the majority of production incidents.

Verification procedure.

This gate requires deliberate failure injection. Happy-path testing does not reach it.

Tool call failure injection. For each tool the agent uses, inject each failure mode and verify the agent's response: HTTP 429 rate limit, HTTP 503 service unavailable, connection timeout, malformed response body, response with unexpected schema, empty response where content was expected, and response with truncated content.

For each injection, verify: does the agent retry appropriately, does it fall back to an alternative approach, does it inform the user that a step could not complete, or does it silently proceed with incomplete information and produce a confidently wrong final answer?

The last behavior is the dangerous one. An agent that continues after a failed tool call, reasoning around the missing information without flagging it, produces output that appears complete and is wrong. This failure is invisible in output quality scoring unless the injection test specifically checks for it.

Model failure injection. Verify behavior when the model endpoint returns 5xx, times out, or is rate-limited. Confirm that fallback routing applies and that the fallback model can continue the session with the accumulated context intact.

Mid-task interruption. Terminate the agent process mid-task and verify recovery: does the session resume from the last checkpoint, or does it restart from the beginning, or does it fail unrecoverably? For long-horizon agents, checkpoint recovery is the difference between a 5-minute recovery and a lost 40-minute task.

Pass criteria.

Every injected failure produces one of three acceptable outcomes: successful recovery, graceful degradation with the limitation communicated to the user, or clean failure with a diagnostic log entry. Zero instances of silent continuation with incomplete information.

Common failure at this gate. Skipping it entirely because mocked tools always succeed during development. Real tool integrations in a test environment are required; mocks cannot produce the failure modes this gate tests.

Gate 3: Resource Bounds

What this gate verifies: the agent cannot consume unbounded resources, in cost, time, or steps.

This is the most commonly skipped gate and the one that prevents the most expensive class of production incident.

Verification procedure.

Cost ceiling. Verify that a per-session cost limit is enforced and that sessions exceeding it terminate with a logged reason. Test by constructing a task that will exceed the ceiling (an intentionally complex task, or an injected retry loop) and confirming that termination occurs at the threshold rather than after unbounded accumulation.

Set the ceiling at 5 to 10 times the expected session cost: high enough to accommodate legitimately complex tasks, low enough to catch runaway sessions before they accumulate significant cost.

Step count limit. Verify a maximum step count per session (model inference calls plus tool calls combined). Test by injecting a scenario that would produce a loop and confirming termination at the limit.

Step limits catch loops that individually stay within cost bounds but accumulate through repetition. A tool call that costs nothing but repeats 5,000 times is a step-count problem, not a cost problem.

Session duration limit. Verify a maximum wall-clock duration per session with termination and logging. Test with a task that stalls (an injected tool call that never returns) and confirm the timeout fires.

Context length handling. Verify behavior when accumulated context approaches the model's context limit. Test by running a task that accumulates context past the threshold. Acceptable outcomes: summarization at a defined threshold, sliding window truncation with the oldest context dropped, or clean termination with a message to the user. Unacceptable: an unhandled context length error that terminates the session with no recovery.

Pass criteria.

All four limits are enforced with automatic termination and logged reasons. Each was verified by triggering it in test, not by reading the configuration.

Common failure at this gate. Configuring the limits and never testing that they fire. A cost ceiling that is set but not wired to the termination path provides no protection.

Gate 4: Observability

What this gate verifies: when something goes wrong in production, the trace exists to diagnose it.

Verification procedure.

Step-level trace completeness. Run a task, then attempt to reconstruct exactly what the agent did from the logs alone, without access to the running system. For each step, the trace should show: the step sequence number, the step type (model inference, tool call, routing decision), the input, the output or result, the latency, the cost, and the outcome including error details when applicable.

The test: hand the trace to someone who was not present during the run and ask them to explain what the agent did and why it produced the output it produced. If they cannot, the trace is incomplete.

Session correlation. Verify that all steps in a session share a session identifier, and that in multi-agent systems all agent sessions in the same pipeline share a correlation identifier. Test by running a multi-agent task and querying for all actions across all agents in that pipeline execution.

Alert verification. Three alerts must exist and must be verified by triggering them: error rate above baseline, per-session cost above ceiling, and latency p99 above threshold. Trigger each in the test environment and confirm the alert fires and routes to the correct channel.

An alert that is configured but never tested is an alert that may not fire when it matters. Triggering each one before launch is a 15-minute exercise that prevents discovering a broken alert during an incident.

Cost attribution. Verify that cost is attributable to a session and to a user or tenant. Test by running sessions under different user identifiers and confirming that the cost data separates correctly.

Pass criteria.

A complete step-level trace exists for every session and is queryable programmatically. All three alerts fire when triggered in test. Cost attributes correctly to session and principal.

Common failure at this gate. Session-level logging only. Session logs show what the user asked and what the agent returned, which tells you something went wrong but not where. Step-level traces are what make a 15-minute diagnosis possible instead of a 4-hour one.

Gate 5: Authorization

What this gate verifies: the agent cannot take actions outside its authorized scope, regardless of what its reasoning produces.

Verification procedure.

Tool allowlist enforcement. For each tool the agent should not have access to, attempt to invoke it (through a test harness or through an injected instruction) and verify the call is rejected at the tool mediation layer with a logged authorization failure. The rejection must happen in the application layer that mediates tool calls, not in the model's reasoning.

Parameter bounds enforcement. For each authorized tool with parameter limits, attempt calls outside those limits and verify rejection. A refund tool with a $500 ceiling must reject a $5,000 call regardless of what the model generated.

Principal permission intersection. Verify that the agent's effective authorization is the intersection of the agent role's permissions and the invoking user's permissions. Test with a user who lacks a permission that the agent role has, and confirm the action is denied.

Cross-tenant isolation. For multi-tenant systems, attempt to access another tenant's data through tool calls with manipulated tenant identifiers. Verify that tenant scoping is enforced at the data access layer rather than depending on correct agent behavior.

Human-in-the-loop gates. For each irreversible action (deleting data, sending external communications, processing payments, modifying production configuration), verify that a confirmation gate exists, that it displays the exact action and parameters, and that the action cannot proceed without explicit approval.

Pass criteria.

Zero successful unauthorized actions across all tests. This gate is pass or fail with no partial credit, because authorization failures in production expose data or take unauthorized actions rather than merely degrading quality.

Common failure at this gate. Relying on the system prompt to enforce authorization. An instruction telling the agent not to call a tool is a policy statement. Enforcement at the tool mediation layer is a control. The test that distinguishes them: inject an instruction that tries to override the policy and see whether the action executes.

Gate 6: Operational Readiness

What this gate verifies: the team can operate the agent after launch.

Verification procedure.

Load test at 2x expected peak. Run the agent at twice the expected peak concurrency for at least 30 minutes. This surfaces the failure modes that only appear at concurrency: connection pool exhaustion, shared state store contention, KV cache eviction breaking session continuity, and session affinity behavior under load.

Measure during the load test: task completion rate (does it degrade under load?), p99 latency (does it exceed the acceptable threshold?), error rate by error type (which component fails first?), and cost per completed task (does it increase under load due to retries?).

Runbooks for the three most likely failures. Written before launch, not after the first incident. The three: stuck agent (high step count, repeated identical tool calls, session active past expected duration), context overflow (session terminated with a context length error), and cost spike (session cost significantly above average). Each runbook needs the trace signature that identifies the failure, the diagnostic steps, and the immediate and longer-term remediation.

Rollback procedure verified. Confirm that the agent can be disabled or reverted without a deployment. Test the rollback path in the test environment. If rollback requires a code deployment, the mean time to recovery during an incident is the deployment time rather than a configuration change.

On-call ownership defined. A named owner for agent incidents with escalation path, and the alerting configured to reach them. An alert routing to a channel nobody watches is not alerting.

Pass criteria.

Load test at 2x peak completes with task completion rate within 10 percent of the sequential baseline and p99 latency within the acceptable threshold. Three runbooks written. Rollback tested. On-call owner named with verified alert routing.

Common failure at this gate. Load testing at expected peak rather than 2x. Expected peak is an estimate, and the estimate is frequently low. Testing at 2x provides the headroom that keeps a traffic surprise from becoming an incident.

The Sign-Off Table

Each gate needs a recorded pass with a named verifier and a date. A checklist verified informally is a checklist that gets partially completed under launch pressure.

Where Private Deployment Fits

The verification above requires an environment that is functionally equivalent to production but has no external users. Four properties are required.

Real model access on production infrastructure. The agent must run against the actual models and compute it will use in production. Failure modes tied to model behavior, latency characteristics, and throughput limits do not appear when testing against a different model or on different hardware.

Real tool integrations, not mocks. Gate 2 (failure handling) cannot be verified against mocked tools, because mocks always succeed. Real integrations pointed at test instances of the backing services are required.

Complete execution tracing. Gate 4 requires that the trace exists and is queryable. If the test environment lacks the tracing that production will have, the gate cannot be verified in the environment where verification happens.

No external access. The agent must not be reachable by anyone outside the team during verification. An agent that is technically available while failing Gate 5 is an authorization risk in the window before verification completes.

GMI Agentbox's private deployment stage provides these conditions: the agent runs on GMI infrastructure with access to 170-plus models through a unified API and real compute, without any external listing or public access. The platform's per-session logging provides the session-level foundation that step-level traces attach to. Model Scope and Allowed Models settings constrain which models can process the organization's data during verification and after launch.

For agents whose load profile requires dedicated capacity during load testing, GMI Prime Inference provides reserved single-tenant GPU capacity, which makes the 2x peak load test measure the agent's behavior rather than shared-pool contention from other tenants.

After the Checklist: The First 48 Hours

Passing the checklist is the launch gate, not the end of verification. The first 48 hours of production traffic surface behaviors that pre-launch testing does not.

Watch the cost distribution, not the average. The average session cost after launch will be close to the pre-launch estimate. The tail will not be. Sessions in the 99th percentile of cost reveal task types that are more expensive than expected, which either need optimization or need a lower per-session ceiling.

Watch the tool call error rate by tool. Production tool call failure rates are higher than test environment rates, because production external APIs experience real rate limits, real capacity constraints, and real downtime. A tool with a 5 percent error rate in production that showed 0 percent in test needs its retry and fallback configuration reviewed.

Watch task types that were not in the test set. Real users send requests the team did not anticipate. Log and review the task type distribution in the first 48 hours against the distribution in the test set. Task types appearing in production that were absent from testing are unverified and should be evaluated for whether they belong in the agent's scope.

Watch p99 latency against the load test result. If p99 in production is substantially worse than the 2x load test predicted, the load test did not replicate a production condition, most commonly the real network topology, the real tool call latency distribution, or real concurrency patterns rather than synthetic ones.

Conclusion

Agent deployment readiness is verifiable, and the verification takes days rather than weeks. Six gates cover the surface area: functional correctness on a realistic task distribution with multiple runs per task, failure handling verified by deliberate injection, resource bounds tested by triggering each limit, observability confirmed by reconstructing a session from traces alone and firing each alert, authorization verified by attempting unauthorized actions, and operational readiness proven by a 2x peak load test with runbooks and a tested rollback path.

The gates that get skipped most often are resource bounds and failure handling, and they produce the two most expensive categories of production incident: unbounded cost accumulation and silent continuation after a failed tool call. Both are testable in an afternoon.

GMI Agentbox's private deployment provides the environment where this verification happens: production infrastructure, real model access, real compute, complete session logging, and no external exposure until the checklist passes.

Start with GMI Agentbox private deployment

FAQs

Which deployment readiness gate is most commonly skipped, and what does skipping it cost? Resource bounds is the most commonly skipped gate. Teams verify that the agent completes tasks correctly and do not verify what happens when it fails to terminate. The result is the most expensive category of production incident: a session stuck in a retry loop that accumulates cost while every individual action remains authorized and logged, so no security or error alert fires. Four limits prevent it: a per-session cost ceiling set at 5 to 10 times expected cost, a maximum step count, a session duration limit, and defined behavior when context approaches the model's limit. Each must be verified by triggering it in test, not by reading the configuration.

Why is deliberate failure injection necessary rather than relying on happy-path testing? Production tool calls fail at rates that test environments with mocked tools never produce. External APIs return rate limit errors, temporary server errors, malformed responses when schemas change, and truncated content. The dangerous agent behavior is not crashing on a failed tool call; it is continuing to reason around the missing information and producing a confidently wrong final answer that appears complete. This failure is invisible in output quality scoring. Only deliberate injection of each failure mode against real tool integrations surfaces it before production users encounter it.

How large should the pre-launch task set be, and how many times should each task run? Twenty to thirty tasks drawn from the actual expected usage distribution, with each task run three times. The task set must include the typical case that dominates traffic, the edge cases that appear less frequently, tasks at the boundary of scope, and tasks just outside scope to verify appropriate declining. Three runs per task is necessary because agentic systems are non-deterministic: a task that succeeds on the first attempt may fail on the second for reasons single-run testing does not surface. Pass criteria: above 85 percent task completion overall with no task type below 60 percent.

What load level should agents be tested at before production launch? Two times the expected peak concurrency, sustained for at least 30 minutes. Expected peak is an estimate and estimates are frequently low, so 2x provides the headroom that keeps a traffic surprise from becoming an incident. Agent failure modes at concurrency do not appear in sequential development testing: connection pool exhaustion, shared state store contention, KV cache eviction breaking session continuity, and session affinity behavior under load all require concurrent load to surface. Measure task completion rate, p99 latency, error rate by error type, and cost per completed task during the test.

Why does authorization verification need to test the tool mediation layer rather than the system prompt? An instruction in the system prompt telling the agent not to call a specific tool is a policy statement that depends on the model correctly interpreting and following it. Under prompt injection, degraded model behavior, or ambiguous input interpretation, the agent may attempt the prohibited action, and if the only control is the prompt, the action executes. Enforcement at the tool mediation layer rejects the call regardless of what the model generated. The test that distinguishes them: inject an instruction attempting to override the policy and verify whether the action executes. This gate is pass or fail with no partial credit, because authorization failures expose data or take unauthorized actions rather than merely degrading output quality.