Thousands of sandboxes, zero infrastructure: how Oqoqo runs agent evals on GMI Cloud Agentbox

Oqoqo, an agent evaluation platform, uses GMI Cloud's Agentbox to run large-scale agent evaluations with high concurrency, launching thousands of fresh, isolated sandboxes in parallel. A single experiment with 300 tasks, eight agents, and five models requires 12,000 separate runs, each in its own clean environment. Oqoqo supplies the Docker images that define each test environment, while Agentbox provisions and tears down a sandbox for every run. With Agentbox, Oqoqo submits a full evaluation grid as one experiment instead of batching it around a provider quota, and does so without maintaining its own sandbox infrastructure or keeping capacity idle between experiments.

• Up to 50,000 concurrent sandboxes available to a single eval experiment

• 6 to 8 GB of RAM per sandbox, and up to 16 GB with MCP sidecars

• No sandbox infrastructure for Oqoqo to build or maintain

The challenge: 12,000 sandboxes, all at once

Every task has to start clean. A leftover file or a warm cache from the previous task and the number is quietly wrong, and nothing fails to tell you. So each task gets a fresh sandbox from a custom Docker image, runs alone, gets its trajectory recorded, and the environment is destroyed. Sandboxes are never reused. If Oqoqo wants to run all 12,000 tasks in parallel, that means provisioning 12,000 clean sandbox instances at once rather than cycling the experiment through a smaller concurrency pool.

At this scale the hard infrastructure problem is not the model call. It is giving every agent run its own clean, fully configured environment at the same time: the agent, its tools, and the customer's own SDK or MCP server at the exact version being tested, in a box that takes 6 to 8 GB of RAM, up to 16 GB with MCP sidecars. Because evals run from CI, the load arrives when an engineer pushes a commit, not on a schedule. Cap concurrency at 500 and a 12,000-run sweep becomes 24 sequential batches, and what could have been one parallel experiment turns into an hours-long feedback loop.

“Every task should start from the same initial state and be triggered independently, so evaluations are fair across models and harnesses. But our biggest requirement is high concurrency: unlike many other sandbox use cases where storage or compute is the main constraint, we need each user to be able to trigger at least 1,000 concurrent runs.” — Haritha, Oqoqo

How Oqoqo uses Agentbox

Oqoqo and GMI Cloud divide responsibility between evaluation software and sandbox infrastructure. Oqoqo configures each test environment in its own Docker image and launches one Agentbox sandbox per evaluation task. Oqoqo records the agent's actions during each run, while GMI Cloud provides the compute and memory needed to execute the tasks in isolation. Through Agentbox, GMI Cloud manages sandbox creation and teardown, including environments that need additional memory for MCP sidecars. Oqoqo can therefore focus its engineering effort on the evaluation product while GMI Cloud operates the underlying sandbox infrastructure.

“Before Agentbox, a sudden burst of runs could exhaust the sandbox provider’s compute pool. Our system had to detect the capacity error, return the affected runs to the queue, and retry them later.” — Haritha, CTO, Oqoqo

What it adds up to

Oqoqo can send a full sweep to Agentbox as one experiment instead of shaping it around a provider quota. Each task gets the environment and the memory it needs, and Oqoqo can request up to 50,000 concurrent sandboxes for a single eval experiment.

For Oqoqo, that means the infrastructure does not have to dictate the shape of the experiment. A large evaluation grid can be submitted together instead of being broken into smaller batches around a provider quota. Each run still starts from a clean environment, while Oqoqo keeps its engineering effort on the evaluation product rather than on the sandbox platform underneath it.

“Now we run far more of an evaluation in parallel instead of shaping it around a smaller provider quota.” — Haritha, CTO, Oqoqo

If you are building an agent, a developer tool, or an MCP server and you want to know whether agents can actually use it, Oqoqo is at oqoqo.ai. If you need the compute to run evals at this scale, Agentbox is at gmicloud.ai.

About Oqoqo

Oqoqo builds evals and private benchmarks for real agentic tasks. A team can point Oqoqo at its product, including its SDK, API, MCP server, CLI, or skills, and measure whether agents like Claude Code, Codex, Cursor, Pi, or Hermes can actually use it.

Some teams use Oqoqo to build private benchmarks: real customer workflows that compare how agents perform with their product versus alternatives. Others use it for regression testing before they change an agent-facing surface, such as an MCP server, CLI, or SDK. And individual devs use it to compare models and harnesses to select the best cost-per-outcome for domain-specific tasks. In all cases, Oqoqo runs the same tasks across agents, models, and treatments, then reports pass rates, token usage, step counts, and the full trajectory, so teams can see not just whether an agent failed, but exactly where it got stuck.

Oqoqo's thesis is that agents are becoming software’s primary user. Its own platform has a user shaped like that too, and it is not a person clicking a button. It is a CI job that asks for 12,000 computers and wants them now.

About GMI Cloud

GMI Cloud is an AI-native cloud infrastructure company powering the next generation of AI applications. The company provides high-performance GPU infrastructure, Model-as-a-Service, dedicated endpoints, and AI workload deployment solutions for developers and enterprises building production AI systems. GMI Cloud helps teams move from experimentation to production with scalable compute, flexible infrastructure, and an ecosystem built for modern AI builders. For more information, visit gmicloud.ai.