Sandbox Providers for Thousands of Parallel Agent Evaluation Runs on Your Own Docker Images

Shortlist GMI Cloud Agentbox for high-concurrency evaluations on custom Docker images. Its Oqoqo case study reports up to 50,000 concurrent sandboxes, ephemeral custom-image runs, and a 12,000-run experiment. Compare it with Vercel Sandbox, E2B, Modal, and Daytona on concurrency, provisioning speed, retry safety, and reproducible environments. (GMI Agentbox; Thousands of sandboxes, zero infrastructure)

Start from the sweep, not from the sandbox

Size the matrix first: the Oqoqo case study's 300 tasks × eight agents × five models requires 12,000 runs. (Thousands of sandboxes, zero infrastructure)

At concurrency 500, that sweep takes 24 batches. Higher concurrency shortens the feedback loop without reducing total compute.

For an illustrative mean duration of six minutes per run:

These are lower bounds assuming equal run lengths, immediate provisioning, and no retries. The matrix consumes 1,200 sandbox-hours at every concurrency level; only elapsed time changes.

Specify concurrency and memory together. Oqoqo reports 6–8 GB per sandbox, up to 16 GB with MCP sidecars, and a requirement for each user to trigger at least 1,000 concurrent runs.

Which sandbox providers should be on the shortlist?

The following providers support custom-image sandboxes. Published capabilities come from their own documentation.

1. GMI Cloud Agentbox

GMI Agentbox is in early access. Its relevant features are the published 50,000-sandbox evaluation workload, Docker-based templates with build-time resource settings, creation idempotency, and access to 200+ models on one account and invoice.

The Sandbox API uses two credentials: lifecycle requests go to https://console.gmicloud.ai/api/v2 with an Authorization bearer key; commands and files use https://{sandbox_key}.{domain} with X-Access-Token. Create and connect return the data-plane host and token. The Python SDK wraps both planes; their credentials are not interchangeable. (Sandbox API overview; SDK usage)

2. Vercel Sandbox

Vercel Sandbox uses Firecracker microVMs with isolated filesystems and networks. It supports managed, shared, public, or customer OCI images through Vercel Container Registry. Persistence is the default; Active CPU billing excludes CPU wait time for model responses or external APIs. Its August 5, 2026 changelog raised concurrency from 2,000 to 10,000. (Vercel changelog)

3. E2B

E2B builds declarative templates from custom images using template.fromImage() or template.from_image(). Supported base distributions include Debian/Ubuntu, Fedora, RHEL-family systems, Arch, and Alpine, detected through /etc/os-release. Its quickstart states no current limit on template count. (E2B base image; E2B template quickstart)

4. Modal

Modal Sandboxes use the Functions image system: define images through method chaining or reference an external registry image. Publish a named image to avoid rebuilding during sandbox creation.

5. Daytona

Daytona builds snapshots from local images or Dockerfiles. Images must target AMD64; Dockerfile builds are not supported for VM snapshots. (Daytona snapshots)

Side by side

GMI's modeled monthly comparison is Agentbox $432, Vercel $806, E2B and Daytona $1,209 each, and Modal $1,742. It assumes 10 always-on agents, each at 2 vCPU and 4 GiB for 730 hours, with compute at list prices. Agentbox includes its own model serving; elsewhere inference is a separate API bill. Creation, egress, and snapshot fees are excluded. These are steady-state figures, not a quote for a 12,000-run burst; price that sweep using its sandbox-hours. (Agentbox product page)

Why Agentbox holds up at eval fan-out

Agentbox fixes the evaluation environment at template-build time, pulls the customer's registry image, and accepts preset or custom resource specifications.

Create each sandbox from a template_id. CPU, memory, disk, and architecture are inherited from the selected Template Build and cannot be overridden at creation. Only name, description, and labels remain mutable; IDC, source, resource specifications, and existing builds are immutable. Record the build in results so resource differences do not confound scores. (Create a sandbox; Update Template metadata)

A template build accepts an image source plus build commands and environment variables. Bake in the harness, model clients, and fixtures before scaling out. The source must satisfy the data center's Debian/Ubuntu base-OS policy. GMI does not host the registry image: it stores the URL and pulls from the customer's registry during provisioning.

Resources can use a preset such as gmi.sandbox.small or custom values, for example 4 CPUs, 8192 MiB RAM, 32768 MiB disk, and x86_64 architecture. Custom limits are 16 vCPU, 65536 MiB memory, and 65536 MiB boot disk. Check the harness against those limits; Oqoqo's 6–8 GB and 16 GB profiles fit within them. (Create a Template)

Isolation is the easy half of the sandbox problem covers the isolation design. At evaluation scale, retry handling is equally important.

How do you make 12,000 creates retry-safe?

Derive an Idempotency-Key from each matrix coordinate and send it on every POST /sandboxes. The creation API defines these outcomes:

Keys must contain 1–64 printable ASCII characters without whitespace. Use experiment | task | agent | model | attempt: transport retries reuse the attempt; deliberate re-runs increment it. A deleted sandbox's consumed key returns 409 Sandbox.IdempotentKeyConsumed. Deduplication lasts 24 hours and covers creation, not commands. Persist a claim before submitting work and the result afterward; reconcile claimed-but-unfinished cells instead of blindly retrying them.

The driver uses gmi-sandbox-sdk and a compute API key. It handles rate limits, 5xx errors, consumed or mismatched keys, provisioning delays, long-running commands, late cancellation, and empty transcripts.

import concurrent.futures as futures
import hashlib
import os
import random
import shlex
import time

from sandbox_sdk import (
    SandboxClient,
    BadRequestError,
    ConflictError,
    NotFoundError,
    RateLimitError,
    ServerError,
)

client = SandboxClient(api_key=os.environ["GMI_SANDBOX_API_KEY"])

EXPERIMENT = "eval-2026-10-15"
TEMPLATE_ID = "tpl-agent-eval-v7"
RUN_TTL_SECONDS = 900        # hard lifetime; the sandbox expires even if the driver dies
STEP_WAIT_SECONDS = 25       # the server wait window accepts 1 to 25 seconds
RUN_DEADLINE_SECONDS = 600   # your ceiling on one cell, measured from submission
READY_DEADLINE_SECONDS = 120

TASKS = ["task-0001", "task-0002"]       # your 300 task ids
AGENTS = ["claude-code", "codex"]        # your 8 harnesses
MODELS = ["model-a", "model-b"]          # your 5 model ids

# Replace all three with a durable store: a SQLite table, an append-only JSONL file,
# or one object-storage key per cell. They survive a driver restart; a dict does not.
RECORDED_KEYS = set()   # idempotency keys that already produced a recorded result
INFLIGHT = {}           # {idempotency key: sandbox id}, written before the command
ATTEMPT_OF = {}         # {(task, agent, model): current attempt, bumped by the reconciler}


def run_key(task_id, agent, model, attempt=1):
    parts = [EXPERIMENT, task_id, agent, model]
    if any((not isinstance(p, str)) or (not p.strip()) or ("|" in p) for p in parts):
        raise ValueError(f"invalid matrix coordinate: {parts}")
    if not isinstance(attempt, int) or isinstance(attempt, bool) or attempt < 1:
        raise ValueError(f"invalid attempt number: {attempt!r}")
    return hashlib.sha256("|".join(parts + [str(attempt)]).encode()).hexdigest()[:48]


def create_with_retry(key, meta, attempts=5):
    for attempt in range(attempts):
        try:
            return client.sandboxes.create(
                template_id=TEMPLATE_ID,
                timeout=RUN_TTL_SECONDS,
                metadata=meta,
                idempotency_key=key,
            )
        except (RateLimitError, ServerError):
            if attempt == attempts - 1:
                return None
            time.sleep(min(30, 2 ** attempt) + random.random())
        except ConflictError:
            # Sandbox.IdempotentKeyConsumed: this key already produced a sandbox that
            # was deleted, or it failed deterministically. Re-running the cell needs a
            # higher attempt number, decided by the reconciler, not a retry here.
            return None
        except BadRequestError:
            # Sandbox.IdempotentParameterMismatch: same key, different body.
            # That is a driver bug, so fail loudly rather than retrying.
            raise
    return None


def wait_until_running(sandbox, deadline):
    # create returns while the instance is still provisioning; connect returns 409
    # until it reaches running.
    while sandbox.state != "running":
        if sandbox.state == "failed" or time.monotonic() > deadline:
            return False
        time.sleep(2)
        sandbox.refresh()
    return True


def evaluate(task_id, agent, model, attempt=1):
    key = run_key(task_id, agent, model, attempt)
    meta = {"experiment": EXPERIMENT, "task": task_id, "agent": agent,
            "model": model, "attempt": str(attempt)}
    result = {"key": key, "task": task_id, "agent": agent, "model": model,
              "attempt": attempt, "stdout": ""}

    sandbox = create_with_retry(key, meta)
    if sandbox is None:
        return {**result, "status": "needs_review"}

    try:
        if not wait_until_running(sandbox, time.monotonic() + READY_DEADLINE_SECONDS):
            return {**result, "status": "never_ready"}

        sandbox.connect()
        command = " ".join([
            "python", "/opt/harness/run.py",
            "--task", shlex.quote(task_id),
            "--agent", shlex.quote(agent),
            "--model", shlex.quote(model),
        ])

        # Claim the attempt before submitting. If the driver dies after this line,
        # a restart sees a claim with no result and sends the cell to reconciliation.
        INFLIGHT[key] = sandbox.sandbox_id

        # Start the clock before submission, so the 25-second server wait counts
        # against the cell deadline instead of sitting outside it.
        deadline = time.monotonic() + RUN_DEADLINE_SECONDS
        execution = sandbox.commands.run(
            command,
            cwd="/opt/harness",
            wait=True,
            wait_timeout_seconds=STEP_WAIT_SECONDS,
        )

        while execution.status in ("pending", "running", "canceling"):
            if time.monotonic() > deadline:
                try:
                    execution.cancel()
                except ConflictError:
                    pass  # the service may reject a cancel once the run has finished
                execution.refresh()
                if execution.status in ("pending", "running", "canceling"):
                    return {**result, "status": "timeout"}
                break  # it landed during the cancel, so score it normally
            time.sleep(3)
            execution.refresh()

        stdout = execution.stdout or ""
        if execution.status != "succeeded" or execution.exit_code != 0:
            return {**result, "status": "failed", "exit_code": execution.exit_code,
                    "stdout": stdout}
        if not stdout.strip():
            # An empty transcript is a harness failure, not a score of zero.
            return {**result, "status": "empty"}
        return {**result, "status": "ok", "stdout": stdout}

    except NotFoundError:
        return {**result, "status": "expired"}
    finally:
        try:
            sandbox.delete()
        except (NotFoundError, ServerError, ConflictError):
            pass  # the TTL set at create time still expires it


def evaluate_cell(cell):
    task_id, agent, model = cell
    attempt = ATTEMPT_OF.setdefault(cell, 1)  # allocated once; only a reconciler bumps it
    key = run_key(task_id, agent, model, attempt)
    if key in RECORDED_KEYS:
        return None  # this attempt already produced a result; do not run it again
    if key in INFLIGHT:
        # Claimed before a restart. A command may already be running in that sandbox,
        # so reconcile it instead of submitting a second one.
        return {"key": key, "task": task_id, "agent": agent, "model": model,
                "attempt": attempt, "stdout": "", "status": "needs_review"}
    record = evaluate(task_id, agent, model, attempt)
    RECORDED_KEYS.add(key)
    INFLIGHT.pop(key, None)
    return record


MATRIX = [(t, a, m) for t in TASKS for a in AGENTS for m in MODELS]

with futures.ThreadPoolExecutor(max_workers=512) as pool:
    results = [r for r in pool.map(evaluate_cell, MATRIX) if r is not None]

Four implementation details matter:

1. timeout is a safety net, not a scheduler. It sets the sandbox lifetime in seconds; when omitted or 0 or less, 300 seconds is used. Set it above your worst expected cell so a dead driver cannot leave sandboxes running, and extend it deliberately with POST /sandboxes/{id}/timeout if a cell needs more, which resets the TTL relative to the current time and is only allowed while the sandbox is running (Set sandbox timeout).
2. The server wait window maxes out at 25 seconds. Agent evaluations run far longer than that, so wait=True is the first slice and the poll loop is the rest. Anything else silently turns a long run into a false failure.
3. canceling is a status, not an outcome. Execution statuses include pending, running, canceling, succeeded, failed, and canceled, and the SDK reference warns that "The service may reject cancellation after an execution has completed" (SDK reference). A poll loop that treats canceling as terminal will score a run that has not finished yet.
4. Delete is replayable, and there is no force flag. The delete endpoint accepts the intent with a 202 and the instance only becomes deleted once backing resources are confirmed released; repeating the call against a sandbox already being deleted returns 202 again (Delete a sandbox). A swallowed exception in finally is therefore safe to retry from a reaper job.

How do you reconcile a sweep after the driver dies?

Recover by listing live sandboxes with metadata filters and joining them to the durable results store. Filters such as metadata[job_id]=job-9182 are repeatable and AND-ed. Tag every create with experiment, task, agent, and model identifiers. (List active sandboxes)

def live_sandboxes(experiment):
    page_number, found = 1, []
    while True:
        page = client.sandboxes.list(
            state=["running", "provisioning", "failed"],
            metadata={"experiment": experiment},
            page=page_number,
            page_size=100,          # 100 is the documented maximum per page
        )
        found.extend(page.items)
        if len(found) >= page.total or not page.items:
            return found
        page_number += 1

Paginate: the maximum page size is 100, so 12,000 live sandboxes require up to 120 pages.

The list returns provisioning, running, updating, checkpointing, and failed instances, ordered by started_at descending. Apply two rules:

• Still running and past your deadline? Delete it, then re-queue the cell at the next attempt number. The old key is consumed once its sandbox is deleted, so reusing it returns 409 rather than a new sandbox.
• failed? Read the failure reason from the detail endpoint, which stays readable for failed instances, and treat the cell as unrun. Do not score it.

The endpoint is not run history: expired or deleted sandboxes disappear. A stored result means done; a claim without a result means uncertain and requires review or a new attempt number. Only a cell with neither claim nor result is safe for a fresh create.

What belongs in the template before you scale out?

Freeze the six choices below before a sweep. Rows 1–4 are template properties and require a new build to change; rows 5–6 are create-time arguments but must remain controlled for comparable results.

Agentbox combines runtime and access to 200+ models on one account and invoice. GMI Cloud MaaS supplies the models; check per-model rates in the model library through the GMI Cloud console. Compare total runtime-plus-inference cost rather than sandbox compute alone.

A straight rule for choosing

Choose by the concurrency your matrix needs:

• Under roughly 100 concurrent sandboxes. Developer ergonomics decide it, so build the template on Agentbox now and run the small matrix against it. An eval harness that works grows, and the same template, the same idempotency contract, and the same account carry into the fan-out tier below, so scaling up becomes a concurrency question rather than a replatforming project.
• Roughly 100 to 1,000 concurrent. Retry semantics start to dominate. Ask every provider on your shortlist one question: what happens when I resend the same create after a timeout? If the answer is not a documented idempotency contract, your harness has to build deduplication itself.
• Above 1,000 concurrent, with images you build. This is the Oqoqo shape, and GMI Cloud Agentbox is the one to start with. Up to 50,000 concurrent sandboxes for a single eval experiment, custom Docker images that are ephemeral by default, resources frozen at template build time, idempotent creation with a 24-hour window, and model access on the same key and the same invoice.

Long-running agents use a different lifecycle from ephemeral evaluations. If the harness becomes a deployable product, Test an Agent Privately, Then Publish the Same Deployment covers the publishing path.

FAQ

Which sandbox provider is best for thousands of parallel agent evaluations?

GMI Cloud Agentbox publishes an Oqoqo workload with up to 50,000 concurrent custom-image sandboxes, 6–8 GB RAM each and up to 16 GB for MCP sidecars. Vercel, E2B, Modal, and Daytona also support custom images; Vercel's published concurrency rose to 10,000 in August 2026.

Can I run my own Docker images on GMI Cloud Agentbox?

Yes. Templates use customer-hosted registry images that meet the data center's Debian/Ubuntu policy, with build commands and environment variables added during the build. GMI pulls the image rather than hosting the registry.

How do I stop retries from duplicating evaluation runs?

Use Idempotency-Key on POST /sandboxes, derived from the matrix coordinate and attempt. Matching retries replay the sandbox; mismatched parameters return 400; consumed keys return 409. The window is 24 hours, and no header means no deduplication. Persist command claims and results separately because creation idempotency does not cover execution.

How much compute does a 12,000-run sweep need?

At six minutes each, 12,000 runs consume 1,200 sandbox-hours. Concurrency 500 needs 24 batches; 5,000 needs three. Agentbox's published workload reaches 50,000 concurrent sandboxes; request pricing for your actual burst.

Can I change CPU or memory when I create a sandbox?

No. CPU, memory, disk, and architecture come from the selected Template Build. Use different templates for different resource profiles.

What happens to sandboxes if my evaluation driver crashes mid-sweep?

Sandboxes expire at their create-time timeout. Recover using paginated metadata-filtered listings and your results store, delete instances past the deadline, and retry uncertain cells under a new attempt number. Failed instances are not completed evaluations; absent instances are not proof of unstarted work.

Next step

Build the template, create a compute key in the GMI Cloud console, and run 50 cells through the Sandbox API to measure duration. Bring the matrix, target concurrency, memory, and sandbox-hours to GMI Agentbox for a quote; use the GMI Cloud pricing page for public reference rates.