Sandbox

A Sandbox provider is the infrastructure that creates and manages isolated runtime environments. NiceEval wraps them all behind the same Sandbox interface, so an Adapter does not need to know whether the current provider is local Docker, a Vercel micro-VM, E2B, or a third-party cloud service.

The `Sandbox` interface

Common Adapter operations include:

Method	Use
`runCommand(cmd, args)`	run a command
`runCommandOrThrow(cmd, args)`	run a command and throw on non-zero exit
`runShell(script)`	run a shell script
`readText(path)` / `readBytes(path)`	read text or exact bytes from the Sandbox
`writeText(path, content)` / `writeBytes(path, content)`	write in-memory content into the Sandbox
`pathExists(path)`	check whether a file or directory exists
`uploadFile(source, target)` / `uploadDirectory(source, target?)`	transfer host files into the Sandbox
`downloadFile(source, target)` / `downloadDirectory(source, target)`	transfer Sandbox files to the host
`workdir`	the provider’s real working directory; omitted `cwd` / `targetDir` resolve here
`runCommand(..., { cwd })`	temporarily switch directory for one command; relative paths resolve from `workdir`
`stop()`	destroy the environment

Choose a provider

Set the sandbox field in the experiment:

// experiments/local.ts
import { defineExperiment } from "niceeval";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: myCodingAgent,
  model: "claude-sonnet-4-6",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }), // or vercelSandbox({ snapshotId }) / e2bSandbox({ template })
});

If neither the Eval nor Experiment declares a template-bearing layer, link planning fails before creating a Sandbox instead of guessing a provider. The SDKs for the three built-in providers do not install alongside NiceEval. Install whichever one you use, so your project does not carry dependencies (and their native build scripts) it does not need:

Provider	Install command
`dockerSandbox({ source: { type: "image", image: "node:24-slim" } })`	`pnpm add dockerode @types/dockerode`
`vercelSandbox({ snapshotId })`	`pnpm add @vercel/sandbox`
`e2bSandbox({ template })`	`pnpm add e2b`
`localSandbox()`	no extra install

A missing SDK does not fail silently: NiceEval errors the moment it creates the sandbox and prints the install command above, for example Docker sandbox requires 'dockerode'. Install it with: pnpm add dockerode @types/dockerode.

Docker Compose: `workspaceService` names the main Sandbox

When a task needs more than one container — an app container plus a database, say — declare the whole Compose environment with dockerComposeSandbox:

import { dockerComposeSandbox } from "niceeval/sandbox";

sandbox: dockerComposeSandbox({
  file: new URL("docker-compose.yaml", import.meta.url),
  workspaceService: "client", // the main Sandbox
})

workspaceService names which service in the Compose file is the main Sandbox. The agent, t.sandbox commands and file operations, workdir, and the change diff all land on that one container. Every other service in the Compose file — a database, a mock server, whatever — is an accompanying resource that never goes through the Sandbox interface: you cannot runCommand against it, upload or read files on it, or see its diff. To reach an accompanying service, rely on the task’s own networking — for example, reaching db:5432 from the client container over Compose’s built-in DNS.

Lifecycle

The SandboxLayer returned by dockerSandbox({ source: { type: "image", image } }) / vercelSandbox({ snapshotId }) / e2bSandbox({ template }) has chainable .prepare(command), .setup(fn), and .teardown(fn) methods. They handle the parts of the environment you only know at runtime — writing a small config per experiment, checking that a prebuilt tool is available, installing a hook, or loading and saving state between attempts. When a shared helper needs to explicitly annotate a callback’s type, import it from the public entry point — do not reverse-engineer it from some provider’s spec:

import type { SandboxHook, SandboxHookContext } from "niceeval/sandbox";

Do not reinstall stable, bulky dependencies here. System packages, Agent CLIs, compiled binaries, and large model caches belong in a Docker image, a Vercel Sandbox snapshot, or an E2B template. Every attempt then starts from a prebuilt environment, and .setup() does only a thin layer of dynamic configuration and fail-fast checks.

export default defineExperiment({
  agent: codexAgent({ mcpServers: [mempalMcp] }),
  sandbox: e2bSandbox({ template: "fasteval-agents-mempal" }) // already contains the binaries and model cache
    .setup(mempalSetup("codex"))        // preflight, write hooks, load state
    .teardown(mempalTeardown("codex")), // save state back
  maxConcurrency: 1,                    // no concurrency between load and save — declare serial execution
});

The rules:

Signature: a hook function is (sandbox, ctx) and returns nothing. The ctx here is a narrow Sandbox Hook Context — only the experiment identity, the cancellation signal, and the feedback methods. It carries no Agent session and no telemetry.
Immutable: every .setup() / .teardown() returns a new spec and leaves the original untouched, so you can keep chaining.
Multiple hooks: multiple .setup() hooks run in the order they were appended; multiple .teardown() hooks run in the reverse of that order (LIFO). If a .setup() chain throws partway through, the remaining .setup() hooks do not run, but the full .teardown() chain still runs to completion — a half-initialized Sandbox still needs cleanup.
Timing: setup hooks run first — after the Sandbox is created and before the git baseline. The files they write go into the baseline and are not counted in the agent’s diff. teardown hooks run last — after the Adapter’s teardown and before the Sandbox is destroyed. That is exactly the right moment to save state to an external store.
Trigger rule: teardown runs if and only if the same Sandbox’s setup point was reached — a setup throw does not exempt it, so cleanup code must defend against handles that may not have been assigned; if setup never ran (the Sandbox failed to be created), teardown is skipped too.
Failure semantics: if a setup hook throws, the attempt is recorded as errored (an environment problem, not the Agent getting the task wrong). A teardown hook can report a diagnostic and, by default, does not change the verdict you already have. If some cleanup action is a precondition for the result to be valid, throw — the runner records it explicitly as a fatal error.
Agents without a sandbox: an Agent built with defineAgent has no sandbox, the sandbox field does not apply to it, and the hooks never run.

Pass setup’s output to teardown

setup returns nothing. When teardown needs a handle that setup built — an already-open connection, a temp file path — it cannot be stored in an ordinary module variable: the same spec is reused by the same module across multiple concurrent attempts, and a module variable gets overwritten by whichever attempt runs later. Key it off the sandbox instance instead: each attempt has its own sandbox, which is a natural per-attempt key:

import type { Sandbox } from "niceeval/sandbox";

// Concurrent attempts each get their own sandbox: key the handle by sandbox, not by a plain module variable
const forwarders = new WeakMap<Sandbox, { stop(): Promise<void> }>();

const spec = e2bSandbox({ template: "niceeval-agents" })
  .setup(async (sandbox, ctx) => {
    forwarders.set(sandbox, await startLogForwarder(sandbox, { signal: ctx.signal }));
  })
  .teardown(async (sandbox) => {
    await forwarders.get(sandbox)?.stop(); // also runs when setup threw: skip if there's nothing to get
  });

Cleanup that only loads and saves state does not need a setup → teardown handle: the state key is an external KV keyed on ctx.experimentId, and ordinary code is enough — see the next paragraph. Inside a hook you can use ctx.experimentId (the experiment id derived from the path) as the key for isolating state — for example, so different experiments each keep their own cross-attempt cache. Loading and saving cross-attempt state is ordinary code you write in the hook yourself; NiceEval does not provide state storage. To make sure attempts of the same experiment do not read and write the same state concurrently, declare maxConcurrency: 1 on the experiment.

Report progress and problems from hooks

Long installs, warm-ups, and state restores can call ctx.progress(...). It only updates the short-lived status of the current attempt; it does not write every step into the result. For problems you still want to see after the run finishes, use ctx.diagnostic(...):

const sandbox = e2bSandbox({ template: "niceeval-agents" })
  .setup(async (sandbox, ctx) => {
    ctx.progress({ message: "Checking the project helper", current: 1, total: 2 });
    await ensureProjectHelper(sandbox);

    ctx.progress({ message: "Warming up the project build cache", current: 2, total: 2 });
    try {
      await warmProjectBuildCache(sandbox);
    } catch (error) {
      ctx.diagnostic({
        code: "project-build-cache-degraded",
        level: "warning",
        message: "Build-cache warm-up failed; continuing without the warm cache",
        data: { reason: String(error) },
        dedupeKey: "project-build-cache-degraded",
      });
    }
  });

progress, diagnostic, and fact are mutually exclusive feedback channels. Use progress for short-lived status, diagnostic only for a real exception, degradation, or problem that needs attention, and ctx.fact(...) for neutral runtime facts. A fact uses a reverse-domain name and its complete JSON document is limited to 65,536 UTF-8 bytes. Normal capacity, cache size, version, and hit status are not warnings by default; emit a diagnostic only when a clear, explainable risk condition is met. If the environment cannot continue, throw.

Neither progress nor diagnostic can pick a global phase, a color, or an output stream. The Runner knows the current callback belongs to sandbox.setup and shows the phase as Sandbox setup in the human terminal. A diagnostic is written to an Attempt-owned diagnostics channel and can later be reviewed with niceeval show --run <runId> --page attempt-<attemptId>. It does not change the Verdict by itself. Throw when the Sandbox cannot continue. Each of the three setups has its own job: the Adapter’s setup handles connecting to the agent under test — for example, installing the CLI and writing auth config; the code at the top of test(t) in an eval prepares the task’s starting files; the Sandbox’s .setup() handles the current experiment’s extra environment setup. MCP servers, skills, and the model — the configuration of the agent under test — still only come in through the Adapter factory arguments.

Prebuilt environments and runtime checkpoints

NiceEval references prebuilt environments through a typed spec, but does not offer a fake universal build command:

dockerSandbox({ source: { type: "image", image: "my-evals:node24" } })
vercelSandbox({ snapshotId: "snap_abc123" })
e2bSandbox({ template: "my-evals" })

Docker images, Vercel Sandbox snapshots, and E2B templates differ in credentials, build context, publishing, and expiry. Your project should maintain build scripts with the provider’s official tooling and put the final ID or name into the experiment. Deciding what to prebuild is simple: if every attempt downloads or installs the same content, and that content is stable, expensive, or large, move it into the prebuilt environment.

Build on the official baselines to speed things up

Content that is stable, large, and identical for every attempt — system packages, Agent CLIs, compiled binaries, large model caches — should be baked into the provider’s publishable artifact before you run evals, so every attempt starts from a prebuilt environment and skips the runtime install. All three built-in providers can derive from an official baseline, so you never have to install the agent from a blank environment. Their build tooling, credentials, and publishing semantics differ, though, so NiceEval only unifies how you consume the artifact ID (image / snapshotId / template) and does not fabricate a cross-provider build DSL. Adapters and Sandboxes do not guess each other’s configuration: the Adapter checks for the CLI it needs, and the Sandbox spec picks the provider and the artifact. Claude Code and Codex fall back to a runtime install when the CLI is missing, so baking them in is purely a speed-up. Bub also verifies an install fingerprint over its version, OTel plugin, and Python plugin set, so command -v bub alone is not enough to skip the install — it requires a prebuilt environment. For all three providers, the build runs once when environment dependencies change, and the artifact takes a new versioned name. Do not push it into every attempt’s .setup().

E2B: derive from official and public templates

E2B already provides a claude template for Claude Code and a codex template for Codex. NiceEval ships a thin E2B-specific wrapper that lets you keep chaining the native E2B API from those two official starting points. E2B has no official Bub template yet, so the Bub branch uses a NiceEval install recipe with pinned versions (a Bub release plus a same-generation OTel plugin). NiceEval also publishes three public templates that any E2B team can reference. NiceEval itself maintains the full namespace and the verified release tag; downstream code just takes the complete reference:

import {
  NICEEVAL_CLAUDE_CODE_E2B_TEMPLATE,
  NICEEVAL_CODEX_E2B_TEMPLATE,
  NICEEVAL_BUB_E2B_TEMPLATE,
} from "niceeval/sandbox/e2b-template";

e2bSandbox({ template: NICEEVAL_CLAUDE_CODE_E2B_TEMPLATE })
e2bSandbox({ template: NICEEVAL_CODEX_E2B_TEMPLATE })
e2bSandbox({ template: NICEEVAL_BUB_E2B_TEMPLATE })

These baselines have been verified by actually booting them: Claude Code 2.1.207, Codex 0.144.1, Bub 0.4.0 (its install fingerprint moves with the version and OTel plugin pins). A template’s version follows the agent it ships (0.144.1-r1: agent version plus NiceEval’s recipe revision); the three agents are released independently, and NiceEval’s own library version never appears in the tag. So use the constants directly instead of copying these strings or tracking versions — including when a derived template needs to record which base it came from.

scripts/build-e2b-template.ts

import { Template } from "e2b";
import { e2bCodingAgentTemplate } from "niceeval/sandbox/e2b-template";

const template = e2bCodingAgentTemplate("codex")
  .aptInstall(["ripgrep", "jq"])
  .runCmd("corepack enable && pnpm --version")
  .copy("fixtures/toolchain.lock", "/opt/evals/toolchain.lock");

await Template.build(template, "acme-codex-evals:0.144.1-r1", {
  cpuCount: 2,
  memoryMB: 4096,
});

e2b auth login
pnpm tsx scripts/build-e2b-template.ts

Then reference only the build result in the experiment:

import { e2bSandbox } from "niceeval/sandbox";

sandbox: e2bSandbox({ template: "acme-codex-evals:0.144.1-r1" })

You can also derive directly from a NiceEval public template and pay only for the build cost of your project’s own dependencies:

import { NICEEVAL_CODEX_E2B_TEMPLATE } from "niceeval/sandbox/e2b-template";

const template = Template()
  .fromTemplate(NICEEVAL_CODEX_E2B_TEMPLATE)
  .aptInstall(["ripgrep", "jq"])
  .runCmd("corepack enable");

e2bCodingAgentTemplate("claude-code" | "codex" | "bub") returns a native TemplateBuilder, not a private NiceEval build DSL. You can keep using .aptInstall(), .runCmd(), .copy(), and the rest of E2B’s capabilities. Building your own alias freezes the official starting point and your project’s dependencies into a single reproducible artifact; rebuild and pick a new versioned alias when dependencies change. If the Bub Adapter is configured with pythonPlugins, pass the same set of packages to the factory when you build the template. Only then does the plugin set enter the compatibility fingerprint and actually hit the preinstalled environment:

e2bCodingAgentTemplate("bub", {
  bubPythonPackages: ["bub-plugin-memory==1.3.0"],
})

Docker: use NiceEval-maintained images, or derive from the official node base image

To run NiceEval’s built-in claude-code, codex, or bub Adapters directly, use the matching public image: niceeval/claude-code, niceeval/codex, or niceeval/bub. Each image contains only its own Agent CLI and publishes a manifest for linux/amd64 and linux/arm64. Its tag matches the corresponding E2B public template — the version position is the version of the agent inside the image. Stable CI should use the named constant or a digest, not the moving latest:

import { NICEEVAL_CODEX_DOCKER_IMAGE, dockerSandbox } from "niceeval/sandbox";

sandbox: dockerSandbox({ source: { type: "image", image: NICEEVAL_CODEX_DOCKER_IMAGE } })

This image is a public image maintained by NiceEval, not a Docker library/* Official Image. A new tag is published when the agent version or the build recipe changes, independently of NiceEval’s own release cadence. If you only need one agent, or you also need to add project-specific dependencies, write a Dockerfile that derives from Docker’s official baseline, node:24-slim:

Dockerfile

FROM node:24-slim
# the slim image ships without ca-certificates / git, which both the agent and npm need
RUN apt-get update \
  && apt-get install -y --no-install-recommends ca-certificates git \
  && rm -rf /var/lib/apt/lists/*
# npm installs globals into /usr/local/bin, which is exactly on the PATH the Sandbox injects
RUN npm install -g @openai/codex@0.144.1

docker build -t acme-codex-evals:0.144.1-r1 .

Then reference only the build result in the experiment:

import { dockerSandbox } from "niceeval/sandbox";

sandbox: dockerSandbox({ source: { type: "image", image: "acme-codex-evals:0.144.1-r1" } })

The Docker Sandbox follows whatever execution identity the image itself declares: the Dockerfile above never sets USER, so the container runs commands as root by default, and /usr/local/bin is already on root’s PATH, so global binaries installed with npm install -g are visible out of the box. When you need a non-root identity (for example, Claude Code refuses --dangerously-skip-permissions under root), add USER node to the Dockerfile (the node:24-slim image already has that user), or override it explicitly with dockerSandbox({ source: { type: "image", image }, user: "node" }). If an agent installs somewhere else (for example into ~/.local/bin), remember to put that directory on the PATH. dockerSandbox requires an explicit image; stable CI should reference an immutable tag.

Vercel: snapshot the official runtime

Vercel has no E2B-style template registry and no Dockerfile; a Sandbox snapshot is taken from a microVM that is already running. Use the Vercel SDK to start a Sandbox from the official runtime (node24), install the Agent CLI, call .snapshot() to get a snap_..., then hand it to vercelSandbox({ snapshotId }):

scripts/build-vercel-snapshot.ts

import { Sandbox } from "@vercel/sandbox";

const sandbox = await Sandbox.create({ runtime: "node24" }); // start a microVM from the official runtime
await sandbox.runCommand({
  cmd: "npm",
  args: ["install", "-g", "@openai/codex@0.144.1"],
  sudo: true, // install globally into /usr/local/bin, which is on the Sandbox PATH
});
const { snapshotId } = await sandbox.snapshot();
console.log(snapshotId); // snap_...
await sandbox.stop();

pnpm tsx scripts/build-vercel-snapshot.ts

Then reference the printed ID in the experiment:

import { vercelSandbox } from "niceeval/sandbox";

sandbox: vercelSandbox({ snapshotId: "snap_xxx" })

Vercel snapshots do not support E2B-style public publishing: a snapshot ID is governed by the permissions of the team/project that created it. Members of the same project can reuse it, but an outside user has to take their own snapshot in their own Vercel project. The never-expiring snapshot the NiceEval project currently keeps verified is snap_7sIjfs71xfmVly0WEUTGhTBoMGeL, but it is not a cross-account public ID.

Runtime checkpoints

createCheckpoint() / restoreCheckpoint() are a different thing. They pack the Linux paths you name into a Buffer, so you can restore a slice of the file system into an already-created Sandbox:

import { createCheckpoint, restoreCheckpoint } from "niceeval/sandbox";

const data = await createCheckpoint(sandbox, ["/home/user/.cache/tool"]);
await restoreCheckpoint(nextSandbox, data);

This suits runtime caches. It does not create a publishable image/template/snapshot, and it does not manage sharing, versions, or expiry. A failed archive or restore throws.

Transient error retries

When a built-in provider creates a Sandbox, transient failures — rate limiting, fetch failed, connection resets, 5xx, temporary network unreachability — automatically retry with exponential backoff. Configuration errors, such as a missing template or missing credentials, fail on the first try. Once retries are exhausted, the attempt is recorded as errored. defineSandbox’s custom provider create is your own function; NiceEval does not retry it for you. readText, readBytes, writeText, writeBytes, pathExists, and the upload/download operations automatically make a bounded number of retries on transient transport errors: 429, 5xx, fetch failed, connection resets. Missing files, permission errors, cancellation, and a terminated Sandbox are not retried. runCommand and runShell do not retry automatically. A command may already have produced side effects, so retry it explicitly in a hook or an eval only when you can confirm it is safe to repeat.

Docker

Docker is good for local development and standard CI. It is simple, controllable, and has no cloud dependency. The tradeoff is limited machine resources, plus slower cold starts and dependency installs.

Vercel Sandbox

Vercel Sandbox is good when you want cloud isolation, more resources, or a more stable environment. It requires the right token or OIDC setup.

Local directory

localSandbox() runs the agent directly inside a Git repository on your own machine — no Docker, no cloud credentials:

import { defineExperiment } from "niceeval";
import { localSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: myCodingAgent,
  sandbox: localSandbox(), // defaults to the current Git repo root as the working directory
});

To evaluate a different directory, pass localSandbox({ dir: "/path/to/repo" }). NiceEval only observes — it never reverts. Whatever the agent changes lands for real on your working tree; NiceEval captures the diff and scores it with its own private Git ledger, which never touches your .git, your staged changes, or your uncommitted work, and never runs git reset when the run finishes. Whether to keep or discard the agent’s changes is entirely up to you. Three things to keep in mind before you use it:

The agent runs commands as you, on your machine. Only use it when you trust the task and the prompt; use Docker or a cloud provider when you need isolation.
The local directory runs only one attempt at a time (forced serial); --max-concurrency cannot raise it.
When you run several evals back to back, one eval’s changes stay on the working tree and become the starting point for the next. Use a container provider when you need a clean starting point for every eval.

{ user: "..." } and --keep-sandbox are unavailable for the local directory and fail immediately: NiceEval does not escalate privileges or switch identity on your machine, and there is nothing to “retain” — the workspace already sits exactly where it was.

Custom provider

Use defineSandbox to plug in another service. The feedback given to create is already bound to the sandbox.create phase, so it can report on allocating an instance, pulling an image, or restoring a Sandbox snapshot:

import { defineSandbox } from "niceeval/sandbox";

export default defineSandbox({
  name: "modal",
  recommendedConcurrency: 8,
  async create({ timeout, runtime, feedback }) {
    feedback.progress({ message: "Allocating Modal Sandbox" });
    const instance = await allocateModal({ timeout, runtime });

    if (instance.usedFallbackRegion) {
      feedback.diagnostic({
        code: "modal-fallback-region",
        level: "warning",
        message: `Primary region unavailable, using ${instance.region}`,
        data: { region: instance.region },
      });
    }

    return new ModalSandbox(instance);
  },
});

The return value just has to implement the Sandbox interface. Do not write the provider SDK’s raw logs straight to the host process’s stdout / stderr. Short-lived status goes through feedback.progress, problems worth keeping go through feedback.diagnostic, and if the environment cannot be created, throw. That keeps the human dashboard from being torn apart by logs and keeps CI output single and ordered.

Permissions and execution identity

Providers differ in what they allow around execution identity, networking, the file system, and process lifecycle. When writing Fixtures, avoid depending on the host machine’s environment; put dependencies in package.json or in Fixture setup.

Performance advice

Bake stable, heavy dependencies into an image/template/snapshot instead of reinstalling them in every attempt’s .setup().
Keep Fixture dependencies small.
Use small, explicit caches or preflight checks for dynamic content.
Tune maxConcurrency (the experiment field or --max-concurrency) so local Docker does not run out of resources.
Split slow tests into required gates and optional soft checks.

Warm pools and reuse belong to the runner / scheduler layer. See Runner.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

The `Sandbox` interface

Choose a provider

Docker Compose: `workspaceService` names the main Sandbox

Lifecycle

Pass setup’s output to teardown

Report progress and problems from hooks

Prebuilt environments and runtime checkpoints

Build on the official baselines to speed things up

E2B: derive from official and public templates

Docker: use NiceEval-maintained images, or derive from the official node base image

Vercel: snapshot the official runtime

Runtime checkpoints

Transient error retries

Docker

Vercel Sandbox

Local directory

Custom provider

Permissions and execution identity

Performance advice

​The Sandbox interface

​Choose a provider

​Docker Compose: workspaceService names the main Sandbox

​Lifecycle

​Pass setup’s output to teardown

​Report progress and problems from hooks

​Prebuilt environments and runtime checkpoints

​Build on the official baselines to speed things up

​E2B: derive from official and public templates

​Docker: use NiceEval-maintained images, or derive from the official node base image

​Vercel: snapshot the official runtime

​Runtime checkpoints

​Transient error retries

​Docker

​Vercel Sandbox

​Local directory

​Custom provider

​Permissions and execution identity

​Performance advice

The `Sandbox` interface

Choose a provider

Docker Compose: `workspaceService` names the main Sandbox

Lifecycle

Pass setup’s output to teardown

Report progress and problems from hooks

Prebuilt environments and runtime checkpoints

Build on the official baselines to speed things up

E2B: derive from official and public templates

Docker: use NiceEval-maintained images, or derive from the official node base image

Vercel: snapshot the official runtime

Runtime checkpoints

Transient error retries

Docker

Vercel Sandbox

Local directory

Custom provider

Permissions and execution identity

Performance advice