Sandbox Agent: Evaluate Claude Code, Codex, and bub

You wrote a Skill for Claude Code and want to know whether the Agent really fixes bugs more accurately with it installed. This kind of eval cannot just look at a reply: the Agent reads code, edits files, and runs commands, and the result shows up in the repository. Sandbox Agents are built for this. NiceEval first creates an isolated Sandbox, puts your project in it, then starts the Coding Agent’s CLI inside and hands it the task. The Agent can freely edit files and run commands without touching your machine. Afterwards, NiceEval collects the conversation record, file changes, and test results for the eval to score.

Built-in Sandbox Agents

These three Coding Agents work out of the box; you only need the matching API key:

claude-code

Runs the Anthropic Claude Code CLI; requires ANTHROPIC_API_KEY.

codex

Runs the OpenAI Codex CLI; requires CODEX_API_KEY.

bub

Runs the bub coding agent; requires BUB_API_KEY and BUB_API_BASE.

Run a built-in Agent

First get a built-in Agent running in local Docker. Set sandbox on the Experiment (or the eval) to tell NiceEval where to create the Sandbox:

// experiments/local.ts
import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: claudeCodeAgent(),
  model: "claude-sonnet-4-6",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});

export ANTHROPIC_API_KEY=sk-ant-...
pnpm exec niceeval exp local fixtures/button

pnpm exec niceeval exp local fixtures/button --attempts 10

The second command runs the same eval 10 times. After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view.

The Sandbox can only be written in the Experiment or the eval. There is no matching CLI flag and no project-level default Provider. When neither place specifies a Sandbox image or template, NiceEval reports an error before creating any Sandbox.

When running Coding Agents in the cloud, you can use the E2B public templates NiceEval publishes directly, saving the time of installing the CLI for every Attempt:

import { codexAgent } from "niceeval/adapter";
import { e2bSandbox } from "niceeval/sandbox";
import { NICEEVAL_CODEX_E2B_TEMPLATE } from "niceeval/sandbox/e2b-template";

export default defineExperiment({
  agent: codexAgent(),
  model: "gpt-5.4",
  sandbox: e2bSandbox({ template: NICEEVAL_CODEX_E2B_TEMPLATE }),
});

Claude Code uses NICEEVAL_CLAUDE_CODE_E2B_TEMPLATE, and Bub uses NICEEVAL_BUB_E2B_TEMPLATE. The constants’ values are complete template references with pinned versions, and the version updates with the Agent version in the template, so you do not need to track it yourself. To add system packages, binaries, or model caches to a template, see Prebuilt Sandbox Environments · Continue building from the official baseline. The built-in Agents exported from niceeval/adapter are factory functions. To configure auth, proxies, MCP, or GitHub Skills, write them in the factory parameters. The model still goes in the Experiment’s model field, and the Sandbox Provider still goes in the sandbox field:

import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: claudeCodeAgent({
    apiKey: process.env.ANTHROPIC_API_KEY,
    baseUrl: process.env.ANTHROPIC_BASE_URL,
    maxTurns: 8,
    mcpServers: [
      {
        name: "browser",
        command: "npx",
        args: ["-y", "@anthropic/mcp-browser"],
      },
    ],
    skills: [{ kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2" }],
  }),
  model: "claude-sonnet-4-6",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});

Agent environment variables

Before running, confirm these variables are set. Without them, the Agent cannot call the model in the Sandbox.

Agent	Required variables
`claude-code`	`ANTHROPIC_API_KEY`
`codex`	`CODEX_API_KEY`
`bub`	`BUB_API_KEY`, `BUB_API_BASE`

These are each Agent’s official variable names. When CI stores only one set of shared credentials, pass them explicitly in the Experiment with apiKey, baseUrl. This does not change the variable names the Adapter reads by default.

What one Attempt does, in order

When debugging, first know where each step happens. An Attempt runs in this order:

Create the Sandbox
  → Run the Sandbox's setup lifecycle
  → Run the eval's setup lifecycle and prepare the Fixture
  → Check the Agent CLI version; install and recheck when needed
  → Run the Adapter's setup, writing auth and run configuration
  → Run test(t), writing starting files
  → The Adapter calls the Agent, recording file changes in the meantime
  → Run verification commands, finish scoring and the verdict
  → Run the Adapter's and Sandbox's teardown lifecycle
  → Stop the Sandbox

Starting files and verification commands are both written in test(t). While the Agent runs, it can only see the files you have already written into the Sandbox. File assertions such as t.sandbox.fileChanged() only look at files the Agent changed during t.send(). NiceEval records the workspace state before and after each t.send(), and the changes in between are attributed to the Agent. Starting files you uploaded and verification files written after t.send() do not count, so fileChanged("src/app.ts") passes only when the Agent really changed that file.

Make file changes recordable

When a file assertion fails unexpectedly, or niceeval show @<locator> --diff shows no changes, go through the points below first. NiceEval creates a separate private Git repository in the Sandbox and records the workspace state before and after each t.send(). This requires:

git and a POSIX shell in the Sandbox. The default images of the built-in Providers have them; confirm it yourself for custom images. Whether the project under test is itself a Git repository does not matter: the private repository lives outside the working directory, does not touch your .git, and the Agent cannot see it.
Files written in the working directory. Only files under sandbox.workdir are recorded. Omit targetDir / cwd when writing files; when you need an absolute path, read sandbox.workdir instead of hard-coding /workspace. Files written outside the working directory are invisible to the Agent and cannot be recorded.
No second Git repository in the working directory. Put the project under test directly at the root of the working directory. When the working directory contains a submodule or another cloned repository, the Attempt reports an execution error listing the paths, because file changes in that repository would be silently missed. For a repository that really does not take part in scoring, exclude it entirely with diff.ignore.
Changes happen during t.send(). Files written after the last t.send() count as your verification files, not as the Agent’s changes.

.git, node_modules, __pycache__, Python virtual environments, common build outputs, and package manager caches are not recorded by default; otherwise a single npm install could produce tens of thousands of files. To adjust this, write the eval’s diff field:

export default defineEval({
  diff: {
    ignore: ["fixtures/**"],               // Agent changes here do not count as its changes
    include: ["node_modules/some-pkg/**"],  // Patched dependencies should count
  },
  async test(t) { /* ... */ },
});

Rules are written like gitignore, counted from the working directory root: entries without / match same-named items at any depth, entries with / match from the working directory root, and a trailing / means a directory. The project’s own .gitignore takes no part, so files the project ignores are still recorded. The exclusion list is fixed at the first recording, and later changes the Agent makes to .gitignore do not affect it. Before and after each t.send(), at most 100,000 paths and 256 MiB of deduplicated text are scanned. At most 10,000 changes are kept; beyond that, only a fixed leading portion is kept and marked partial (reason collection-cap-reached). Only exceeding the scan limit reports an execution error; it never pretends “no files changed” and lets a file assertion pass wrongly. When the workspace is very large, narrow the scope with diff.include or diff.ignore. Experiments, eval groups, evals, and Sandbox Agents can all add command-only .before() preparation steps. When the Agent writes its own .env, put that step in the Agent’s sandbox field with a larger changeFrequency, so fixed tools and Fixtures come first. For how to write them and the caching rules, see Sandbox · Prepare the Sandbox in order.

Custom Sandbox Agents

To evaluate a Coding Agent beyond the built-in ones, write your own with defineSandboxAgent. The Agent below runs the my-agent CLI in the Sandbox, reads back the conversation record it writes, and translates it into events:

import {
  completeEvidenceCoverage,
  createNpmCliInstaller,
  defineSandboxAgent,
  agentBin,
} from "niceeval/adapter";

const { ensure, installer } = createNpmCliInstaller({
  identity: { agent: "my-agent", version: "1.0.0", revision: "1" },
  packageName: "my-agent",
  bin: "my-agent",
});

export default defineSandboxAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  ensure,
  installers: [installer],
  async send(input, ctx) {
    ctx.progress({ message: "Running my-agent CLI" });
    await ctx.sandbox.runCommand("sh", [
      "-lc",
      `exec ${agentBin("my-agent")} run "$1" --json-out agent-events.json`,
      "my-agent",
      input.text,
    ]);
    const transcript = await ctx.sandbox.readText("agent-events.json");
    if (transcript.trim() === "") {
      ctx.diagnostic({
        code: "empty-transcript",
        level: "warning",
        message: "my-agent wrote no transcript; tool assertions may lack evidence",
      });
    }
    return {
      status: "completed",
      events: parseTranscript(transcript),
    };
  },
});

createNpmCliInstaller downloads the npm package on the host, transfers it into the Sandbox, then installs it with npm. So the Sandbox needs Node.js and npm; only a platformPackage that bundles its own runtime does not. For the full steps, see Install the CLI for a Custom Sandbox Agent. setup, each send, and teardown each have their own feedback methods; use them to tell you where the run is and what went wrong:

ctx.progress({ message, current?, total? }) shows what is happening now, suited to steps such as installing the CLI, running a turn, or reading the conversation record. Do not call it for every token or every JSONL line.
ctx.diagnostic({ code, level, message, data?, dedupeKey? }) leaves a problem to review afterwards, such as incomplete output format, a missing conversation record, or a cleanup failure. It is shown in the terminal and saved with this Attempt’s results.
When it cannot continue, throw directly. NiceEval saves the failing phase, error code, error message, cause, and stack, and judges this Attempt errored.

Do not call console.log/error or write to process.stdout/stderr directly in the Adapter. They scramble the terminal’s run panel and also put CI logs out of order. When a Sandbox or Adapter error occurs during the run, the terminal gives this Attempt’s locator:

✗ @5TB8167MXJ30SYZCNAVRHPQ4D2 fixtures/button [local] errored · agent setup
    agent-install-failed: npm install my-agent exited with code 1
    Inspect: niceeval show @5TB8167MXJ30SYZCNAVRHPQ4D2

pnpm exec niceeval show @5TB8167MXJ30SYZCNAVRHPQ4D2
pnpm exec niceeval show @5TB8167MXJ30SYZCNAVRHPQ4D2 --timing

The first shows the full error, diagnostics, and completed phases. The second shows each phase’s duration, so you can see which step errored or timed out. When a script or CI needs the same information, use pnpm --silent exec niceeval query discover to find the Attempt trace request, then run it with query run. When there are many steps, once the phase tree exceeds 80 detail nodes, only failed, slow, and the first and last few items are kept, with a note on how many were omitted. To check every item, read the full trace with query. A Sandbox creation failure can happen before trace collection starts, so the error does not depend on the trace and show @<locator> can always show it.

Pass the model and flags to the CLI

The model and flags written in the Experiment appear on ctx.model and ctx.flags. The Adapter converts them into CLI arguments or a request body, and only then does the Agent really switch models or configuration.

Use a custom Agent in an Experiment

Put your Agent into an Experiment the same way as a built-in Agent:

import { defineExperiment } from "niceeval";
import { dockerSandbox } from "niceeval/sandbox";
import myAgent from "./agents/my-agent";

export default defineExperiment({
  agent: myAgent,
  model: "claude-sonnet-4-6",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
  attempts: 3,
});

pnpm exec niceeval exp <experiment-id>

attempts: 3 runs each eval 3 times. A Coding Agent’s approach varies a lot from run to run, so one run’s result does not say much.

Each Agent’s distinct behavior is written in its own Adapter. Do not branch on Agent names anywhere else.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Sandbox Agent: Evaluate Claude Code, Codex, and bub

Built-in Sandbox Agents

claude-code

codex

bub

Run a built-in Agent

Agent environment variables

What one Attempt does, in order

Make file changes recordable

Custom Sandbox Agents

Pass the model and flags to the CLI

Use a custom Agent in an Experiment