> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Sandbox Agents: Evaluate Claude Code, Codex, and bub
> Use NiceEval built-in Agents or a custom Adapter to run coding-agent CLIs in Docker or cloud Sandboxes.
A Sandbox Agent starts a coding-agent CLI in an isolated environment, gives it a workspace and task, lets it edit files and run commands, then collects its transcript, diff, and test results.
## Built-in Sandbox Agents
Runs the Anthropic Claude Code CLI and requires `ANTHROPIC_API_KEY`.
Runs the OpenAI Codex CLI and requires `CODEX_API_KEY`.
Runs the bub coding agent and follows bub's own authentication rules.
## Run a built-in Agent
Set `sandbox` on an Eval or Experiment to choose where [NiceEval](https://niceeval.com/) creates the isolated environment:
```ts theme={null}
// experiments/local.ts
import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";
export default defineExperiment({
agent: claudeCodeAgent(),
model: "claude-sonnet-4-6",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});
```
```shell theme={null}
export ANTHROPIC_API_KEY=sk-ant-...
npx niceeval exp local fixtures/button
npx niceeval exp local fixtures/button --attempts 10
```
There is no matching CLI flag or project-wide default provider. If neither the Eval nor Experiment contributes a template-bearing layer, link planning fails before NiceEval creates a Sandbox.
For a coding Agent in the cloud, use NiceEval's published E2B public template to avoid installing the CLI for every Attempt:
```ts theme={null}
import { codexAgent } from "niceeval/adapter";
import { e2bSandbox } from "niceeval/sandbox";
import { NICEEVAL_CODEX_E2B_TEMPLATE } from "niceeval/sandbox/e2b-template";
export default defineExperiment({
agent: codexAgent(),
model: "gpt-5.4",
sandbox: e2bSandbox({ template: NICEEVAL_CODEX_E2B_TEMPLATE }),
});
```
Use `NICEEVAL_CLAUDE_CODE_E2B_TEMPLATE` for Claude Code or `NICEEVAL_BUB_E2B_TEMPLATE` for bub. Each constant is a complete, version-pinned reference whose version follows the Agent in that template. See [Sandbox Providers: Prebuilt Environments and Runtime Checkpoints](/docs/tutorials/sandbox-providers#prebuilt-environments-and-runtime-checkpoints) to add system packages, binaries, or model caches.
Built-in Agents exported from `niceeval/adapter` are factories. Put authentication, proxies, MCP, or GitHub skills in the factory arguments. Keep the model in the Experiment's `model` field and the provider in its `sandbox` field:
```ts theme={null}
import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";
export default defineExperiment({
agent: claudeCodeAgent({
apiKey: process.env.ANTHROPIC_API_KEY,
baseUrl: process.env.ANTHROPIC_BASE_URL,
maxTurns: 8,
mcpServers: [
{
name: "browser",
command: "npx",
args: ["-y", "@anthropic/mcp-browser"],
},
],
skills: [{ kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2" }],
}),
model: "claude-sonnet-4-6",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});
```
## Agent environment variables
| Agent | Required variable |
| ------------- | ------------------------------------------ |
| `claude-code` | `ANTHROPIC_API_KEY` |
| `codex` | `CODEX_API_KEY` |
| `bub` | Follow the bub CLI's authentication rules. |
## Workflow
```text theme={null}
Create a Sandbox
→ Run the Sandbox setup lifecycle
→ Run the Eval setup lifecycle and prepare Fixtures
→ Check the Agent CLI version; install it and check again when needed
→ Run Adapter setup and write authentication and run configuration
→ Run test(t) and write starting files
→ The Adapter calls the Agent and records its file changes
→ Run verification commands, score the result, and form a Verdict
→ Run Adapter and Sandbox teardown lifecycles
→ Stop the Sandbox
```
Write starting files and verification commands in `test(t)`. The Agent can see only files you wrote into the Sandbox.
`t.sandbox.fileChanged()` and the other attribution assertions cover only files the Agent changed during `t.send()`. NiceEval records workspace state before and after each `t.send()`, so the changes between those points belong to the Agent. Starting files you uploaded and verification materials you write after `t.send()` are not included. `fileChanged("src/app.ts")` passes only when the Agent really changed that file.
## Requirements for collecting file changes
NiceEval creates a private Git repository in the Sandbox. It records workspace state before and after each `t.send()`; the difference is the Agent's work. Collection requires all of the following:
* **The Sandbox has Git and a POSIX shell.** Built-in provider images include them. Verify them for a custom image. The project under test does not need to be a Git repository: NiceEval's private repository lives outside the workdir, does not touch your `.git`, and is not visible to the Agent.
* **Files are written in the workdir.** Collection covers `sandbox.workdir`. Omit `targetDir` or `cwd` when writing files. For an absolute path, read `sandbox.workdir`; do not hard-code `/workspace`. Files outside the workdir are neither visible to the Agent nor collected.
* **The workdir root has no nested Git repository.** Put the checkout directly in the workdir root. A submodule or another clone there is an execution error that lists the path, because normal file changes inside it would disappear from the record. Exclude a repository that does not participate in scoring with `diff.ignore`.
* **The change happens during `t.send()`.** Files written after the final `t.send()` are your verification material, not Agent work.
NiceEval excludes `.git`, `node_modules`, `__pycache__`, Python virtual environments, common build output, and package-manager caches by default. Otherwise one `npm install` could produce tens of thousands of paths. Adjust that list with an Eval's `diff` field:
```ts theme={null}
export default defineEval({
diff: {
ignore: ["fixtures/**"], // A change here is not Agent work
include: ["node_modules/some-pkg/**"], // Include a patched dependency
},
async test(t) { /* ... */ },
});
```
Patterns use Git-ignore syntax relative to the workdir root. A pattern without `/` matches a name at every depth; one with `/` starts at the workdir root; a trailing `/` denotes a directory. The project's own `.gitignore` does not participate, so project-ignored files are still recorded. NiceEval freezes the list at the first snapshot; a later Agent edit to `.gitignore` cannot affect it.
One `t.send()` window can collect at most 10,000 paths and 64 MiB of content. Exceeding either limit is an execution error, not an empty change set that could let a file assertion pass.
Lifecycle callbacks (`.setup()` and `.teardown()`) belong to the Experiment's `sandbox` specification. Use them for environment setup that changes by Experiment: install an Experiment-specific binary, warm a cache, or load and save state across Attempts. Files they write are environment state, not part of the Agent diff. See [Sandbox Providers: Lifecycle](/docs/tutorials/sandbox-providers#lifecycle) for the API and rules.
## Create a custom Sandbox Agent
```ts theme={null}
import {
completeEvidenceCoverage,
createNpmCliInstaller,
defineSandboxAgent,
resolveAgentBin,
} from "niceeval/adapter";
const { ensure, installer } = createNpmCliInstaller({
identity: { agent: "my-agent", version: "1.0.0", revision: "1" },
packageName: "my-agent",
bin: "my-agent",
});
export default defineSandboxAgent({
name: "my-agent",
evidenceCoverage: completeEvidenceCoverage,
ensure,
installers: [installer],
async send(input, ctx) {
ctx.progress({ message: "Running my-agent CLI" });
const bin = await resolveAgentBin(ctx.sandbox, "my-agent");
await ctx.sandbox.runCommand(bin, ["run", input.text, "--json-out", "agent-events.json"]);
const transcript = await ctx.sandbox.readText("agent-events.json");
if (transcript.trim() === "") {
ctx.diagnostic({
code: "empty-transcript",
level: "warning",
message: "my-agent did not write a transcript; tool assertions might lack evidence",
});
}
return {
status: "completed",
events: parseTranscript(transcript),
};
},
});
```
`createNpmCliInstaller` prepares the npm package on the host, then sends it into the Sandbox. The Sandbox does not need Node.js or npm. The complete workflow is in [Create a Custom Sandbox Agent](#create-a-custom-sandbox-agent).
`setup`, every `send`, and `teardown` each receive feedback methods for their own scope:
* `ctx.progress({ message, current?, total? })` updates short-lived status. Use it for CLI installation, a Turn, or transcript reading. Do not call it for every token or JSONL frame.
* `ctx.diagnostic({ code, level, message, data?, dedupeKey? })` records protocol degradation, missing transcripts, and cleanup problems. It enters the terminal's permanent event stream and is committed as a channel event with the Attempt.
* Throw when execution cannot continue. The Runner writes the stage, code, message, cause, and stack to a diagnostic channel, then forms an `errored` Verdict in the `niceeval.verdict` channel.
Do not call `console.log` or `console.error`, or write `process.stdout` or `process.stderr`, from an Adapter. That breaks up the human dashboard and CI log order.
When a Sandbox or Adapter error appears during a run, the terminal prints an Attempt identity:
```text theme={null}
✗ @5TB8167MXJ30SYZCNAVRHPQ4D2 fixtures/button [local] errored · agent setup
agent-install-failed: npm install my-agent exited with code 1
Inspect: niceeval show --run --page attempt-5TB8167MXJ30SYZCNAVRHPQ4D2
```
Run the displayed `show --run --page attempt-` command to inspect the structured error, diagnostics, and completed lifecycle stages. That page reads the timing channel, including queueing, Sandbox startup, shell work in setup and teardown, Agent CLI installation and startup, each send, correlated OTel model or tool work, and cleanup. It identifies the layer where an error or timeout occurred and where earlier time went.
When the tree has more than 80 detail nodes, it retains failures, slow points, and head and tail samples, then reports how many nodes it omitted. To audit every node, choose the complete planned timing route from the same Sample's page index. An execution route uses events as its structure and attaches timing only to uniquely correlated events when OTel is present. Sandbox creation can fail before telemetry starts, so error review does not depend on a trace.
## `ctx.model` and `ctx.flags`
The model and flags declared by an Experiment appear in the Adapter context. An Adapter can turn them into CLI arguments or an HTTP payload.
## Use a custom Agent in an Experiment
```ts theme={null}
import { defineExperiment } from "niceeval";
import { dockerSandbox } from "niceeval/sandbox";
import myAgent from "./agents/my-agent";
export default defineExperiment({
agent: myAgent,
model: "claude-sonnet-4-6",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
attempts: 3,
});
```
Do not add Agent-specific branches to the Runner. Put behavioral differences in the Adapter.