Skip to main content
The three built-in sandbox agents (claude-code, codex, bub) do not implement exactly the same capabilities — capabilities have no declaration layer and are proven entirely by construction and actual behavior, see the capabilities reference. This page inventories what each built-in agent does per capability, plus two built-ins for connecting AI SDK apps: the non-invasive HTTP adapter uiMessageStreamAgent (with HITL) and the result converter turnFromAiSdk.

Capability overview

What each built-in agent implements

All three are constructed with defineSandboxAgent (Agent.kind is always "sandbox"), so t.sandbox.fileChanged() / diff assertions, file IO, and command execution work on all three agents regardless of this table.
None of the three built-in sandbox agents currently support HITL: their send only returns "completed" / "failed", never "waiting", and never emits input.requested events. When you need t.respond() / t.requireInputRequest(): if the system under test is an AI SDK useChat backend, use the built-in uiMessageStreamAgent below (v7 tool approval maps natively to HITL); for anything else write your own adapter: implement the waiting status + input.requested event + resume handoff — examples/zh/tier1/pi-sdk / examples/zh/tier1/claude-sdk both have ready-made hand-written HITL references.

Per-agent details

claude-code

  • Connection: spawns claude --print --dangerously-skip-permissions in the Sandbox and reads back the latest transcript from ~/.claude/projects/**/*.jsonl.
  • Session continuation: appends --resume <id> when ctx.session.id has a value; the session id parsed from the transcript is written back via ctx.session.capture().
  • Auth: ANTHROPIC_API_KEY, optional ANTHROPIC_BASE_URL; configuration options are listed under ClaudeCodeConfig below.
  • tracing is configured via the claude CLI’s native OTLP trace spans (beta), protocol http/protobuf: setting CLAUDE_CODE_ENABLE_TELEMETRY and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA (a beta toggle, must be explicitly enabled) in env hands the endpoint to the CLI; trace decoding shows a waterfall at the interaction / llm_request / tool level.

codex

  • Connection: runs codex exec --json in the Sandbox (codex exec resume <id> --json when continuing), using stdout JSONL as the transcript. The command carries --dangerously-bypass-approvals-and-sandbox and --dangerously-bypass-hook-trust: nobody in the Sandbox can answer codex’s interactive approval prompts, so hooks installed by a plugin or postSetup take effect without needing interactive trust confirmation.
  • Auth: CODEX_API_KEY (not OPENAI_API_KEY), optional CODEX_BASE_URL for OpenAI-compatible proxies; configuration options are listed under CodexConfig below.
  • tracing is configured through the [otel.trace_exporter.otlp-http] section of ~/.codex/config.toml, protocol http/json.

bub

  • Connection: runs bub run + --session-id in the Sandbox and reads the tape from ~/.bub/tapes/<hash>.jsonl as the transcript.
  • Auth: BUB_API_KEY + BUB_API_BASE (OpenAI-compatible proxy); configuration options are listed under BubConfig below.
  • tracing is injected via environment variables (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, etc.), protocol http/protobuf.
  • Installed via uv tool install (not an npm package); the first install builds a checkpoint cache to speed up later Sandboxes.

Configuration options for the three built-in sandbox agents

ClaudeCodeConfig

apiKey

Anthropic API key. Falls back to the ANTHROPIC_API_KEY env var if omitted.

baseUrl

Custom API base URL (proxy / internal endpoint). Falls back to the ANTHROPIC_BASE_URL env var if omitted; if neither is set, uses the Anthropic official endpoint (the claude CLI’s default behavior).

maxTurns

Maximum number of tool-use turns to run (→ --max-turns). Caps the eval’s cost ceiling; if omitted, uses the CLI’s native default (unlimited).

mcpServers

Additional MCP servers (written into the user-level ~/.claude.json on each Sandbox setup). The stdio form writes command (optionally with args / env); the Streamable HTTP form writes url (optionally with headers, carried through verbatim into the request headers), landing as a { “type”: “http”, “url”: …, “headers”: … } entry.

skills

Skills to install into the Sandbox (a local directory/file, or a repo with an optional pinned ref and an optional enabled subset). They land in the project-level .claude/skills/<name>/, where the claude CLI discovers them natively.

plugins

Claude Code native plugins (connect the marketplace first, then install the named plugin from it).

settingsFile

Path to a complete Claude Code settings.json (official format) in the local project — resolved relative to the project root running niceeval (the directory containing niceeval.config.ts), not a path inside the Sandbox; only relative paths within the project root are accepted — paths containing .., absolute paths, ~ paths, and symlinks that resolve outside the project root all error during setup. The raw bytes are uploaded verbatim as the user-level ~/.claude/settings.json, which is otherwise empty in the Sandbox (no inheriting host config, no merging, no re-serializing); the reserved keys model and env appearing in the file error during setup. The manifest records only the project-relative path and the byte SHA-256, never the file body.

postSetup

User hooks that run, in array order, after install (reusing SandboxHook’s narrow context): executed once writing settings, wiring MCP, installing Skills / plugins, and writing the manifest are all done — suited for process actions that can only run once install artifacts are in place, such as a plugin’s own setup script. A thrown error counts as an infrastructure error (attempt errored). See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The teardown hook paired with postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded into a teardown-failed diagnostic by the teardown phase. See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

CodexConfig

apiKey

Proxy / OpenAI API key. Falls back to the CODEX_API_KEY env var if omitted.

baseUrl

OpenAI-compatible proxy base URL (e.g. https://s2a.example.com/v1). Falls back to the CODEX_BASE_URL env var if omitted.

mcpServers

Additional MCP servers (appended into ~/.codex/config.toml on each Sandbox setup). The stdio form (command/args/env) writes the command line under [mcp_servers.<name>]; the Streamable HTTP form (url/headers) writes the url line, with headers going into the [mcp_servers.<name>.http_headers] subtable.

skills

Skills to install into the Sandbox (a local directory/file, or a repo with an optional pinned ref and an optional enabled subset). They land in .agents/skills/<name>/, together with a discovery instruction written into AGENTS.md — codex has no native Skill tool like Claude Code, so merely installing the files does not make it read them (see memory/codex-no-native-skill-tool.md).

plugins

Codex native plugins (connect the marketplace first, then install the named plugin from it).

configFile

Path to a complete Codex config.toml (official TOML format) in the local project — resolved relative to the project root running niceeval (the directory containing niceeval.config.ts), not a path inside the Sandbox; only relative paths within the project root are accepted — paths containing .., absolute paths, ~ paths, and symlinks that resolve outside the project root all error during setup. The raw bytes are merged verbatim into the user-level ~/.codex/config.toml, which is otherwise empty in the Sandbox (no inheriting host config, no parse-and-rewrite); the reserved keys model, model_provider, model_providers, model_reasoning_effort, mcp_servers, and otel appearing in the file error during setup. The manifest records only the project-relative path and the byte SHA-256, never the file body.

postSetup

User hooks that run, in array order, after install (reusing SandboxHook’s narrow context): executed once writing the main config, wiring MCP, installing Skills / plugins, and writing the manifest are all done — suited for process actions that can only run once install artifacts are in place, such as a plugin’s own setup script. A thrown error counts as an infrastructure error (attempt errored). See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The teardown hook paired with postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded into a teardown-failed diagnostic by the teardown phase. See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

BubConfig

apiKey

API key for the OpenAI-compatible proxy. Falls back to the BUB_API_KEY env var if omitted.

apiBase

Base URL for the OpenAI-compatible proxy. Falls back to the BUB_API_BASE env var if omitted.

skills

Skills to install into the Sandbox (a local directory/file, or a repo with an optional pinned ref and an optional enabled subset). They land in .agents/skills/<name>/, with a discovery instruction written into AGENTS.md (bub has no native Skill loading mechanism).

pythonPlugins

Extra Python packages to install into the bub tool environment; on each Sandbox setup they go into uv tool install … --with <pkg>. The normalized package list is part of the install checkpoint key: two agent variants with different plugin sets never reuse the same install checkpoint (otherwise the second variant would silently inherit the first one’s environment).

postSetup

User hooks that run, in array order, after install (reusing SandboxHook’s narrow context): executed once installing bub, installing Skills / Python packages, and writing the manifest are all done. A thrown error counts as an infrastructure error (attempt errored). See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The teardown hook paired with postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded into a teardown-failed diagnostic by the teardown phase. See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.

uiMessageStreamAgent: built-in non-invasive adapter for AI SDK apps (with HITL)

uiMessageStreamAgent (exported from niceeval/adapter) talks non-invasively to an HTTP endpoint speaking the UI Message Stream protocol (the standard SSE of AI SDK useChat backends) — it only fetches, never imports app code, so it connects wherever the app is deployed:
Capabilities it handles for you:
  • Send/receive + event stream: SSE frames are reduced by the ai package’s official framework-agnostic reducer readUIMessageStream (the same one inside useChat); tool calls/results/message text are built directly from message parts — the app is not required to wire up OTel.
  • Session continuation: the protocol is server-stateless, “client carries the full history” — the factory stores the whole UIMessage[] in an adapter-private typed session slot and replays it verbatim each turn; a new conversation line (after t.newSession()) gets an empty history.
  • HITL: AI SDK v7 tool approval (tools with needsApproval: true) maps natively — when a part stops at approval-requested, the whole turn is status: "waiting" + input.requested; t.respond("approve" / "deny") is translated into approval-responded, rewriting that part in place and resending messages verbatim to trigger the server to continue (identical protocol behavior to a real frontend’s addToolApprovalResponse() + sendMessage(); there is no separate approve endpoint). Denied calls land in the event stream as rejected, and by default carry a “do not retry” reason (denyReason can override) — without one, models often resend the very same call.
  • usage / waterfall: UI Message Stream protocol frames carry no usage, so usage assertions like t.maxTokens have no data by default on this built-in (an app putting usage into message metadata is the app’s own protocol extension). The waterfall is a separate matter: when the app has OTel instrumentation (e.g. the official @ai-sdk/otel), send spans to NiceEval per OTel integration and niceeval view gets a full waterfall — spans only feed the waterfall, never assertions.
You need ai installed in the eval project (optional peer dependency; the protocol reducer comes from it). Full runnable example: examples/zh/tier1/ai-sdk-v7. Full parameters of uiMessageStreamAgent(options) (UiMessageStreamAgentOptions):

name

Agent name (identity for reports / result aggregation). Defaults to “ui-message-stream”.

url

The chat endpoint of the system under test (a full URL, pointing wherever the app is deployed); the function form is resolved every turn.

headers

Additional request headers (auth, etc.); ctx.telemetry.headers (traceparent) is always merged in automatically.

body

Fields merged into the request body besides messages, e.g. (ctx) => ({ model: ctx.model }) (undefined fields are dropped automatically during serialization).

denyReason

The reason carried with approval-responded when an approval is denied. The app/SDK passes it to the model as the tool result text — spelling out “do not retry” measurably lowers the chance the model resends the very same call (observed empirically).

settleMs

How long to wait after the stream ends before returning (milliseconds), giving the app’s observability export (e.g. a BatchSpanProcessor) time to flush.

tracing

How to deliver the endpoint when the app has OTel (for the waterfall); the event stream does not depend on it.

spanMapper

The span-normalizing function when the app has OTel (for the waterfall); the event stream does not depend on it.

OpenAI-compatible response converters: turnFromChatCompletion / turnFromResponses

turnFromChatCompletion(res) / turnFromResponses(res) (exported from niceeval/adapter) map OpenAI’s two response shapes — Chat Completions and Responses — from the whole response into a Turn with zero manual mapping: this is not limited to the official OpenAI API — any service that claims compatibility with either protocol shape works.
The two shapes differ in how trustworthy negative assertions (notCalledTool and friends) are: Chat Completions does not guarantee “response = complete process” (the app may run the whole tool loop server-side and only hand you the final answer), so a negative assertion can only mean “not observed,” not “did not happen”; the Responses protocol’s contract guarantees the output array records everything the model decided to do this turn (including every function_call), so negative assertions there are trustworthy. The Turn shape the two converters produce is identical — this difference only affects how you interpret negative assertions.

SDK event stream converters: createClaudeSdkEventStream / createPiAgentEventStream / createCodexThreadEventStream

Each agent SDK’s streaming protocol is a generic protocol defined by the SDK, not some app’s private format — the native-frame-to-standard-event mapping knowledge lives in the official package (exported from niceeval/adapter), so when writing your own non-invasive adapter only the transport glue remains (which endpoint the app puts the stream on, which endpoint approvals go through). Driving the loop frame by frame is itself an official piece (driveFrameStream, see the next section), so no hand-written for loop:
  • createClaudeSdkEventStream (Claude Agent SDK SDKMessage): assistant text/tool_use blocks, user tool_result blocks, system/permission_denied (→ rejected), result usage; markRejected() registers denied calls.
  • createPiAgentEventStream (pi-agent-core AgentEvent): text/thinking/usage from message_end, tool pairs from tool_execution_start/end.
  • createCodexThreadEventStream (Codex SDK ThreadEvent): message-class frames from agent_message/reasoning, tool items (command_execution / mcp_tool_call / file_change / web_search → paired tool operation.started + operation.finished), usage from turn.completed, error frames. The waterfall needs the codex CLI’s native OTLP (the [otel] block in config.toml) sending spans to NiceEval, normalized with the official mapCodexSpans.
Runnable references: the claude-sdk / pi-sdk / codex-sdk adapters under examples/zh/tier1 are exactly these three converters + driveFrameStream + transport glue.

Generic “assembly” pieces: driveFrameStream / deltaStream + ctx.session

Inside a hand-written send, only three truly independent parts exist: transport (how to send), reduce (raw data → events, handled by the converters above), and orchestration (session continuation + HITL pause/resume). The third part is protocol-agnostic, pure control-flow pattern — session continuation (id/capture) and adapter-private typed slots hang directly off ctx.session (AgentSession, see Adapter concept). Create a slot once with createSessionSlot<T>(), then use set(slot, value) and take(slot) for a paused scene. What actually remains reusable across protocols is just two official pieces:
The three Tier 1 examples (claude-sdk and pi-sdk use driveFrameStream plus typed session slots; codex-sdk uses only driveFrameStream) are built on these pieces — the adapters are down to transport plus the “does this frame need extra handling” decision in onFrame, with no hand-written loops or module-level Maps.

turnFromAiSdk: AI SDK result → event stream converter

turnFromAiSdk (exported from niceeval/adapter) is used inside adapters you write yourself (e.g. server-side direct construction for an HTTP web agent, see examples/zh/ai-sdk/). It maps an AI SDK generateText / streamText result into { events, usage, status } — exact toolCallId pairing, order-faithful, tool-error mapped to a failed tool operation.finished, usage aggregation (v4 / v5 / v7 field drift all covered) — laid straight into the Turn:
The status comes from the converter: "waiting" when there is a tool approval pending human review (with an input.requested event attached), otherwise "completed". Session continuation (multi-turn resume), handing HITL verdicts back, and tracing depend on how you write your own send.

How to choose

  • The system under test is an AI SDK app (useChat backend): the built-in uiMessageStreamAgent connects non-invasively with zero mapping (including HITL).
  • The system under test is another agent system (HTTP / gRPC service): connect non-invasively and hand-write the event mapping (official converters cover most of it) — all five examples under examples/zh/tier1 take this path; if the app already has OTel instrumentation, wire it up for a bonus waterfall.
  • You need the call waterfall (the trace in niceeval view): all three of claude-code / codex / bub emit OTLP themselves (claude-code via the CLI’s beta telemetry; its spans carry structure and timing only); adapters you write yourself just declare tracing (see the OTel guide).
  • You need human approval / multi-step confirmation (HITL): uiMessageStreamAgent supports it natively (AI SDK v7 tool approval); none of the three sandbox agents do; for other systems write your own adapter, assembled from driveFrameStream plus typed session slots (the pi-sdk / claude-sdk Tier 1 examples have ready-made patterns).
  • You want to run a coding agent that edits code, inspect diffs, and judge tool calls: claude-code / codex / bub (send/receive + event stream + session continuation + workspace + sandbox).
  • Your backend protocol is home-grown with no official SDK converter: do not hand-write send from scratch — pick a reduce shape (a small mapping for whole-chunk landing, deltaStream for token-by-token deltas), use ctx.session.id/capture for session continuation, add a typed session slot if you have HITL, and the assembly is your send.