Skip to main content
The three built-in Sandbox Agents—claude-code, codex, and bub—do not all provide the same capabilities. There is no capability-declaration layer: construction and actual behavior prove them. See the Capabilities reference. This page inventories what each built-in Agent provides, plus two built-ins for AI SDK applications: the non-intrusive HTTP Adapter uiMessageStreamAgent, including HITL, and the result converter turnFromAiSdk.

Capability overview

What each built-in Agent provides

All three use defineSandboxAgent (Agent.kind is always "sandbox"), so filesystem assertions and file I/O and command execution—including t.sandbox.fileChanged() and diff assertions—work with all three Agents regardless of the table.
The three built-in Sandbox Agents do not support HITL. send returns only "completed" / "failed"; it never returns "waiting" or emits input.requested.When you need t.respond() / t.requireInputRequest(): if the system under test is an AI SDK useChat backend, use the built-in uiMessageStreamAgent below, which natively maps v7 tool approval to HITL. For another system under test, write an Adapter that implements a waiting status, input.requested event, and resume handoff. examples/zh/tier1/pi-sdk and examples/zh/tier1/claude-sdk both provide ready-made references for manual HITL.

Details by Agent

claude-code

  • Connection: spawns claude --print --dangerously-skip-permissions in the Sandbox and reads the newest transcript at ~/.claude/projects/**/*.jsonl.
  • Session continuation: when ctx.session.id has a value, appends --resume <id>; the session ID decoded from the transcript is written back with ctx.session.capture().
  • Authentication: ANTHROPIC_API_KEY, optionally ANTHROPIC_BASE_URL. See ClaudeCodeConfig below for options.
  • tracing uses the claude CLI’s native beta OTLP trace spans over http/protobuf. Set CLAUDE_CODE_ENABLE_TELEMETRY and CLAUDE_CODE_ENHANCED_TELEMETRY_BETA in the environment—the explicit beta opt-in—to give the endpoint to the CLI. Trace decoding displays an interaction / llm_request / tool hierarchy waterfall.

codex

  • Connection: runs codex exec --json in the Sandbox, or codex exec resume <id> --json for continuation, and uses stdout JSONL as the transcript. The command includes --dangerously-bypass-approvals-and-sandbox and --dangerously-bypass-hook-trust: nobody can answer Codex’s interactive confirmations in a Sandbox, so a Hook installed by a Plugin or postSetup can take effect without interactive trust.
  • Authentication: CODEX_API_KEY, and optionally CODEX_BASE_URL for an OpenAI-compatible proxy. See CodexConfig below for options.
  • tracing is configured through [otel.trace_exporter.otlp-http] in ~/.codex/config.toml, over http/json.

bub

  • Connection: runs bub run with --session-id in the Sandbox and reads the tape at ~/.bub/tapes/<hash>.jsonl as the transcript.
  • Authentication: BUB_API_KEY plus BUB_API_BASE for an OpenAI-compatible proxy. See BubConfig below for options.
  • tracing is injected through environment variables such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, over http/protobuf.
  • Installation uses uv tool install, not an npm package. The first install creates a checkpoint cache to speed later Sandboxes.

Configuration for the three built-in Sandbox Agents

ClaudeCodeConfig

apiKey

The Anthropic API key. When omitted, read ANTHROPIC_API_KEY from the environment.

baseUrl

A custom API base URL, for a proxy or internal endpoint. When omitted, read ANTHROPIC_BASE_URL from the environment. When neither exists, use the Anthropic official endpoint, the claude CLI default behavior.

env

Extra environment variables for the Claude Code process. Values remain only at runtime and do not enter result carry identity. A non-sensitive value that changes behavior must also appear in Experiment Flags or the owning Plugin’s identity.

maxTurns

The maximum number of tool-use turns, passed to --max-turns. It limits Eval cost; when omitted, the CLI native default is unlimited.

mcpServers

Extra MCP servers, written to user-level ~/.claude.json during each Sandbox setup. The stdio form writes command, optionally with args / env. The Streamable HTTP form writes url, optionally with headers, copied verbatim into request headers, as a { "type": "http", "url": …, "headers": … } entry.

skills

Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to project-level .claude/skills/<name>/, where the claude CLI discovers them natively.

plugins

Native Claude Code Plugins: first connect a Marketplace, then install the named Plugin from it.

settingsFile

The path to a complete official-format Claude Code settings.json in the local project. It resolves relative to the project root that runs niceeval—the directory containing niceeval.config.ts—not inside the Sandbox. It accepts only a relative path within the project root. A path containing .., an absolute path, a ~ path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim as the otherwise-empty user-level ~/.claude/settings.json in the Sandbox. It does not inherit host configuration, concatenate, or reserialize. Reserved keys model and env cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.

postSetup

User Hooks run in array order after installation. They reuse SandboxCommand’s narrow context and run only after settings are written, MCP attached, Skills and Plugins installed, and the manifest written. They suit process actions that can run only after installed artifacts are ready, such as a Plugin’s setup script. A throw counts as an infrastructure error (attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The cleanup Hook paired with postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

CodexConfig

apiKey

Proxy / OpenAI API key. Falls back to the CODEX_API_KEY env var if omitted.

baseUrl

OpenAI-compatible proxy base URL (e.g. https://s2a.example.com/v1). Falls back to the CODEX_BASE_URL env var if omitted.

env

Extra environment variables injected into every Codex CLI process. The first codex exec and continuing codex exec resume use the same declaration; lifecycle Hooks started by Codex, dynamic MCP headers, and command subprocesses all inherit it. Values enter only through Sandbox command options: they are not concatenated into shell text or written to the setup manifest, and are all redacted as potential sensitive values from timing, execution, and error evidence. CODEX_API_KEY still comes from apiKey or the same host environment variable; the Adapter’s authentication value overrides a same-named key here. An environment value does not enter carry identity. A non-sensitive value that changes behavior must also be declared in Experiment Flags or the owning Plugin identity; rotating credentials alone does not invalidate old results. PATH is Sandbox-managed and cannot be declared here. Its presence errors at factory construction; use the Sandbox factory’s pathPrepend instead. See docs/feature/sandbox/library.md.

mcpServers

Extra MCP servers, appended to ~/.codex/config.toml during each Sandbox setup. The stdio form (command / args / env) writes a command line in [mcp_servers.<name>]. The Streamable HTTP form (url / headers) writes the url line and headers in the [mcp_servers.<name>.http_headers] subtable.

skills

Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to .agents/skills/<name>/, and a discovery instruction is written into AGENTS.md. Codex has no native Skill tool like Claude Code, so installing the file alone does not make it read it. See memory/codex-no-native-skill-tool.md.

plugins

Native Codex Plugins: first connect a Marketplace, then install the named Plugin from it.

configFile

The path to a complete official-TOML Codex config.toml in the local project. It resolves relative to the project root that runs niceeval—the directory containing niceeval.config.ts—not inside the Sandbox. It accepts only a relative path within the project root. A path containing .., an absolute path, a ~ path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim into the otherwise-empty user-level ~/.codex/config.toml in the Sandbox. It does not inherit host configuration or parse and rewrite it. Reserved keys model, model_provider, model_providers, model_reasoning_effort, mcp_servers, and otel cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.

postSetup

User Hooks run in array order after installation. They reuse SandboxCommand’s narrow context and run only after the main configuration is written, MCP attached, Skills and Plugins installed, and the manifest written. They suit process actions that can run only after installed artifacts are ready, such as a Plugin’s setup script. A throw counts as an infrastructure error (attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The cleanup Hook paired with postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

BubConfig

apiKey

API key for the OpenAI-compatible proxy. Falls back to the BUB_API_KEY env var if omitted.

apiBase

Base URL for the OpenAI-compatible proxy. Falls back to the BUB_API_BASE env var if omitted.

skills

Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to .agents/skills/<name>/, and a discovery instruction is written into AGENTS.md. Bub has no native Skill-loading mechanism.

version

Which Bub version to install, as a PyPI version such as "0.4.0". When omitted, use the default version pinned by NiceEval. It is always a deterministic version and never latest: the system under test’s version must be visible from Experiment configuration.

otelPlugin

The git dependency for the OTel tape-store Plugin, the source of the timeline. When omitted, use NiceEval’s default pin. The Plugin and Bub must be from the same generation of the tape protocol. The default pin takes types from bub.tape and requires Bub ≥ 0.3.10. Earlier Plugin commits validate against republic types and pair with Bub ≤ 0.3.9. A mismatched generation does not fail installation; it rejects every span and leaves the timeline silently empty. When pinning version backward, pin the matching Plugin commit in the same change.

pythonPlugins

Extra Python packages installed into the bub tool environment. Each Sandbox setup adds them through uv tool install … --with <pkg>. The normalized package list enters the installation checkpoint key, so two Agent variants with different Plugin sets never reuse the same checkpoint; otherwise the second could silently receive the first variant’s environment.

postSetup

User Hooks run in array order after installation. They reuse SandboxCommand’s narrow context and run only after bub, Skills and Python packages are installed, and the manifest is written. A throw counts as an infrastructure error (attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

preTeardown

The cleanup Hook paired with postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.

uiMessageStreamAgent: built-in non-intrusive Adapter for AI SDK applications, including HITL

uiMessageStreamAgent, exported from niceeval/adapter, sends and receives non-intrusively against an HTTP endpoint implementing the UI Message Stream protocol, the standard SSE for an AI SDK useChat backend. It only uses fetch; it does not import application code, so it can connect wherever the application is deployed:
Capabilities it provides for you:
  • Send/receive and event stream: SSE frames are reduced through the framework-agnostic readUIMessageStream reducer from the official ai package—the same reducer useChat uses internally. Tool calls, results, and message text are constructed directly from message parts. The application does not need OTel.
  • Session continuation: the protocol is stateless on the server and the client carries complete history. The factory stores complete UIMessage[] history in an Adapter-private typed slot and replays it verbatim each turn. A new session line after t.newSession() has not written that slot yet.
  • HITL: AI SDK v7 tool approval—a tool with needsApproval: true—maps natively. When a part stops at approval-requested, that Turn has status: "waiting" plus input.requested. t.respond("approve" / "deny") translates to approval-responded, rewrites that part in place, and resends messages to continue server execution. That is the same protocol behavior as a real frontend’s addToolApprovalResponse() plus sendMessage(). There is no separate approval endpoint.
  • Rejection: an invocation enters the event stream with rejected and, by default, a “do not retry” reason that denyReason can override. Without it, models often repeat the exact call.
  • Usage and waterfall: UI Message Stream frames have no usage, so usage assertions such as t.maxTokens have no data by default on this built-in. An application that puts usage in message metadata is adding its own protocol extension. The waterfall is separate: when the application has OTel instrumentation, such as official @ai-sdk/otel, send spans to NiceEval according to OTel integration and niceeval view has the complete waterfall. Spans go only to the waterfall, not assertions.
Install ai in the Eval project. It is an optional peer dependency because the protocol reducer comes from it. For a complete runnable example, see examples/zh/tier1/ai-sdk-v7. Complete uiMessageStreamAgent(options) parameters (UiMessageStreamAgentOptions):

name

The Agent name used for Report and result-aggregation identity. Defaults to "ui-message-stream".

url

The chat endpoint of the application under test. Use its full URL wherever the application is deployed. The function form resolves once per turn.

headers

Additional request headers, such as authentication. ctx.telemetry.headers for traceparent always merges in automatically.

body

Fields to merge into the request body in addition to messages, such as (ctx) => ({ model: ctx.model }). Fields whose value is undefined are automatically omitted during serialization.

projectToolCommand

Lets the endpoint owner classify every logical tool call as command or not-command. An unknown call returns undefined and leaves actions coverage partial. NiceEval does not guess from a name or input.

denyReason

The reason sent with approval-responded when approval is denied. The application and SDK make it tool-result text that the model sees. A clear “do not retry” message measurably lowers the chance that a model repeats the same call.

settleMs

Milliseconds to wait after the stream ends before returning, giving application observability exporters such as BatchSpanProcessor time to export.

tracing

How an application with OTel delivers spans to the endpoint, for the waterfall. The event stream does not depend on it.

spanMapper

How an application with OTel normalizes spans, for the waterfall. The event stream does not depend on it.

OpenAI-compatible response converters: turnFromChatCompletion / turnFromResponses

turnFromChatCompletion(res) and turnFromResponses(res), exported from niceeval/adapter, convert OpenAI’s two response shapes—Chat Completions and Responses—into a Turn with zero mapping. They are not limited to official OpenAI: any service that declares compatibility with either protocol shape can use them.
The two shapes differ in how trustworthy a negative assertion such as notCalledTool is. Chat Completions does not promise that its response is a complete process—the application might run a whole tool loop server-side and return only the final answer—so a negative assertion means only “not seen,” not “did not happen.” The Responses protocol guarantees that its output array records everything the model decided to do this turn, including every function_call, so negative assertions are trustworthy. The Turn shapes output by the two converters are the same; this distinction only affects how you interpret negative assertions.

SDK event-stream converters: createClaudeSdkEventStream / createPiAgentEventStream / createCodexThreadEventStream

Each Agent SDK defines its own streaming protocol, rather than a private application format. niceeval/adapter provides converters from native frames to standard events. When writing a non-intrusive Adapter yourself, only provide the application-interface and approval-interface addresses. driveFrameStream, described in the next section, reads results frame by frame, so you do not write another for loop:
  • createClaudeSdkEventStream (SDKMessage): assistant text/tool_use blocks, user tool_result blocks, system / permission_denied (→ rejected), and result usage. markRejected() records a rejected invocation.
  • createPiAgentEventStream (pi-agent-core AgentEvent): message_end text/thinking/usage and paired tool_execution_start/end tools.
  • createCodexThreadEventStream (Codex SDK ThreadEvent): messages from agent_message / reasoning. Tool items are command_execution / mcp_tool_call / file_change / web_search, each mapped to paired tool operation.started + operation.finished. It includes turn.completed usage and error frames. For a waterfall, use native Codex CLI OTLP—[otel] in config.toml—to send spans to NiceEval, then normalize with official mapCodexSpans.
Runnable reference implementations are in examples/zh/tier1. The Claude SDK, pi SDK, and Codex SDK Adapters there each consist of a converter, driveFrameStream, and application-interface configuration.

General assembly helpers: driveFrameStream / deltaStream + ctx.session

Only three pieces of a hand-written send are truly independent: transport (how to send), reduction (raw data → events, handled by the converters above), and orchestration (session continuation plus HITL pause and resume). The third has nothing to do with a particular protocol; it is a pure control-flow pattern. For server history, use ctx.session.id / capture. For client history and paused HITL state, declare an Adapter-private typed slot with createSessionSlot<T>(), then use ctx.session.get / set / take to access it. See AgentSession in Adapter concepts. Only two official helpers actually reuse across protocols:
The three Tier 1 examples—claude-sdk and pi-sdk with driveFrameStream plus a typed slot, and codex-sdk with driveFrameStream only—have already been rewritten around these helpers. An Adapter is left only with transport and the decision in onFrame about whether a frame needs extra handling; it has no hand-written loop or module-level Map.

turnFromAiSdk: AI SDK result → event-stream converter

turnFromAiSdk, exported from niceeval/adapter, is for your own Adapter, such as direct server construction for an HTTP web Agent. See examples/zh/ai-sdk/. It maps AI SDK generateText / streamText results into { events, usage, status }. It pairs toolCallId exactly, preserves order, maps tool-error to a failed tool operation.finished, and aggregates usage across v4 / v5 / v7 field drift. Spread it directly into a Turn:
The converter supplies status: it is "waiting" when a tool approval awaits a person, with an input.requested event; otherwise it is "completed". Session continuation for multiple-turn resume, returning an HITL decision, and tracing depend on how you write your own send.

How to choose

  • For an AI SDK application with a useChat backend: use the built-in uiMessageStreamAgent for non-intrusive, zero-mapping integration, including HITL.
  • For another Agent system, such as an HTTP / gRPC service: integrate non-intrusively and write event mapping. The official converters cover most of it. The five examples under examples/zh/tier1 all take this route. If the application already has OTel, connect it too for an extra waterfall.
  • For a call waterfall—the trace in niceeval view: all three claude-code / codex / bub send OTLP themselves. claude-code uses CLI beta telemetry, whose spans have only structure and timing. A self-written Adapter only needs to declare tracing; see the OTel guide.
  • For human approval or multi-step confirmation (HITL): uiMessageStreamAgent provides it natively through AI SDK v7 tool approval. The three Sandbox Agents do not. For another system, write an Adapter and compose it with driveFrameStream plus a typed slot; Tier 1 pi-sdk / claude-sdk have ready patterns.
  • To run a coding Agent that changes code, inspect diff, and evaluate tool calls: use claude-code / codex / bub for send/receive, event stream, session continuation, workspace, and Sandbox.
  • When the backend protocol is your own and no official SDK converter applies, do not write send from zero. Choose a reduction shape: a small mapping for complete frames or deltaStream for token deltas. Use ctx.session.id / capture for server session continuation, a typed slot for client history or HITL state, and compose them into send.
  • Connect your Agent — how to implement each capability and which assertion it supports when writing an Adapter.
  • Sandbox Agent — run a built-in Sandbox Agent and write your own.
  • Adapter Reference — complete parameters for defineAgent / defineSandboxAgent.
  • OTel integration — send application spans to NiceEval to get the call waterfall in niceeval view.