claude-code, codex, and bub—do not all provide the same capabilities. There is no capability-declaration layer: construction and actual behavior prove them. See the Capabilities reference. This page inventories what each built-in Agent provides, plus two built-ins for AI SDK applications: the non-intrusive HTTP Adapter uiMessageStreamAgent, including HITL, and the result converter turnFromAiSdk.
Capability overview
What each built-in Agent provides
All three use
defineSandboxAgent (Agent.kind is always "sandbox"), so filesystem assertions and file I/O and command execution—including t.sandbox.fileChanged() and diff assertions—work with all three Agents regardless of the table.
Details by Agent
claude-code
- Connection: spawns
claude --print --dangerously-skip-permissionsin the Sandbox and reads the newest transcript at~/.claude/projects/**/*.jsonl. - Session continuation: when
ctx.session.idhas a value, appends--resume <id>; the session ID decoded from the transcript is written back withctx.session.capture(). - Authentication:
ANTHROPIC_API_KEY, optionallyANTHROPIC_BASE_URL. SeeClaudeCodeConfigbelow for options. tracinguses the claude CLI’s native beta OTLP trace spans overhttp/protobuf. SetCLAUDE_CODE_ENABLE_TELEMETRYandCLAUDE_CODE_ENHANCED_TELEMETRY_BETAin the environment—the explicit beta opt-in—to give the endpoint to the CLI. Trace decoding displays an interaction / llm_request / tool hierarchy waterfall.
codex
- Connection: runs
codex exec --jsonin the Sandbox, orcodex exec resume <id> --jsonfor continuation, and uses stdout JSONL as the transcript. The command includes--dangerously-bypass-approvals-and-sandboxand--dangerously-bypass-hook-trust: nobody can answer Codex’s interactive confirmations in a Sandbox, so a Hook installed by a Plugin orpostSetupcan take effect without interactive trust. - Authentication:
CODEX_API_KEY, and optionallyCODEX_BASE_URLfor an OpenAI-compatible proxy. SeeCodexConfigbelow for options. tracingis configured through[otel.trace_exporter.otlp-http]in~/.codex/config.toml, overhttp/json.
bub
- Connection: runs
bub runwith--session-idin the Sandbox and reads the tape at~/.bub/tapes/<hash>.jsonlas the transcript. - Authentication:
BUB_API_KEYplusBUB_API_BASEfor an OpenAI-compatible proxy. SeeBubConfigbelow for options. tracingis injected through environment variables such asOTEL_EXPORTER_OTLP_TRACES_ENDPOINT, overhttp/protobuf.- Installation uses
uv tool install, not an npm package. The first install creates a checkpoint cache to speed later Sandboxes.
Configuration for the three built-in Sandbox Agents
ClaudeCodeConfig
apiKey
ANTHROPIC_API_KEY from the environment.
baseUrl
ANTHROPIC_BASE_URL from the environment. When neither exists, use the Anthropic official endpoint, the claude CLI default behavior.
env
maxTurns
--max-turns. It limits Eval cost; when omitted, the CLI native default is unlimited.
mcpServers
~/.claude.json during each Sandbox setup. The stdio form writes command, optionally with args / env. The Streamable HTTP form writes url, optionally with headers, copied verbatim into request headers, as a { "type": "http", "url": …, "headers": … } entry.
skills
.claude/skills/<name>/, where the claude CLI discovers them natively.
plugins
settingsFile
settings.json in the local project. It resolves relative to the project root that runs niceeval—the directory containing niceeval.config.ts—not inside the Sandbox. It accepts only a relative path within the project root. A path containing .., an absolute path, a ~ path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim as the otherwise-empty user-level ~/.claude/settings.json in the Sandbox. It does not inherit host configuration, concatenate, or reserialize. Reserved keys model and env cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.
postSetup
attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
preTeardown
postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
CodexConfig
apiKey
CODEX_API_KEY env var if omitted.
baseUrl
https://s2a.example.com/v1). Falls back to the CODEX_BASE_URL env var if omitted.
env
codex exec and continuing codex exec resume use the same declaration; lifecycle Hooks started by Codex, dynamic MCP headers, and command subprocesses all inherit it. Values enter only through Sandbox command options: they are not concatenated into shell text or written to the setup manifest, and are all redacted as potential sensitive values from timing, execution, and error evidence. CODEX_API_KEY still comes from apiKey or the same host environment variable; the Adapter’s authentication value overrides a same-named key here.
An environment value does not enter carry identity. A non-sensitive value that changes behavior must also be declared in Experiment Flags or the owning Plugin identity; rotating credentials alone does not invalidate old results. PATH is Sandbox-managed and cannot be declared here. Its presence errors at factory construction; use the Sandbox factory’s pathPrepend instead. See docs/feature/sandbox/library.md.
mcpServers
~/.codex/config.toml during each Sandbox setup. The stdio form (command / args / env) writes a command line in [mcp_servers.<name>]. The Streamable HTTP form (url / headers) writes the url line and headers in the [mcp_servers.<name>.http_headers] subtable.
skills
.agents/skills/<name>/, and a discovery instruction is written into AGENTS.md. Codex has no native Skill tool like Claude Code, so installing the file alone does not make it read it. See memory/codex-no-native-skill-tool.md.
plugins
configFile
config.toml in the local project. It resolves relative to the project root that runs niceeval—the directory containing niceeval.config.ts—not inside the Sandbox. It accepts only a relative path within the project root. A path containing .., an absolute path, a ~ path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim into the otherwise-empty user-level ~/.codex/config.toml in the Sandbox. It does not inherit host configuration or parse and rewrite it. Reserved keys model, model_provider, model_providers, model_reasoning_effort, mcp_servers, and otel cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.
postSetup
attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
preTeardown
postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
BubConfig
apiKey
BUB_API_KEY env var if omitted.
apiBase
BUB_API_BASE env var if omitted.
skills
.agents/skills/<name>/, and a discovery instruction is written into AGENTS.md. Bub has no native Skill-loading mechanism.
version
"0.4.0". When omitted, use the default version pinned by NiceEval. It is always a deterministic version and never latest: the system under test’s version must be visible from Experiment configuration.
otelPlugin
bub.tape and requires Bub ≥ 0.3.10. Earlier Plugin commits validate against republic types and pair with Bub ≤ 0.3.9. A mismatched generation does not fail installation; it rejects every span and leaves the timeline silently empty. When pinning version backward, pin the matching Plugin commit in the same change.
pythonPlugins
uv tool install … --with <pkg>. The normalized package list enters the installation checkpoint key, so two Agent variants with different Plugin sets never reuse the same checkpoint; otherwise the second could silently receive the first variant’s environment.
postSetup
attempt errored). See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
preTeardown
postSetup. It uses postSetup’s reverse order and runs before the Agent’s own teardown step: a LIFO mirror—postSetup runs after Agent installation, while preTeardown runs before Agent cleanup. It runs only if execution reached the postSetup point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in docs/feature/adapters/library/coding-agent-extensions.md.
uiMessageStreamAgent: built-in non-intrusive Adapter for AI SDK applications, including HITL
uiMessageStreamAgent, exported from niceeval/adapter, sends and receives non-intrusively against an HTTP endpoint implementing the UI Message Stream protocol, the standard SSE for an AI SDK useChat backend. It only uses fetch; it does not import application code, so it can connect wherever the application is deployed:
- Send/receive and event stream: SSE frames are reduced through the framework-agnostic
readUIMessageStreamreducer from the officialaipackage—the same reduceruseChatuses internally. Tool calls, results, and message text are constructed directly from message parts. The application does not need OTel. - Session continuation: the protocol is stateless on the server and the client carries complete history. The factory stores complete
UIMessage[]history in an Adapter-private typed slot and replays it verbatim each turn. A new session line aftert.newSession()has not written that slot yet. - HITL: AI SDK v7 tool approval—a tool with
needsApproval: true—maps natively. When a part stops atapproval-requested, that Turn hasstatus: "waiting"plusinput.requested.t.respond("approve" / "deny")translates toapproval-responded, rewrites that part in place, and resendsmessagesto continue server execution. That is the same protocol behavior as a real frontend’saddToolApprovalResponse()plussendMessage(). There is no separate approval endpoint. - Rejection: an invocation enters the event stream with
rejectedand, by default, a “do not retry” reason thatdenyReasoncan override. Without it, models often repeat the exact call. - Usage and waterfall: UI Message Stream frames have no usage, so usage assertions such as
t.maxTokenshave no data by default on this built-in. An application that puts usage in message metadata is adding its own protocol extension. The waterfall is separate: when the application has OTel instrumentation, such as official@ai-sdk/otel, send spans to NiceEval according to OTel integration andniceeval viewhas the complete waterfall. Spans go only to the waterfall, not assertions.
ai in the Eval project. It is an optional peer dependency because the protocol reducer comes from it. For a complete runnable example, see examples/zh/tier1/ai-sdk-v7.
Complete uiMessageStreamAgent(options) parameters (UiMessageStreamAgentOptions):
name
"ui-message-stream".
url
headers
ctx.telemetry.headers for traceparent always merges in automatically.
body
messages, such as (ctx) => ({ model: ctx.model }). Fields whose value is undefined are automatically omitted during serialization.
projectToolCommand
undefined and leaves actions coverage partial. NiceEval does not guess from a name or input.
denyReason
approval-responded when approval is denied. The application and SDK make it tool-result text that the model sees. A clear “do not retry” message measurably lowers the chance that a model repeats the same call.
settleMs
tracing
spanMapper
OpenAI-compatible response converters: turnFromChatCompletion / turnFromResponses
turnFromChatCompletion(res) and turnFromResponses(res), exported from niceeval/adapter, convert OpenAI’s two response shapes—Chat Completions and Responses—into a Turn with zero mapping. They are not limited to official OpenAI: any service that declares compatibility with either protocol shape can use them.
notCalledTool is. Chat Completions does not promise that its response is a complete process—the application might run a whole tool loop server-side and return only the final answer—so a negative assertion means only “not seen,” not “did not happen.” The Responses protocol guarantees that its output array records everything the model decided to do this turn, including every function_call, so negative assertions are trustworthy. The Turn shapes output by the two converters are the same; this distinction only affects how you interpret negative assertions.
SDK event-stream converters: createClaudeSdkEventStream / createPiAgentEventStream / createCodexThreadEventStream
Each Agent SDK defines its own streaming protocol, rather than a private application format. niceeval/adapter provides converters from native frames to standard events. When writing a non-intrusive Adapter yourself, only provide the application-interface and approval-interface addresses. driveFrameStream, described in the next section, reads results frame by frame, so you do not write another for loop:
createClaudeSdkEventStream(SDKMessage):assistanttext/tool_use blocks,usertool_result blocks,system/permission_denied(→rejected), andresultusage.markRejected()records a rejected invocation.createPiAgentEventStream(pi-agent-coreAgentEvent):message_endtext/thinking/usage and pairedtool_execution_start/endtools.createCodexThreadEventStream(Codex SDKThreadEvent): messages fromagent_message/reasoning. Tool items arecommand_execution/mcp_tool_call/file_change/web_search, each mapped to paired tooloperation.started+operation.finished. It includesturn.completedusage and error frames. For a waterfall, use native Codex CLI OTLP—[otel]in config.toml—to send spans to NiceEval, then normalize with officialmapCodexSpans.
examples/zh/tier1. The Claude SDK, pi SDK, and Codex SDK Adapters there each consist of a converter, driveFrameStream, and application-interface configuration.
General assembly helpers: driveFrameStream / deltaStream + ctx.session
Only three pieces of a hand-written send are truly independent: transport (how to send), reduction (raw data → events, handled by the converters above), and orchestration (session continuation plus HITL pause and resume). The third has nothing to do with a particular protocol; it is a pure control-flow pattern. For server history, use ctx.session.id / capture. For client history and paused HITL state, declare an Adapter-private typed slot with createSessionSlot<T>(), then use ctx.session.get / set / take to access it. See AgentSession in Adapter concepts. Only two official helpers actually reuse across protocols:
driveFrameStream plus a typed slot, and codex-sdk with driveFrameStream only—have already been rewritten around these helpers. An Adapter is left only with transport and the decision in onFrame about whether a frame needs extra handling; it has no hand-written loop or module-level Map.
turnFromAiSdk: AI SDK result → event-stream converter
turnFromAiSdk, exported from niceeval/adapter, is for your own Adapter, such as direct server construction for an HTTP web Agent. See examples/zh/ai-sdk/. It maps AI SDK generateText / streamText results into { events, usage, status }. It pairs toolCallId exactly, preserves order, maps tool-error to a failed tool operation.finished, and aggregates usage across v4 / v5 / v7 field drift. Spread it directly into a Turn:
status: it is "waiting" when a tool approval awaits a person, with an input.requested event; otherwise it is "completed". Session continuation for multiple-turn resume, returning an HITL decision, and tracing depend on how you write your own send.
How to choose
- For an AI SDK application with a
useChatbackend: use the built-inuiMessageStreamAgentfor non-intrusive, zero-mapping integration, including HITL. - For another Agent system, such as an HTTP / gRPC service: integrate non-intrusively and write event mapping. The official converters cover most of it. The five examples under
examples/zh/tier1all take this route. If the application already has OTel, connect it too for an extra waterfall. - For a call waterfall—the trace in
niceeval view: all threeclaude-code/codex/bubsend OTLP themselves. claude-code uses CLI beta telemetry, whose spans have only structure and timing. A self-written Adapter only needs to declaretracing; see the OTel guide. - For human approval or multi-step confirmation (HITL):
uiMessageStreamAgentprovides it natively through AI SDK v7 tool approval. The three Sandbox Agents do not. For another system, write an Adapter and compose it withdriveFrameStreamplus a typed slot; Tier 1 pi-sdk / claude-sdk have ready patterns. - To run a coding Agent that changes code, inspect diff, and evaluate tool calls: use
claude-code/codex/bubfor send/receive, event stream, session continuation, workspace, and Sandbox. - When the backend protocol is your own and no official SDK converter applies, do not write send from zero. Choose a reduction shape: a small mapping for complete frames or
deltaStreamfor token deltas. Usectx.session.id/capturefor server session continuation, a typed slot for client history or HITL state, and compose them into send.
Related reading
- Connect your Agent — how to implement each capability and which assertion it supports when writing an Adapter.
- Sandbox Agent — run a built-in Sandbox Agent and write your own.
- Adapter Reference — complete parameters for
defineAgent/defineSandboxAgent. - OTel integration — send application spans to NiceEval to get the call waterfall in
niceeval view.