claude-code, codex, bub) do not implement exactly the same capabilities — capabilities have no declaration layer and are proven entirely by construction and actual behavior, see the capabilities reference. This page inventories what each built-in agent does per capability, plus two built-ins for connecting AI SDK apps: the non-invasive HTTP adapter uiMessageStreamAgent (with HITL) and the result converter turnFromAiSdk.
Capability overview
What each built-in agent implements
All three are constructed with
defineSandboxAgent (Agent.kind is always "sandbox"), so t.sandbox.fileChanged() / diff assertions, file IO, and command execution work on all three agents regardless of this table.
Per-agent details
claude-code
- Connection: spawns
claude --print --dangerously-skip-permissionsin the Sandbox and reads back the latest transcript from~/.claude/projects/**/*.jsonl. - Session continuation: appends
--resume <id>whenctx.session.idhas a value; the session id parsed from the transcript is written back viactx.session.capture(). - Auth:
ANTHROPIC_API_KEY, optionalANTHROPIC_BASE_URL; configuration options are listed underClaudeCodeConfigbelow. tracingis configured via the claude CLI’s native OTLP trace spans (beta), protocolhttp/protobuf: settingCLAUDE_CODE_ENABLE_TELEMETRYandCLAUDE_CODE_ENHANCED_TELEMETRY_BETA(a beta toggle, must be explicitly enabled) in env hands the endpoint to the CLI; trace decoding shows a waterfall at the interaction / llm_request / tool level.
codex
- Connection: runs
codex exec --jsonin the Sandbox (codex exec resume <id> --jsonwhen continuing), using stdout JSONL as the transcript. The command carries--dangerously-bypass-approvals-and-sandboxand--dangerously-bypass-hook-trust: nobody in the Sandbox can answer codex’s interactive approval prompts, so hooks installed by a plugin orpostSetuptake effect without needing interactive trust confirmation. - Auth:
CODEX_API_KEY(notOPENAI_API_KEY), optionalCODEX_BASE_URLfor OpenAI-compatible proxies; configuration options are listed underCodexConfigbelow. tracingis configured through the[otel.trace_exporter.otlp-http]section of~/.codex/config.toml, protocolhttp/json.
bub
- Connection: runs
bub run+--session-idin the Sandbox and reads the tape from~/.bub/tapes/<hash>.jsonlas the transcript. - Auth:
BUB_API_KEY+BUB_API_BASE(OpenAI-compatible proxy); configuration options are listed underBubConfigbelow. tracingis injected via environment variables (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, etc.), protocolhttp/protobuf.- Installed via
uv tool install(not an npm package); the first install builds a checkpoint cache to speed up later Sandboxes.
Configuration options for the three built-in sandbox agents
ClaudeCodeConfig
apiKey
ANTHROPIC_API_KEY env var if omitted.
baseUrl
ANTHROPIC_BASE_URL env var if omitted;
if neither is set, uses the Anthropic official endpoint (the claude CLI’s default behavior).
maxTurns
--max-turns).
Caps the eval’s cost ceiling; if omitted, uses the CLI’s native default (unlimited).
mcpServers
~/.claude.json on each Sandbox setup).
The stdio form writes command (optionally with args / env); the Streamable HTTP form writes url (optionally with headers,
carried through verbatim into the request headers), landing as a { “type”: “http”, “url”: …, “headers”: … } entry.
skills
.claude/skills/<name>/, where the claude CLI discovers them natively.
plugins
settingsFile
settings.json (official format) in the local project — resolved relative to the project
root running niceeval (the directory containing niceeval.config.ts), not a path inside the Sandbox; only relative
paths within the project root are accepted — paths containing .., absolute paths, ~ paths, and symlinks that
resolve outside the project root all error during setup. The raw bytes are uploaded verbatim as the user-level
~/.claude/settings.json, which is otherwise empty in the Sandbox (no inheriting host config, no merging, no
re-serializing); the reserved keys model and env appearing in the file error during setup. The manifest records
only the project-relative path and the byte SHA-256, never the file body.
postSetup
preTeardown
postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown
step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and
fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded
into a teardown-failed diagnostic by the teardown phase.
See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.
CodexConfig
apiKey
CODEX_API_KEY env var if omitted.
baseUrl
https://s2a.example.com/v1). Falls back to the CODEX_BASE_URL env var if omitted.
mcpServers
~/.codex/config.toml on each Sandbox setup).
The stdio form (command/args/env) writes the command line under [mcp_servers.<name>];
the Streamable HTTP form (url/headers) writes the url line, with headers going into the
[mcp_servers.<name>.http_headers] subtable.
skills
.agents/skills/<name>/, together with a discovery instruction written into AGENTS.md — codex has no native Skill tool like Claude Code, so merely installing the files does not make it read them (see memory/codex-no-native-skill-tool.md).
plugins
configFile
config.toml (official TOML format) in the local project — resolved relative to the project
root running niceeval (the directory containing niceeval.config.ts), not a path inside the Sandbox; only relative
paths within the project root are accepted — paths containing .., absolute paths, ~ paths, and symlinks that
resolve outside the project root all error during setup. The raw bytes are merged verbatim into the user-level
~/.codex/config.toml, which is otherwise empty in the Sandbox (no inheriting host config, no parse-and-rewrite);
the reserved keys model, model_provider, model_providers, model_reasoning_effort, mcp_servers, and otel
appearing in the file error during setup. The manifest records only the project-relative path and the byte SHA-256,
never the file body.
postSetup
preTeardown
postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown
step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and
fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded
into a teardown-failed diagnostic by the teardown phase.
See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.
BubConfig
apiKey
BUB_API_KEY env var if omitted.
apiBase
BUB_API_BASE env var if omitted.
skills
.agents/skills/<name>/, with a discovery instruction written into AGENTS.md (bub has no native Skill loading mechanism).
pythonPlugins
uv tool install … --with <pkg>.
The normalized package list is part of the install checkpoint key: two agent variants with different plugin sets never reuse the same
install checkpoint (otherwise the second variant would silently inherit the first one’s environment).
postSetup
preTeardown
postSetup: runs in reverse of postSetup’s ordering, before the agent’s own teardown
step (a LIFO mirror — postSetup runs after the agent installs, preTeardown runs before the agent tears down), and
fires if and only if postSetup’s point in time was reached. A thrown error counts as an infrastructure error, folded
into a teardown-failed diagnostic by the teardown phase.
See “Running scripts after install” in docs/feature/adapters/library/coding-agent-extensions.md.
uiMessageStreamAgent: built-in non-invasive adapter for AI SDK apps (with HITL)
uiMessageStreamAgent (exported from niceeval/adapter) talks non-invasively to an HTTP endpoint speaking the UI Message Stream protocol (the standard SSE of AI SDK useChat backends) — it only fetches, never imports app code, so it connects wherever the app is deployed:
- Send/receive + event stream: SSE frames are reduced by the
aipackage’s official framework-agnostic reducerreadUIMessageStream(the same one insideuseChat); tool calls/results/message text are built directly from message parts — the app is not required to wire up OTel. - Session continuation: the protocol is server-stateless, “client carries the full history” — the factory stores the whole
UIMessage[]in an adapter-private typed session slot and replays it verbatim each turn; a new conversation line (aftert.newSession()) gets an empty history. - HITL: AI SDK v7 tool approval (tools with
needsApproval: true) maps natively — when a part stops atapproval-requested, the whole turn isstatus: "waiting"+input.requested;t.respond("approve" / "deny")is translated intoapproval-responded, rewriting that part in place and resendingmessagesverbatim to trigger the server to continue (identical protocol behavior to a real frontend’saddToolApprovalResponse()+sendMessage(); there is no separate approve endpoint). Denied calls land in the event stream asrejected, and by default carry a “do not retry” reason (denyReasoncan override) — without one, models often resend the very same call. - usage / waterfall: UI Message Stream protocol frames carry no usage, so usage assertions like
t.maxTokenshave no data by default on this built-in (an app putting usage into message metadata is the app’s own protocol extension). The waterfall is a separate matter: when the app has OTel instrumentation (e.g. the official@ai-sdk/otel), send spans to NiceEval per OTel integration andniceeval viewgets a full waterfall — spans only feed the waterfall, never assertions.
ai installed in the eval project (optional peer dependency; the protocol reducer comes from it). Full runnable example: examples/zh/tier1/ai-sdk-v7.
Full parameters of uiMessageStreamAgent(options) (UiMessageStreamAgentOptions):
name
url
headers
ctx.telemetry.headers (traceparent) is always merged in automatically.
body
messages, e.g. (ctx) => ({ model: ctx.model }) (undefined fields are dropped automatically during serialization).
denyReason
approval-responded when an approval is denied. The app/SDK passes it to the model as the tool result
text — spelling out “do not retry” measurably lowers the chance the model resends the very same call (observed empirically).
settleMs
tracing
spanMapper
OpenAI-compatible response converters: turnFromChatCompletion / turnFromResponses
turnFromChatCompletion(res) / turnFromResponses(res) (exported from niceeval/adapter) map OpenAI’s two response shapes — Chat Completions and Responses — from the whole response into a Turn with zero manual mapping: this is not limited to the official OpenAI API — any service that claims compatibility with either protocol shape works.
notCalledTool and friends) are: Chat Completions does not guarantee “response = complete process” (the app may run the whole tool loop server-side and only hand you the final answer), so a negative assertion can only mean “not observed,” not “did not happen”; the Responses protocol’s contract guarantees the output array records everything the model decided to do this turn (including every function_call), so negative assertions there are trustworthy. The Turn shape the two converters produce is identical — this difference only affects how you interpret negative assertions.
SDK event stream converters: createClaudeSdkEventStream / createPiAgentEventStream / createCodexThreadEventStream
Each agent SDK’s streaming protocol is a generic protocol defined by the SDK, not some app’s private format — the native-frame-to-standard-event mapping knowledge lives in the official package (exported from niceeval/adapter), so when writing your own non-invasive adapter only the transport glue remains (which endpoint the app puts the stream on, which endpoint approvals go through). Driving the loop frame by frame is itself an official piece (driveFrameStream, see the next section), so no hand-written for loop:
createClaudeSdkEventStream(Claude Agent SDKSDKMessage):assistanttext/tool_use blocks,usertool_result blocks,system/permission_denied(→rejected),resultusage;markRejected()registers denied calls.createPiAgentEventStream(pi-agent-coreAgentEvent): text/thinking/usage frommessage_end, tool pairs fromtool_execution_start/end.createCodexThreadEventStream(Codex SDKThreadEvent): message-class frames fromagent_message/reasoning, tool items (command_execution/mcp_tool_call/file_change/web_search→ paired tooloperation.started+operation.finished), usage fromturn.completed, error frames. The waterfall needs the codex CLI’s native OTLP (the[otel]block in config.toml) sending spans to NiceEval, normalized with the officialmapCodexSpans.
examples/zh/tier1 are exactly these three converters + driveFrameStream + transport glue.
Generic “assembly” pieces: driveFrameStream / deltaStream + ctx.session
Inside a hand-written send, only three truly independent parts exist: transport (how to send), reduce (raw data → events, handled by the converters above), and orchestration (session continuation + HITL pause/resume). The third part is protocol-agnostic, pure control-flow pattern — session continuation (id/capture) and adapter-private typed slots hang directly off ctx.session (AgentSession, see Adapter concept). Create a slot once with createSessionSlot<T>(), then use set(slot, value) and take(slot) for a paused scene. What actually remains reusable across protocols is just two official pieces:
driveFrameStream plus typed session slots; codex-sdk uses only driveFrameStream) are built on these pieces — the adapters are down to transport plus the “does this frame need extra handling” decision in onFrame, with no hand-written loops or module-level Maps.
turnFromAiSdk: AI SDK result → event stream converter
turnFromAiSdk (exported from niceeval/adapter) is used inside adapters you write yourself (e.g. server-side direct construction for an HTTP web agent, see examples/zh/ai-sdk/). It maps an AI SDK generateText / streamText result into { events, usage, status } — exact toolCallId pairing, order-faithful, tool-error mapped to a failed tool operation.finished, usage aggregation (v4 / v5 / v7 field drift all covered) — laid straight into the Turn:
status comes from the converter: "waiting" when there is a tool approval pending human review (with an input.requested event attached), otherwise "completed". Session continuation (multi-turn resume), handing HITL verdicts back, and tracing depend on how you write your own send.
How to choose
- The system under test is an AI SDK app (
useChatbackend): the built-inuiMessageStreamAgentconnects non-invasively with zero mapping (including HITL). - The system under test is another agent system (HTTP / gRPC service): connect non-invasively and hand-write the event mapping (official converters cover most of it) — all five examples under
examples/zh/tier1take this path; if the app already has OTel instrumentation, wire it up for a bonus waterfall. - You need the call waterfall (the trace in
niceeval view): all three ofclaude-code/codex/bubemit OTLP themselves (claude-code via the CLI’s beta telemetry; its spans carry structure and timing only); adapters you write yourself just declaretracing(see the OTel guide). - You need human approval / multi-step confirmation (HITL):
uiMessageStreamAgentsupports it natively (AI SDK v7 tool approval); none of the three sandbox agents do; for other systems write your own adapter, assembled fromdriveFrameStreamplus typed session slots (the pi-sdk / claude-sdk Tier 1 examples have ready-made patterns). - You want to run a coding agent that edits code, inspect diffs, and judge tool calls:
claude-code/codex/bub(send/receive + event stream + session continuation + workspace + sandbox). - Your backend protocol is home-grown with no official SDK converter: do not hand-write send from scratch — pick a reduce shape (a small mapping for whole-chunk landing,
deltaStreamfor token-by-token deltas), usectx.session.id/capturefor session continuation, add a typed session slot if you have HITL, and the assembly is your send.
Related reading
- Connect your agent — how to implement each capability in your own adapter and which assertions it maps to.
- Sandbox Agent — how to run the built-in sandbox agents and write your own.
- defineAgent reference — full parameters of
defineAgent/defineSandboxAgent. - OTel integration — send your app’s spans to NiceEval in exchange for the call waterfall in
niceeval view.