Official adapters overview

​

Sandbox adapters

​

claude-code

• Auth: ANTHROPIC_API_KEY (overridable with the factory option apiKey), optional ANTHROPIC_BASE_URL (factory option baseUrl).
• Installing MCP servers: the mcpServers option, written during setup into the Sandbox’s user-level ~/.claude.json (the top-level mcpServers field). The two shapes are distinguished by field: a local stdio process sets command (optionally with args / env); a remote Streamable HTTP endpoint sets url (optionally with headers, passed verbatim into request headers, commonly used for Authorization), written as a { "type": "http", "url": …, "headers": … } entry. url must be reachable from inside the Sandbox: if the service runs on your own machine, expose it as a public address first with a tunnel like cloudflared or tailscale.
• Installing Skills: skills: SkillSpec[] — a local Skill ({ kind: "local", path }, a file or directory read from your project root) or a repo Skill ({ kind: "repo", source, ref, skills }, which can pin a commit/tag and enable only part of a multi-Skill repository). Skills land in the project-level .claude/skills/<name>/ inside the Sandbox, where the claude CLI discovers them natively (the adapter normalizes the native Skill tool call into a skill.loaded event, so it is not also counted as a tool call; assert it with t.loadedSkill(), not t.calledTool("Skill", ...)).
• Installing native plugins: plugins: ClaudeCodePluginSpec[]; each entry declares the marketplace connection (name / source / optional ref) and the plugin name inside it. That type belongs to claude-code only — it cannot be passed to codex.
• Official config file: settingsFile is a local project path on the machine running NiceEval, not a path inside the Sandbox; it points to a complete Claude Code settings.json. The path is resolved relative to the project root and must be a plain relative path or use a ./ prefix — .., absolute paths, ~, and symlinks that resolve outside the project root all error. The Adapter reads the file locally, uploads it, and uses it verbatim to replace the Sandbox’s otherwise-empty user-level ~/.claude/settings.json; it does not inherit the host machine’s config, and it is never deep-merged or re-serialized. model and env belong to the experiment and the Adapter — if either key appears in the file, setup errors and names the conflicting key. Keep secrets in environment variables, not in the config file.
• Post-install scripts: postSetup: SandboxHook[] runs your Hook functions in the Sandbox, in array order, after writing settings and installing MCP servers, Skills, and plugins is all done. A typical use is running a plugin’s own setup script (for example, one that needs to register a hook into a global config) — this kind of script can only run once everything it depends on is already installed. Its counterpart, preTeardown: SandboxHook[], runs in reverse order before the agent’s own teardown step, and only fires if execution has already reached the point where postSetup ran. A Hook that throws counts as an infrastructure error (the Attempt is recorded as errored), not as the agent failing the task.
• Tracing: the claude CLI’s beta native telemetry (CLAUDE_CODE_ENHANCED_TELEMETRY_BETA); spans carry only structure and timing — details in OTel Integration.

{
  "$schema": "https://json.schemastore.org/claude-code-settings.json",
  "permissions": { "deny": ["WebSearch", "WebFetch"] }
}

import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: claudeCodeAgent({
    mcpServers: [
      { name: "browser", command: "npx", args: ["-y", "@anthropic/mcp-browser"] },
    ],
    skills: [
      { kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2", skills: ["effect"] },
      { kind: "local", path: "skills/repository-guide.md" },
    ],
    plugins: [
      {
        marketplace: { name: "acme", source: "acme/claude-code-plugins", ref: "v1.3.0" },
        name: "safe-shell",
      },
    ],
    settingsFile: "configs/claude-code/no-web.json",
  }),
  model: "claude-sonnet-4-6",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});

​

codex

• Auth: CODEX_API_KEY (overridable with the factory option apiKey, not OPENAI_API_KEY), optional CODEX_BASE_URL (factory option baseUrl) for an OpenAI-compatible proxy.
• Installing MCP servers: the mcpServers option, appended during setup into the [mcp_servers.<name>] section of ~/.codex/config.toml. The two shapes are distinguished by field: a local stdio process sets command (optionally with args / env); a remote Streamable HTTP endpoint sets url (optionally with headers, written as a [mcp_servers.<name>.http_headers] subtable). url must be reachable from inside the Sandbox: if the service runs on your own machine, expose it as a public address first with a tunnel like cloudflared or tailscale.
  It is plural, mcp_servers: the singular [mcp_server.x] is silently ignored by the codex CLI — MCP simply never attaches, with no error. Check with codex mcp list.
• Installing Skills: skills: SkillSpec[], the same type as claude-code. Skills land in .agents/skills/<name>/, and the adapter also writes a discovery instruction into AGENTS.md — codex has no native Skill tool like claude-code, so merely installing the files does not make it read them. Asserting “was it used” can only look at whether it actually ran the shell command that reads that file — there is no tool call to assert directly.
• Installing native plugins: plugins: CodexPluginSpec[]; each entry declares the marketplace connection (name / source / optional ref / optional sparse) and the plugin name inside it. sparse is a list of paths (e.g. [".agents", "plugins/repo-map"]); each one adds a --sparse <path> to codex plugin marketplace add so large repos only fetch what the plugin needs, without changing what gets installed. That type belongs to codex only — it cannot be passed to claude-code.
• Official config file: configFile is a local project path on the machine running NiceEval, not a path inside the Sandbox; it points to a complete Codex config.toml. The path is resolved relative to the project root and must be a plain relative path or use a ./ prefix — .., absolute paths, ~, and symlinks that resolve outside the project root all error. The Adapter reads the file locally, uploads it, and uses it verbatim to replace the Sandbox’s otherwise-empty user-level ~/.codex/config.toml; it does not inherit the host machine’s config, and it is never concatenated, deep-merged, or parsed and rewritten. model, model_provider, model_providers, model_reasoning_effort, mcp_servers, and otel belong to the experiment and the Adapter — if any of these keys appear in the file, setup errors and names the conflicting key. Keep secrets in environment variables, not in the config file.
• Post-install scripts: postSetup: SandboxHook[] works the same as for claude-code: it runs your Hook functions in the Sandbox, in order, after every install step is done — a good place to run a plugin’s own setup script. Its counterpart, preTeardown: SandboxHook[], runs in reverse order before the agent’s teardown, and only fires if execution has already reached the point where postSetup ran. A hook that a script registers into codex’s global config does not need interactive trust confirmation — at run time, codex exec already bypasses the hook trust gate, so the hook takes effect directly.
• Tracing: built in, configured through the [otel.trace_exporter.otlp-http] section of config.toml, protocol http/json.

#:schema https://developers.openai.com/codex/config-schema.json
web_search = "disabled"

import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: codexAgent({
    mcpServers: [
      { name: "browser", command: "npx", args: ["-y", "@anthropic/mcp-browser"] },
      { name: "team-memory", url: "https://mem.example.com/mcp/", headers: { Authorization: `Bearer ${process.env.MEM_API_KEY}` } },
    ],
    skills: [{ kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2", skills: ["effect"] }],
    plugins: [
      {
        marketplace: { name: "acme", source: "acme/codex-plugins", ref: "8f3c1a2" },
        name: "repo-map",
      },
    ],
    configFile: "configs/codex/no-web.toml",
  }),
  model: "gpt-5.4",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});

​

bub

• Auth: BUB_API_KEY + BUB_API_BASE (an OpenAI-compatible proxy), overridable with the factory options apiKey / apiBase.
• Installing Skills: skills: SkillSpec[], the same type as the other two adapters. Skills land in .agents/skills/<name>/, with a discovery instruction written into AGENTS.md.
• Installing plugins: pythonPlugins: PythonPluginSpec[] ({ package }: a PyPI package, a version specifier, or a git URL), added during setup to uv tool install … --with <package>. That type belongs to bub only; the package set is part of the install checkpoint key, so two variants with different plugins never reuse the same install cache.
• Prebuilt Bub: NiceEval’s E2B recipe folds Bub, the OTel plugin, and the set of Python plugins into an install fingerprint. The Adapter only reuses an environment whose fingerprint matches exactly; simply having a bub on PATH is not proof of compatibility. For where to build one, see Sandbox providers · Build on the official baselines to speed things up.
• bub has no mcpServers — MCP belongs to the adapters that support it, and the field simply does not exist on this config.
• Post-install scripts: postSetup: SandboxHook[] works the same as for the other two adapters: it runs your Hook functions in the Sandbox, in order, after every install step is done. Its counterpart, preTeardown: SandboxHook[], runs in reverse order before the agent’s teardown, and only fires if execution has already reached the point where postSetup ran.
• Install method: via uv tool install (a PyPI package, not an npm package); the first install builds a checkpoint cache to speed up later Sandboxes.
• Tracing: built in, injected via environment variables, protocol http/protobuf.

import { defineExperiment } from "niceeval";
import { bubAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";

export default defineExperiment({
  agent: bubAgent({
    skills: [{ kind: "local", path: "skills/repository-guide.md" }],
    pythonPlugins: [{ package: "bub-plugin-memory==1.3.0" }],
  }),
  model: "gpt-5.4",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});

​

Comparing the three Sandbox adapters

​

Non-Sandbox adapters

• uiMessageStreamAgent: built in, connects non-intrusively to an AI SDK useChat backend’s HTTP endpoint, zero mapping (HITL included).
  import { uiMessageStreamAgent } from "niceeval/adapter";
  
  export default uiMessageStreamAgent({
    name: "my-assistant",
    url: "http://localhost:3000/api/chat",
    body: (ctx) => ({ model: ctx.model }),
  });
  
• SDK event stream converters (createClaudeSdkEventStream / createPiAgentEventStream / createCodexThreadEventStream): when hand-writing an adapter that connects to a service running the Claude Agent SDK / pi-agent-core / Codex SDK, the mapping from native frames to standard events is already done officially — you only need to write the transport glue, paired with driveFrameStream to drive it frame by frame.
• turnFromAiSdk: a converter from AI SDK generateText / streamText results to the standard event stream, used when writing your own HTTP web-agent adapter.

​

How to choose

• The system under test is a coding agent that must edit code and run commands on a real filesystem: claude-code / codex / bub — see the Sandbox adapters section above for configuration.
• The system under test is an AI SDK application (a useChat backend): uiMessageStreamAgent, zero mapping and HITL included.
• The system under test is some other already-deployed agent system (HTTP / gRPC): go non-Sandbox and hand-write the adapter — the official SDK converters cover most of the mapping work.

​

Related reading

• Connect Your Agent — the full picture: how to configure an experiment and write an eval.
• Sandbox Agent — how to run the built-in Sandbox agents, and how to write your own.
• Built-in agent capabilities reference — a capability-by-capability inventory of each adapter, plus complete example code for the non-Sandbox adapters.
• OTel Integration — send spans to NiceEval in exchange for the call waterfall in niceeval view.
• defineAgent reference — the full parameters of defineAgent / defineSandboxAgent.