> ## Documentation Index > Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # defineAgent and defineSandboxAgent: adapter reference > Reference for defineAgent and defineSandboxAgent: AgentContext, AgentSession, the Sandbox interface, and shared sandbox helpers. Every [NiceEval](https://niceeval.com/) agent is an adapter: code that knows how to drive a specific backend and translate its output into the standard event stream. The runner only calls `agent.send(input, ctx)`. Full round trip of one t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter returns a Turn with standard events. ## `defineAgent` Use this for direct agent integrations: ```ts theme={null} import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter"; ``` `defineDirectAgent` remains available as a deprecated compatibility alias. It references the same function; use `defineAgent` in new code. `Agent.kind` produced by `defineAgent` is always `"direct"` (an internal discriminant field you never declare). There are no capability bits on `t` that need declaring to unlock — filesystem assertions like `t.sandbox` exist only on agents constructed with `defineSandboxAgent` (`kind: "sandbox"`); everything else is judged by the events `send` actually returned and whether `ctx.session` was used. See the [capabilities reference](/docs/reference/capabilities). The full field set is in "Agent and Adapter Context fields" below. ### Direct agent example ```ts theme={null} export default defineAgent({ name: "echo", evidenceCoverage: completeEvidenceCoverage, async send(input) { return { status: "completed", events: [{ type: "message", role: "assistant", text: input.text }], }; }, }); ``` ## `input` (`TurnInput`) Text passed by the current `t.send(...)`. Files attached to this turn (images, etc.). Adapters that do not support multimodal input can ignore it. Present only on answer turns (`t.respond` / `t.respondAll`): per-request structured answers, matched by `requestId`. ## `defineSandboxAgent` Use this for coding agent CLIs. The produced `Agent.kind` is always `"sandbox"` — filesystem assertions like `t.sandbox` and `t.sandbox.fileChanged()` unlock only on this kind of agent. The full field set is likewise in "Agent and Adapter Context fields" below, under `SandboxAgentDef`; the complete methods of `ctx.sandbox` (the `Sandbox` interface) are in the "Sandbox interface" section further below. ```ts theme={null} import { completeEvidenceCoverage, defineSandboxAgent } from "niceeval/adapter"; import { command } from "niceeval/sandbox"; export default defineSandboxAgent({ name: "my-cli-agent", evidenceCoverage: completeEvidenceCoverage, ensure: { identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" }, probe: command("my-agent", ["--version"]), }, async send(input, ctx) { await ctx.sandbox.runCommand("my-agent", ["run", input.text]); return { status: "completed", events: [] }; }, }); ``` ## Registration ```ts theme={null} import { defineExperiment } from "niceeval"; import echo from "./agents/echo"; export default defineExperiment({ agent: echo, attempts: 1, }); ``` ## Agent and Adapter Context fields Construction parameters for `defineAgent` (`DirectAgentDef`) and `defineSandboxAgent` (`SandboxAgentDef`), plus the `ctx` (`AgentContext`) that both receive in `send(input, ctx)`: ### `DirectAgentDef` #### `name` ```ts theme={null} name: string; ``` The agent's display name/identifier, passed as-is into `Agent.name` — not a registry lookup key, only used for display, result attribution, and dedup fingerprinting. #### `evidenceCoverage` ```ts theme={null} evidenceCoverage: EvidenceCoverage; ``` This adapter's steady-state evidence coverage declaration. Use `completeEvidenceCoverage` only when all six evidence channels are fully collected. #### `setup` ```ts theme={null} setup?: DirectAgentSetup; ``` Runs once per attempt. A Direct Agent receives no Sandbox; use this for one-time work such as establishing a connection or authentication. #### `tracing` ```ts theme={null} tracing?: Omit; ``` OTLP export configuration: how the remote system under test sends traces to the endpoint (env-based injection / file-based configuration). #### `spanMapper` ```ts theme={null} spanMapper?: SpanMapper; ``` Thin mapper from native span → canonical; omit it to fall back to the generic heuristic. Only affects the waterfall. #### `send` ```ts theme={null} send(input: TurnInput, ctx: AgentContext): Promise; ``` Once per turn: send one turn's prompt to the remote system under test (HTTP/SDK, etc.), and parse the response into events. #### `classifySendFailure` ```ts theme={null} classifySendFailure?: SendFailureClassifier; ``` Optional classifier for rejected `send` calls. It classifies structured execution failures; a trusted failed `Turn` remains a domain result. #### `teardown` ```ts theme={null} teardown?: DirectAgentTeardown; ``` Cleanup before the run ends, run if and only if this attempt reached the `setup` point in time (a `setup` throw does not exempt it); runs once in `finally`. ### `SandboxAgentDef` #### `name` ```ts theme={null} name: string; ``` The agent's display name/identifier, passed as-is into `Agent.name` — not a registry lookup key, only used for display, result attribution, and dedup fingerprinting. #### `evidenceCoverage` ```ts theme={null} evidenceCoverage: EvidenceCoverage; ``` This adapter's steady-state evidence coverage declaration. Use `completeEvidenceCoverage` only when all six evidence channels are fully collected. #### `ensure` ```ts theme={null} ensure: AgentEnsure | readonly AgentEnsure[]; ``` One or more stable probe obligations. The runner executes them in declaration order before Agent setup. #### `installers` ```ts theme={null} installers?: readonly AgentInstaller[]; ``` Installers paired to exact `ensure` identities. When omitted, a failed probe reports the missing identity instead of guessing an installer. #### `setup` ```ts theme={null} setup?: AgentSetup; ``` Runs once per Attempt (not once per turn): write config.toml / auth config (model/base/auth and other runtime settings that do not change within the Attempt). CLI probing, installation, and verification belong to `agent.ensure`. The runner calls setup once after the Sandbox is ready (after layer prepare/ensure/baseline) and before the first send; it returns no value. #### `tracing` ```ts theme={null} tracing?: AgentTracing; ``` OTLP export configuration: how the CLI inside the sandbox sends traces to the endpoint (env / config file), split out from setup. #### `spanMapper` ```ts theme={null} spanMapper?: SpanMapper; ``` Thin mapper from native span → canonical; omit it to fall back to the generic heuristic. Only affects the waterfall. #### `send` ```ts theme={null} send(input: TurnInput, ctx: SandboxAgentContext): Promise; ``` Once per turn: run the prompt (fresh / resume) and parse it into events. #### `classifySendFailure` ```ts theme={null} classifySendFailure?: SendFailureClassifier; ``` Optional classifier for rejected `send` calls. It does not reinterpret a returned failed `Turn`. #### `teardown` ```ts theme={null} teardown?: AgentTeardown; ``` Cleanup before the sandbox is destroyed, run if and only if this attempt reached the `setup` point in time (a `setup` throw does not exempt it); runs once in `finally`. ### `AgentContext` #### `signal` ```ts theme={null} readonly signal: AbortSignal; ``` Soft-cancellation signal: merges the attempt timeout, run-level interruption (user Ctrl+C), and the eval's own interruption request (see src/runner/attempt.ts). Adapters can optionally check it (or pass it straight to `fetch`) to exit early and gracefully, but this is not the only hard boundary — even if the adapter ignores it entirely, the runner still force-finishes with `Effect.timeoutTo` as a fallback (stopping the sandbox container). #### `model` ```ts theme={null} readonly model?: string; ``` The model name used for this attempt, passed through from the experiment's `model` field. Sandbox-type agents typically use it in setup to write config; Direct Agents typically use it in send to pick the model. #### `reasoningEffort` ```ts theme={null} readonly reasoningEffort?: string; ``` Model reasoning effort; belongs alongside model — decided by the experiment, and when omitted does not override the agent's native default. #### `flags` ```ts theme={null} readonly flags: Readonly>; ``` The experiment's `flags` field, passed through as-is; its content and shape are entirely defined by the experiment author (e.g. `{ webResearch: true }`, `{ systemPrompt: "..." }`). Adapters read fields from it by their own convention; the framework itself does not interpret or validate its content. The name is deliberately distinct from the CLI-parsed `flag` (run-level things like --timeout/--budget) — the two are unrelated concepts. #### `experimentId` ```ts theme={null} readonly experimentId?: string; ``` The experiment id derived from the path (same source as the result attribution `runWho` / `AgentRun.experimentId`); undefined when the run does not go through an experiment (e.g. outside the CLI, constructing an `AgentRun` directly). Typical uses: a `SandboxLayer.setup` hook isolating cross-attempt state per experiment (cache directory names, snapshot tags, etc. partitioned by `ctx.experimentId`), or an adapter switching auth / routing per experiment. It is a different dimension from `flags` (the concrete values of the experiment's conditions) — this is only the stable identifier of "which experiment is running", and carries no condition content. #### `evalId` ```ts theme={null} readonly evalId?: string; ``` The current path-derived eval id. NiceEval fills it for normal runner calls; third-party callers constructing `AgentContext` directly may omit it. #### `attempt` ```ts theme={null} readonly attempt?: AttemptRef; ``` Reference to the current attempt. NiceEval fills it for normal runner calls. #### `session` ```ts theme={null} readonly session: AgentSession; ``` #### `telemetry` ```ts theme={null} readonly telemetry?: Telemetry; ``` Present only when OTel integration is configured (the agent's `tracing` block / the config's telemetry exists): this run's OTLP trace receiving information (endpoint + env-based export env). How to hand it to the CLI is declared by the agent's `tracing` block: env-based agents spread ctx.telemetry.env into send; file-based agents write config in tracing.configure. For remote HTTP integrations, send only needs to spread the headers into the request headers (a fresh traceparent per turn); the endpoint is fixed at startup-time configuration (defineConfig(\{ telemetry: \{ port } })) and is not passed here. #### `progress` ```ts theme={null} progress(update: ProgressUpdate): void; ``` Scoped feedback: report what's happening right now (turn / tool / install progress). Short-lived state — the `human` profile updates the active row; `agent`/`ci` do not print it line by line, and it never lands in the final result. Do not call it for every token/delta. The runner attributes it to whichever lifecycle phase the callback is currently in (agent.setup / agent.run / agent.teardown); the caller cannot pretend to be in a different phase (see docs/feature/experiments/library.md). #### `diagnostic` ```ts theme={null} diagnostic(input: DiagnosticInput): void; ``` Scoped feedback: report a problem that should still be visible after the run ends (protocol downgrade, incomplete data, cleanup issues). A permanent event — it lands in the attempt's diagnostics and each profile's permanent output; deduplicated by `dedupeKey`. Even a level of "error" does not change `Turn.status` / verdict — throw an exception when you cannot continue. #### `fact` ```ts theme={null} fact(name: string, value: JsonValue): void; ``` Writes a generic custom-fact document for this Attempt. The name uses a reverse-domain format and cannot begin with `niceeval.`. One owner/name can be written once; a second write is a typed error and never replaces or appends. The value can be any JsonValue. The JSON-stringified `{ observedAt, value }` document is limited to 65,536 UTF-8 bytes; exceeding it throws `record-custom-fact-too-large` synchronously and leaves no partial file. A fact does not change Turn status, Verdict, score, or fingerprint. #### `log` ```ts theme={null} log(msg: string): void; ``` An alias for `progress({ message: msg })`, not a second channel (see "Attempt phases" in `docs/feature/experiments/cli.md`). On a timeout failure, the most recent lines are merged into the result's error information, to help locate where it got stuck. `ctx.session` (`AgentSession`) is the state slot for one conversation line: every `send` on the same conversation line gets the same `ctx.session`, and a new conversation line (the eval's first turn / after `t.newSession()`) gets a fresh one. Accessors: * `id?: string` / `capture(id): void` — session continuation when the server keeps history: `id` is the session id recorded on this line (`undefined` on a new line), `capture` records the returned id (it only lands when none has been recorded yet). * `get(slot: SessionSlot): T | undefined` / `set(slot, value): void` / `take(slot): T | undefined` — adapter-private typed slots. Create each slot once with `createSessionSlot()`; `take` clears on read (consume once), which suits a paused HITL scene. Full contract in the [Adapter concept](/docs/explanation/adapter#context-agentcontext). ## Sandbox interface `ctx.sandbox` (`Sandbox`) on sandbox-type agents is the handle to the current isolated environment; `CommandOptions` are the options for `runCommand` / `runShell`: ### `Sandbox` #### `workdir` ```ts theme={null} readonly workdir: string; ``` Absolute path to the project/workspace root inside the sandbox (the default cwd for agent commands, and where the git baseline is committed). Relative paths in every method resolve against this, and it's also where things land when `cwd`/`targetDir` is omitted. #### `runCommand` ```ts theme={null} runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise; ``` Run a single command; `args` is passed as a separate argv array and is not interpreted by a shell (no `&&`, pipes, or glob expansion). Prefer this when you just want to run one executable, arguments come from external input, and you're worried about injection. #### `runShell` ```ts theme={null} runShell(script: string, opts?: CommandOptions): Promise; ``` Run a whole script, interpreted by a shell (bash), supporting `&&`, pipes, `$()`, redirection, etc. Use this when you need to chain multiple commands or do conditional logic. #### `runCommandOrThrow` ```ts theme={null} runCommandOrThrow( cmd: string, args?: readonly string[], opts?: CommandOptions, ): Promise; ``` Run one executable like `runCommand`, but throw when it exits non-zero. A successful result has `exitCode: 0` in its type. #### `runShellOrThrow` ```ts theme={null} runShellOrThrow(script: string, opts?: CommandOptions): Promise; ``` Run a shell script like `runShell`, but throw when it exits non-zero. A successful result has `exitCode: 0` in its type. #### `readText` ```ts theme={null} readText(path: string): Promise; ``` Read a UTF-8 file inside the sandbox. Throws if the file does not exist. #### `writeText` ```ts theme={null} writeText(path: string, content: string): Promise; ``` Write one UTF-8 file inside the sandbox, creating parent directories when needed. #### `readBytes` ```ts theme={null} readBytes(path: string): Promise; ``` Read a sandbox file as exact bytes. The public contract is runtime-neutral and does not expose Node.js `Buffer`. #### `writeBytes` ```ts theme={null} writeBytes(path: string, content: Uint8Array): Promise; ``` Write exact bytes to a sandbox file, creating parent directories when needed. #### `pathExists` ```ts theme={null} pathExists(path: string): Promise; ``` Check whether a file or directory path exists inside the sandbox. #### `uploadDirectory` ```ts theme={null} uploadDirectory( sourceDir: string | URL, targetDir?: string, options?: { readonly ignore?: readonly string[] }, ): Promise; ``` Upload an entire host directory into the sandbox. `targetDir` defaults to `workdir`; `options.ignore` excludes matching basenames. #### `stop` ```ts theme={null} stop(): Promise; ``` Destroy the compute resource the sandbox occupies (container/microVM). The sandbox is unusable after this; whether it's safe to call repeatedly varies by provider — do not rely on it. #### `sandboxId` ```ts theme={null} readonly sandboxId: string; ``` Stable identifier for this sandbox (each provider's native ID, e.g. a Docker container ID prefix); used to correlate session state for the same sandbox across calls, and for log display. #### `otlpHost` ```ts theme={null} readonly otlpHost: string | null; ``` Where the OTLP receiver can be placed. * `string`: inside the sandbox, this hostname can reach the host's receiver. * `null`: the provider does not promise that the sandbox can reach back to the host; the runner tries to start an attempt-scope receiver inside the sandbox instead. This does not guarantee tracing succeeds; when the image lacks the runtime the receiver needs, only a supplemental diagnostic is recorded. `defineConfig({ telemetry: { host } })` can override this explicitly when you have already provided a controlled tunnel. #### `appendLog` ```ts theme={null} appendLog?(line: string): Promise; ``` Optional: write a line into the container's "main log" (the one PID1 tails) — so `docker logs` / the Docker UI's Logs tab shows the agent's turn-by-turn activity live. Implemented by the docker provider; others can omit it. #### `downloadFile` ```ts theme={null} downloadFile(sourcePath: string, target: string | URL): Promise; ``` Download one sandbox file to a host path. Use `readBytes` when the caller needs bytes in memory. #### `uploadFile` ```ts theme={null} uploadFile(source: string | URL, targetPath: string): Promise; ``` Upload one host file to a sandbox path. Use `writeBytes` when the content is already in memory. #### `downloadDirectory` ```ts theme={null} downloadDirectory( sourceDir: string, targetDir: string | URL, options?: { readonly ignore?: readonly string[] }, ): Promise; ``` Recursively download a sandbox directory to a required host target. `options.ignore` excludes matching basenames. ### `CommandOptions` #### `env` ```ts theme={null} readonly env?: Readonly>; ``` Append/override environment variables for this command (merged on top of the sandbox's default environment, not clearing defaults; each provider keeps some of its own fixed variables like `PATH`, which are not guaranteed to be overridable here). #### `cwd` ```ts theme={null} readonly cwd?: string; ``` Working directory for this command; falls back to `Sandbox.workdir` when omitted. Relative paths resolve against workdir, absolute paths are used as-is. #### `stream` ```ts theme={null} readonly stream?: boolean; ``` Also send this command's output into the sandbox's "native log stream" (so `docker logs` / the Docker UI's Logs tab shows it live). Turn this on for agent commands (codex exec / bub run / claude) and you'll see the agent's raw output in the container logs. Each provider implements this its own way (docker: tee to the file PID1 tails; providers that don't support it ignore it) — how the log surfaces is the provider's concern, the adapter only declares intent. #### `onStdout` ```ts theme={null} readonly onStdout?: (chunk: string) => void | Promise; ``` Called once for each chunk of the command's stdout as it arrives. The callback is only for short-lived feedback during the run; the complete stdout still appears as-is in the returned `CommandResult`. When a provider does not support true streaming, it must at least call this once with the complete stdout after the command finishes — it cannot silently drop it. #### `onStderr` ```ts theme={null} readonly onStderr?: (chunk: string) => void | Promise; ``` The stderr counterpart to `onStdout`; the complete stderr is still retained in `CommandResult`. #### `user` ```ts theme={null} readonly user?: string; ``` Overrides the execution identity for this command; omit it to use the Sandbox's default identity (whatever the environment itself declares — the Docker image `USER`, the Compose service `user:`, the E2B template's default user, or the host's current user). Semantics stay consistent across providers, and each provider maps this to its own native mechanism (docker: `exec --user`; E2B: `{ user }`; Vercel: only accepts `"root"`, mapped to `{ sudo: true }`, other values throw; local: any value throws). Providers that always run as root treat it as a no-op; providers with no way to switch identity at all may not support it (throw) — but **the semantics for omission and explicit values stay consistent** regardless of provider. #### `timeoutMs` ```ts theme={null} readonly timeoutMs?: number; ``` Optional command-specific timeout in milliseconds. When omitted, the command inherits the remaining attempt deadline. #### `signal` ```ts theme={null} readonly signal?: AbortSignal; ``` Cancellation signal for this managed command tree. Providers settle only after the command tree has terminated or the sandbox has been retired. ## `ctx` and `t` `ctx` is the run context seen by the adapter side; `t` is the test context seen by the eval author. They use the same run data but have different responsibilities.