defineAgent and defineSandboxAgent: adapter reference

Every NiceEval agent is an adapter: code that knows how to drive a specific backend and translate its output into the standard event stream. The runner only calls agent.send(input, ctx).

Full round trip of one t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter returns a Turn with standard events.

`defineAgent`

Use this for direct agent integrations:

import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

defineDirectAgent remains available as a deprecated compatibility alias. It references the same function; use defineAgent in new code. Agent.kind produced by defineAgent is always "direct" (an internal discriminant field you never declare). There are no capability bits on t that need declaring to unlock — filesystem assertions like t.sandbox exist only on agents constructed with defineSandboxAgent (kind: "sandbox"); everything else is judged by the events send actually returned and whether ctx.session was used. See the capabilities reference. The full field set is in “Agent and Adapter Context fields” below.

Direct agent example

export default defineAgent({
  name: "echo",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input) {
    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: input.text }],
    };
  },
});

`input` (`TurnInput`)

string

Text passed by the current t.send(...).

readonly InputFile[] | undefined

Files attached to this turn (images, etc.). Adapters that do not support multimodal input can ignore it.

readonly InputResponse[] | undefined

Present only on answer turns (t.respond / t.respondAll): per-request structured answers, matched by requestId.

`defineSandboxAgent`

Use this for coding agent CLIs. The produced Agent.kind is always "sandbox" — filesystem assertions like t.sandbox and t.sandbox.fileChanged() unlock only on this kind of agent. The full field set is likewise in “Agent and Adapter Context fields” below, under SandboxAgentDef; the complete methods of ctx.sandbox (the Sandbox interface) are in the “Sandbox interface” section further below.

import { completeEvidenceCoverage, defineSandboxAgent } from "niceeval/adapter";
import { command } from "niceeval/sandbox";

export default defineSandboxAgent({
  name: "my-cli-agent",
  evidenceCoverage: completeEvidenceCoverage,
  ensure: {
    identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" },
    probe: command("my-agent", ["--version"]),
  },
  async send(input, ctx) {
    await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
    return { status: "completed", events: [] };
  },
});

Registration

import { defineExperiment } from "niceeval";
import echo from "./agents/echo";

export default defineExperiment({
  agent: echo,
  attempts: 1,
});

Agent and Adapter Context fields

Construction parameters for defineAgent (DirectAgentDef) and defineSandboxAgent (SandboxAgentDef), plus the ctx (AgentContext) that both receive in send(input, ctx):

`DirectAgentDef`

`name`

name: string;

The agent’s display name/identifier, passed as-is into Agent.name — not a registry lookup key, only used for display, result attribution, and dedup fingerprinting.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This adapter’s steady-state evidence coverage declaration. Use completeEvidenceCoverage only when all six evidence channels are fully collected.

`setup`

setup?: DirectAgentSetup;

Runs once per attempt. A Direct Agent receives no Sandbox; use this for one-time work such as establishing a connection or authentication.

`tracing`

tracing?: Omit<AgentTracing, "configure">;

OTLP export configuration: how the remote system under test sends traces to the endpoint (env-based injection / file-based configuration).

`spanMapper`

spanMapper?: SpanMapper;

Thin mapper from native span → canonical; omit it to fall back to the generic heuristic. Only affects the waterfall.

`send`

send(input: TurnInput, ctx: AgentContext): Promise<Turn>;

Once per turn: send one turn’s prompt to the remote system under test (HTTP/SDK, etc.), and parse the response into events.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional classifier for rejected send calls. It classifies structured execution failures; a trusted failed Turn remains a domain result.

`teardown`

teardown?: DirectAgentTeardown;

Cleanup before the run ends, run if and only if this attempt reached the setup point in time (a setup throw does not exempt it); runs once in finally.

`SandboxAgentDef`

`name`

name: string;

The agent’s display name/identifier, passed as-is into Agent.name — not a registry lookup key, only used for display, result attribution, and dedup fingerprinting.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This adapter’s steady-state evidence coverage declaration. Use completeEvidenceCoverage only when all six evidence channels are fully collected.

`ensure`

ensure: AgentEnsure | readonly AgentEnsure[];

One or more stable probe obligations. The runner executes them in declaration order before Agent setup.

`installers`

installers?: readonly AgentInstaller[];

Installers paired to exact ensure identities. When omitted, a failed probe reports the missing identity instead of guessing an installer.

`setup`

setup?: AgentSetup;

Runs once per Attempt (not once per turn): write config.toml / auth config (model/base/auth and other runtime settings that do not change within the Attempt). CLI probing, installation, and verification belong to agent.ensure. The runner calls setup once after the Sandbox is ready (after layer prepare/ensure/baseline) and before the first send; it returns no value.

`tracing`

tracing?: AgentTracing;

OTLP export configuration: how the CLI inside the sandbox sends traces to the endpoint (env / config file), split out from setup.

`spanMapper`

spanMapper?: SpanMapper;

Thin mapper from native span → canonical; omit it to fall back to the generic heuristic. Only affects the waterfall.

`send`

send(input: TurnInput, ctx: SandboxAgentContext): Promise<Turn>;

Once per turn: run the prompt (fresh / resume) and parse it into events.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional classifier for rejected send calls. It does not reinterpret a returned failed Turn.

`teardown`

teardown?: AgentTeardown;

Cleanup before the sandbox is destroyed, run if and only if this attempt reached the setup point in time (a setup throw does not exempt it); runs once in finally.

`AgentContext`

`signal`

readonly signal: AbortSignal;

Soft-cancellation signal: merges the attempt timeout, run-level interruption (user Ctrl+C), and the eval’s own interruption request (see src/runner/attempt.ts). Adapters can optionally check it (or pass it straight to fetch) to exit early and gracefully, but this is not the only hard boundary — even if the adapter ignores it entirely, the runner still force-finishes with Effect.timeoutTo as a fallback (stopping the sandbox container).

`model`

readonly model?: string;

The model name used for this attempt, passed through from the experiment’s model field. Sandbox-type agents typically use it in setup to write config; Direct Agents typically use it in send to pick the model.

`reasoningEffort`

readonly reasoningEffort?: string;

Model reasoning effort; belongs alongside model — decided by the experiment, and when omitted does not override the agent’s native default.

`flags`

readonly flags: Readonly<Record<string, JsonValue>>;

The experiment’s flags field, passed through as-is; its content and shape are entirely defined by the experiment author (e.g. { webResearch: true }, { systemPrompt: "..." }). Adapters read fields from it by their own convention; the framework itself does not interpret or validate its content. The name is deliberately distinct from the CLI-parsed flag (run-level things like —timeout/—budget) — the two are unrelated concepts.

`experimentId`

readonly experimentId?: string;

The experiment id derived from the path (same source as the result attribution runWho / AgentRun.experimentId); undefined when the run does not go through an experiment (e.g. outside the CLI, constructing an AgentRun directly). Typical uses: a SandboxLayer.setup hook isolating cross-attempt state per experiment (cache directory names, snapshot tags, etc. partitioned by ctx.experimentId), or an adapter switching auth / routing per experiment. It is a different dimension from flags (the concrete values of the experiment’s conditions) — this is only the stable identifier of “which experiment is running”, and carries no condition content.

`evalId`

readonly evalId?: string;

The current path-derived eval id. NiceEval fills it for normal runner calls; third-party callers constructing AgentContext directly may omit it.

`attempt`

readonly attempt?: AttemptRef;

Reference to the current attempt. NiceEval fills it for normal runner calls.

`session`

readonly session: AgentSession;

`telemetry`

readonly telemetry?: Telemetry;

Present only when OTel integration is configured (the agent’s tracing block / the config’s telemetry exists): this run’s OTLP trace receiving information (endpoint + env-based export env). How to hand it to the CLI is declared by the agent’s tracing block: env-based agents spread ctx.telemetry.env into send; file-based agents write config in tracing.configure. For remote HTTP integrations, send only needs to spread the headers into the request headers (a fresh traceparent per turn); the endpoint is fixed at startup-time configuration (defineConfig({ telemetry: { port } })) and is not passed here.

`progress`

progress(update: ProgressUpdate): void;

Scoped feedback: report what’s happening right now (turn / tool / install progress). Short-lived state — the human profile updates the active row; agent/ci do not print it line by line, and it never lands in the final result. Do not call it for every token/delta. The runner attributes it to whichever lifecycle phase the callback is currently in (agent.setup / agent.run / agent.teardown); the caller cannot pretend to be in a different phase (see docs/feature/experiments/library.md).

`diagnostic`

diagnostic(input: DiagnosticInput): void;

Scoped feedback: report a problem that should still be visible after the run ends (protocol downgrade, incomplete data, cleanup issues). A permanent event — it lands in the attempt’s diagnostics and each profile’s permanent output; deduplicated by dedupeKey. Even a level of “error” does not change Turn.status / verdict — throw an exception when you cannot continue.

`fact`

fact(name: string, value: JsonValue): void;

Writes a generic custom-fact document for this Attempt. The name uses a reverse-domain format and cannot begin with niceeval.. One owner/name can be written once; a second write is a typed error and never replaces or appends. The value can be any JsonValue. The JSON-stringified { observedAt, value } document is limited to 65,536 UTF-8 bytes; exceeding it throws record-custom-fact-too-large synchronously and leaves no partial file. A fact does not change Turn status, Verdict, score, or fingerprint.

`log`

log(msg: string): void;

An alias for progress({ message: msg }), not a second channel (see “Attempt phases” in docs/feature/experiments/cli.md). On a timeout failure, the most recent lines are merged into the result’s error information, to help locate where it got stuck. ctx.session (AgentSession) is the state slot for one conversation line: every send on the same conversation line gets the same ctx.session, and a new conversation line (the eval’s first turn / after t.newSession()) gets a fresh one. Accessors:

id?: string / capture(id): void — session continuation when the server keeps history: id is the session id recorded on this line (undefined on a new line), capture records the returned id (it only lands when none has been recorded yet).
get<T>(slot: SessionSlot<T>): T | undefined / set<T>(slot, value): void / take<T>(slot): T | undefined — adapter-private typed slots. Create each slot once with createSessionSlot<T>(); take clears on read (consume once), which suits a paused HITL scene.

Full contract in the Adapter concept.

Sandbox interface

ctx.sandbox (Sandbox) on sandbox-type agents is the handle to the current isolated environment; CommandOptions are the options for runCommand / runShell:

`Sandbox`

`workdir`

readonly workdir: string;

Absolute path to the project/workspace root inside the sandbox (the default cwd for agent commands, and where the git baseline is committed). Relative paths in every method resolve against this, and it’s also where things land when cwd/targetDir is omitted.

`runCommand`

runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise<CommandResult>;

Run a single command; args is passed as a separate argv array and is not interpreted by a shell (no &&, pipes, or glob expansion). Prefer this when you just want to run one executable, arguments come from external input, and you’re worried about injection.

`runShell`

runShell(script: string, opts?: CommandOptions): Promise<CommandResult>;

Run a whole script, interpreted by a shell (bash), supporting &&, pipes, $(), redirection, etc. Use this when you need to chain multiple commands or do conditional logic.

`runCommandOrThrow`

runCommandOrThrow(
  cmd: string,
  args?: readonly string[],
  opts?: CommandOptions,
): Promise<SuccessfulCommandResult>;

Run one executable like runCommand, but throw when it exits non-zero. A successful result has exitCode: 0 in its type.

`runShellOrThrow`

runShellOrThrow(script: string, opts?: CommandOptions): Promise<SuccessfulCommandResult>;

Run a shell script like runShell, but throw when it exits non-zero. A successful result has exitCode: 0 in its type.

`readText`

readText(path: string): Promise<string>;

Read a UTF-8 file inside the sandbox. Throws if the file does not exist.

`writeText`

writeText(path: string, content: string): Promise<void>;

Write one UTF-8 file inside the sandbox, creating parent directories when needed.

`readBytes`

readBytes(path: string): Promise<Uint8Array>;

Read a sandbox file as exact bytes. The public contract is runtime-neutral and does not expose Node.js Buffer.

`writeBytes`

writeBytes(path: string, content: Uint8Array): Promise<void>;

Write exact bytes to a sandbox file, creating parent directories when needed.

`pathExists`

pathExists(path: string): Promise<boolean>;

Check whether a file or directory path exists inside the sandbox.

`uploadDirectory`

uploadDirectory(
  sourceDir: string | URL,
  targetDir?: string,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

Upload an entire host directory into the sandbox. targetDir defaults to workdir; options.ignore excludes matching basenames.

`stop`

stop(): Promise<void>;

Destroy the compute resource the sandbox occupies (container/microVM). The sandbox is unusable after this; whether it’s safe to call repeatedly varies by provider — do not rely on it.

`sandboxId`

readonly sandboxId: string;

Stable identifier for this sandbox (each provider’s native ID, e.g. a Docker container ID prefix); used to correlate session state for the same sandbox across calls, and for log display.

`otlpHost`

readonly otlpHost: string | null;

Where the OTLP receiver can be placed.

string: inside the sandbox, this hostname can reach the host’s receiver.
null: the provider does not promise that the sandbox can reach back to the host; the runner tries to start an attempt-scope receiver inside the sandbox instead. This does not guarantee tracing succeeds; when the image lacks the runtime the receiver needs, only a supplemental diagnostic is recorded. defineConfig({ telemetry: { host } }) can override this explicitly when you have already provided a controlled tunnel.

`appendLog`

appendLog?(line: string): Promise<void>;

Optional: write a line into the container’s “main log” (the one PID1 tails) — so docker logs / the Docker UI’s Logs tab shows the agent’s turn-by-turn activity live. Implemented by the docker provider; others can omit it.

`downloadFile`

downloadFile(sourcePath: string, target: string | URL): Promise<void>;

Download one sandbox file to a host path. Use readBytes when the caller needs bytes in memory.

`uploadFile`

uploadFile(source: string | URL, targetPath: string): Promise<void>;

Upload one host file to a sandbox path. Use writeBytes when the content is already in memory.

`downloadDirectory`

downloadDirectory(
  sourceDir: string,
  targetDir: string | URL,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

Recursively download a sandbox directory to a required host target. options.ignore excludes matching basenames.

`CommandOptions`

`env`

readonly env?: Readonly<Record<string, string>>;

Append/override environment variables for this command (merged on top of the sandbox’s default environment, not clearing defaults; each provider keeps some of its own fixed variables like PATH, which are not guaranteed to be overridable here).

`cwd`

readonly cwd?: string;

Working directory for this command; falls back to Sandbox.workdir when omitted. Relative paths resolve against workdir, absolute paths are used as-is.

`stream`

readonly stream?: boolean;

Also send this command’s output into the sandbox’s “native log stream” (so docker logs / the Docker UI’s Logs tab shows it live). Turn this on for agent commands (codex exec / bub run / claude) and you’ll see the agent’s raw output in the container logs. Each provider implements this its own way (docker: tee to the file PID1 tails; providers that don’t support it ignore it) — how the log surfaces is the provider’s concern, the adapter only declares intent.

`onStdout`

readonly onStdout?: (chunk: string) => void | Promise<void>;

Called once for each chunk of the command’s stdout as it arrives. The callback is only for short-lived feedback during the run; the complete stdout still appears as-is in the returned CommandResult. When a provider does not support true streaming, it must at least call this once with the complete stdout after the command finishes — it cannot silently drop it.

`onStderr`

readonly onStderr?: (chunk: string) => void | Promise<void>;

The stderr counterpart to onStdout; the complete stderr is still retained in CommandResult.

`user`

readonly user?: string;

Overrides the execution identity for this command; omit it to use the Sandbox’s default identity (whatever the environment itself declares — the Docker image USER, the Compose service user:, the E2B template’s default user, or the host’s current user). Semantics stay consistent across providers, and each provider maps this to its own native mechanism (docker: exec --user; E2B: { user }; Vercel: only accepts "root", mapped to { sudo: true }, other values throw; local: any value throws). Providers that always run as root treat it as a no-op; providers with no way to switch identity at all may not support it (throw) — but the semantics for omission and explicit values stay consistent regardless of provider.

`timeoutMs`

readonly timeoutMs?: number;

Optional command-specific timeout in milliseconds. When omitted, the command inherits the remaining attempt deadline.

`signal`

readonly signal?: AbortSignal;

Cancellation signal for this managed command tree. Providers settle only after the command tree has terminated or the sandbox has been retired.

`ctx` and `t`

ctx is the run context seen by the adapter side; t is the test context seen by the eval author. They use the same run data but have different responsibilities.

​defineAgent

​Direct agent example

​input (TurnInput)

​defineSandboxAgent

​Registration

​Agent and Adapter Context fields

​DirectAgentDef

​name

​evidenceCoverage

​setup

​tracing

​spanMapper

​send

​classifySendFailure

​teardown

​SandboxAgentDef

​name

​evidenceCoverage

​ensure

​installers

​setup

​tracing

​spanMapper

​send

​classifySendFailure

​teardown

​AgentContext

​signal

​model

​reasoningEffort

​flags

​experimentId

​evalId

​attempt

​session

​telemetry

​progress

​diagnostic

​fact

​log

​Sandbox interface

​Sandbox

​workdir

​runCommand

​runShell

​runCommandOrThrow

​runShellOrThrow

​readText

​writeText

​readBytes

​writeBytes

​pathExists

​uploadDirectory

​stop

​sandboxId

​otlpHost

​appendLog

​downloadFile

​uploadFile

​downloadDirectory

​CommandOptions

​env

​cwd

​stream

​onStdout

​onStderr

​user

​timeoutMs

​signal

​ctx and t

`defineAgent`

Direct agent example

`input` (`TurnInput`)

`defineSandboxAgent`

Registration

Agent and Adapter Context fields

`DirectAgentDef`

`name`

`evidenceCoverage`

`setup`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`teardown`

`SandboxAgentDef`

`name`

`evidenceCoverage`

`ensure`

`installers`

`setup`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`teardown`

`AgentContext`

`signal`

`model`

`reasoningEffort`

`flags`

`experimentId`

`evalId`

`attempt`

`session`

`telemetry`

`progress`

`diagnostic`

`fact`

`log`

Sandbox interface

`Sandbox`

`workdir`

`runCommand`

`runShell`

`runCommandOrThrow`

`runShellOrThrow`

`readText`

`writeText`

`readBytes`

`writeBytes`

`pathExists`

`uploadDirectory`

`stop`

`sandboxId`

`otlpHost`

`appendLog`

`downloadFile`

`uploadFile`

`downloadDirectory`

`CommandOptions`

`env`

`cwd`

`stream`

`onStdout`

`onStderr`

`user`

`timeoutMs`

`signal`

`ctx` and `t`