defineAgent and defineSandboxAgent: Adapter Reference

Every NiceEval Agent is an Adapter: code that knows how to drive a particular backend and translate its output into the standard event stream. The Runner only calls agent.send(input, ctx).

`defineAgent`

Use this for direct Agent integration:

import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

An Agent produced by defineAgent always has Agent.kind: "direct". That is an internal discriminator; you do not declare it. t has no capability that must be declared to unlock it. Filesystem assertions such as t.sandbox exist only on an Agent constructed by defineSandboxAgent, whose kind is "sandbox". Everything else is determined by the events that send actually returns and by use of ctx.session; see the Capabilities reference. The complete field set appears below in Agent and Adapter Context fields.

Direct Agent example

export default defineAgent({
  name: "echo",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input) {
    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: input.text }],
    };
  },
});

`input` (`TurnInput`)

string

The text supplied to the current t.send(...).

readonly InputFile[] | undefined

Files attached to this turn, such as images. An Adapter that does not support multimodal input can ignore them.

readonly InputResponse[] | undefined

Present only on response turns (t.respond / t.respondAll): structured responses for each request, matched by requestId.

`defineSandboxAgent`

Use this for a coding-Agent CLI. The resulting Agent.kind is always "sandbox". Filesystem assertions such as t.sandbox and t.sandbox.fileChanged() are available only for this kind of Agent. The complete field set appears below under SandboxAgentDef in Agent and Adapter Context fields. The complete methods on ctx.sandbox—the Sandbox interface—appear further below under Sandbox interface.

import { completeEvidenceCoverage, defineSandboxAgent } from "niceeval/adapter";
import { command } from "niceeval/sandbox";

export default defineSandboxAgent({
  name: "my-cli-agent",
  evidenceCoverage: completeEvidenceCoverage,
  ensure: {
    identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" },
    probe: command("my-agent", ["--version"]),
  },
  async send(input, ctx) {
    await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
    return { status: "completed", events: [] };
  },
});

Register it

import { defineExperiment } from "niceeval";
import echo from "./agents/echo";

export default defineExperiment({
  agent: echo,
  attempts: 1,
});

Agent and Adapter Context fields

The construction parameters for defineAgent (DirectAgentDef) and defineSandboxAgent (SandboxAgentDef), plus ctx (AgentContext) received by both send(input, ctx) functions:

`DirectAgentDef`

`name`

name: string;

The Agent display name and identifier. It enters Agent.name unchanged. It is not a registry lookup key; it is used only for display, result attribution, and deduplication fingerprints.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This Adapter’s normal evidence-coverage declaration. Use completeEvidenceCoverage for complete collection.

`setup`

setup?: DirectAgentSetup;

Once per Attempt. A Direct Agent receives no Sandbox. Use it for one-time preparation such as opening a connection or authenticating.

`tracing`

tracing?: Omit<AgentTracing, "configure">;

OTLP-export configuration: how the system under test sends traces to an endpoint through environment-based injection.

`spanMapper`

spanMapper?: SpanMapper;

A thin native-span-to-canonical mapper. When omitted, the generic heuristic applies. It affects only the waterfall.

`send`

send(input: TurnInput, ctx: AgentContext): Promise<Turn>;

Once per turn: sends a turn prompt directly to a function, SDK, or service endpoint, then parses the response into events.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional classifier for a send execution failure. See Agent.classifySendFailure.

`teardown`

teardown?: DirectAgentTeardown;

Cleanup before the run ends. It runs once in finally if and only if this Attempt reached the setup point. A setup throw does not exempt it.

`SandboxAgentDef`

`name`

name: string;

The Agent display name and identifier. It enters Agent.name unchanged. It is not a registry lookup key; it is used only for display, result attribution, and deduplication fingerprints.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This Adapter’s normal evidence-coverage declaration. Use completeEvidenceCoverage for complete collection.

`ensure`

ensure: AgentEnsure | readonly AgentEnsure[];

One item or an array is normalized into Agent layers in declaration order.

`installers`

installers?: readonly AgentInstaller[];

Paired installation layers. Omission means this Adapter supplies only a probe protocol. On a miss, the Runner names the exact missing identity at agent.ensure. The factory normalizes omission to an empty array.

`setup`

setup?: AgentSetup;

Once per Attempt, not once per send: write config.toml, authentication configuration, and stable run configuration such as model/base/auth for this Attempt. CLI probing, installation, and rechecking belong to agent.ensure. The Runner calls it once after the Sandbox is ready—after layer prepare/ensure/baseline—and before the first send. It returns no value.

`tracing`

tracing?: AgentTracing;

OTLP-export configuration: how a CLI in the Sandbox sends traces to an endpoint through environment variables or a configuration file. It is separated from setup.

`spanMapper`

spanMapper?: SpanMapper;

A thin native-span-to-canonical mapper. When omitted, the generic heuristic applies. It affects only the waterfall.

`send`

send(input: TurnInput, ctx: SandboxAgentContext): Promise<Turn>;

Once per turn: run the prompt fresh or with resume, then parse it into events.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional classifier for a send execution failure. See Agent.classifySendFailure.

`teardown`

teardown?: AgentTeardown;

Cleanup before the Sandbox is destroyed. It runs once in finally if and only if this Attempt reached the setup point. A setup throw does not exempt it.

`AgentContext`

`signal`

readonly signal: AbortSignal;

A soft cancellation signal that combines an Attempt timeout, a run-level interruption such as user Ctrl+C, and an Eval’s own cancellation request. See src/runner/attempt.ts. An Adapter can check it selectively or pass it directly to fetch to exit gracefully early, but it is not the only hard boundary. Even when an Adapter ignores it completely, the Runner uses Effect.timeoutTo as a backstop to force cleanup, including stopping a Sandbox container.

`model`

readonly model?: string;

The model name for this Attempt, passed through from the Experiment’s model field. A Sandbox Agent typically writes it to configuration during setup; a Direct Agent typically chooses a model with it during send.

`reasoningEffort`

readonly reasoningEffort?: string;

The model reasoning-effort level. It belongs with model: the Experiment decides it, and omission does not override the Agent’s native default.

`flags`

readonly flags: Readonly<globalThis.Record<string, JsonValue>>;

The Experiment’s flags field passes through unchanged. Its contents and shape are entirely defined by the Experiment author, such as { webResearch: true } or { systemPrompt: "..." }. An Adapter reads fields according to its own convention; the framework neither interprets nor validates them. The name intentionally avoids the CLI-parsed flag, which represents how the run operates, such as --timeout / --budget; the two concepts are unrelated.

`experimentId`

readonly experimentId?: string;

The path-derived Experiment ID, from the same source as result attribution runWho / AgentRun.experimentId. It is undefined when a run does not use an Experiment, such as an AgentRun constructed directly outside the CLI. Typical uses include a SandboxLayer command isolating cross-Attempt infrastructure by Experiment, or an Adapter selecting authentication or routing by Experiment. It is a separate dimension from flags, which holds concrete Experiment-condition values: this is only a stable identifier for which Experiment runs, not its conditions.

`evalId`

readonly evalId?: string;

The Eval ID for the current Attempt. The NiceEval Runner always fills it from the discovered Eval identity; a third party constructing AgentContext directly can omit it. An Adapter can use it to locate read-only host assets beside the task, but cannot use it to bypass isolation of hidden Sandbox criteria.

`evalGroup`

readonly evalGroup?: {
  readonly id: string;
  readonly definitionHash: string;
};

The current Attempt’s Eval Group. It is omitted for an ungrouped Eval.

`attempt`

readonly attempt?: AttemptRef;

The current Attempt reference supplied by the Runner. A third party constructing context directly can omit it.

`session`

readonly session: AgentSession;

`telemetry`

readonly telemetry?: Telemetry;

Present only when OTel integration is configured, through the Agent’s tracing block or Config telemetry. It provides OTLP-trace receiving information for this run: the endpoint plus environment variables for environment-based export. The Agent’s tracing block declares how to give it to a CLI. For environment-based export, spread ctx.telemetry.env into send. For file-based export, write configuration in tracing.configure. For remote HTTP integration, send only needs to spread headers into the request headers, with one new traceparent per turn. The endpoint is startup configuration fixed by defineConfig({ telemetry: { port } }), not passed from here.

`progress`

progress(update: ProgressUpdate): void;

Scoped feedback that reports what is happening now, such as a turn, tool, or installation progress. It is short-lived state: the Human profile updates its active line, agent / ci do not print every update, and it does not enter final results. Do not call it for every token or delta. The Runner attributes the call to the current lifecycle phase—agent.setup, agent.run, or agent.teardown—so callers cannot impersonate another phase. See docs/feature/experiments/library.md.

`diagnostic`

diagnostic(input: DiagnosticInput): void;

Scoped feedback that reports a problem that should remain after the run ends, such as protocol degradation, incomplete data, or a cleanup problem. It is a durable event: it enters the Attempt’s diagnostics and permanent output in every profile, and dedupeKey deduplicates it. Even level: "error" does not change Turn.status or a Verdict. Throw when execution cannot continue.

`log`

log(msg: string): void;

An alias for progress({ message: msg }), not a second channel. See Attempt phases in docs/feature/experiments/cli.md. When timeout fails, the most recent lines join the result’s error information to help locate the step that stalled. ctx.session (AgentSession) is a state slot for one session line. Every send on the same session line receives the same ctx.session. A new session line—the first turn of an Eval or after t.newSession()—receives a fresh one. Its accessors are:

id?: string / capture(id): void: use these for session continuation when the service remembers history. id is the session ID recorded for this line, or undefined for a new line. capture records the returned ID only when none has been recorded.
createSessionSlot<T>(name): creates a typed slot in Adapter module scope. Slots are isolated by symbol identity.
get(slot) / set(slot, value): get and set client history or Adapter-private state.
take(slot): reads and removes a HITL paused-turn state for one-time consumption.

For the complete contract, see Adapter concepts.

Sandbox interface

For a Sandbox Agent, ctx.sandbox (Sandbox) is the handle for the current isolated environment. Sandbox extends SandboxOperations and SandboxTransferOperations; the members below are grouped by their declaring interface. CommandOptions are optional parameters for runCommand / runShell:

`SandboxOperations`

`workdir`

readonly workdir: string;

The absolute project or Workspace root inside the Sandbox. It is the default cwd for Agent commands and the location of the Git baseline commit. Relative paths for every method resolve from it; omitted cwd and targetDir also use it.

`runCommand`

runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise<CommandResult>;

Runs one command. args are separate argv values and are not interpreted by a shell, so no &&, pipes, or glob expansion occurs. Prefer it for one executable, externally supplied arguments, or input that must avoid shell injection.

`runShell`

runShell(script: string, opts?: CommandOptions): Promise<CommandResult>;

Runs a complete script through the shell (bash), with support for &&, pipes, $(), redirects, and similar shell syntax. Use it when you need to compose commands or branch on conditions.

`runCommandOrThrow`

runCommandOrThrow(
  cmd: string,
  args?: readonly string[],
  opts?: CommandOptions,
): Promise<SuccessfulCommandResult>;

Runs one command like runCommand, but throws SandboxCommandExitError on a nonzero exit. Its message contains a bounded, cleaned, redacted stderr tail, falling back to stdout when stderr is empty. The complete output remains in the error’s result; a successful result has exitCode: 0 at the type level.

`runShellOrThrow`

runShellOrThrow(script: string, opts?: CommandOptions): Promise<SuccessfulCommandResult>;

Runs a shell script like runShell, but throws SandboxCommandExitError on a nonzero exit. Its error summary, complete output, and successful-result semantics match runCommandOrThrow.

`readText`

readText(path: string): Promise<string>;

Reads UTF-8 text from a file in the Sandbox. It throws when the file does not exist.

`writeText`

writeText(path: string, content: string): Promise<void>;

Writes a UTF-8 text file in the Sandbox, creating missing parent directories.

`readBytes`

readBytes(path: string): Promise<Uint8Array>;

Reads exact file bytes from the Sandbox. The public contract does not depend on Node Buffer.

`writeBytes`

writeBytes(path: string, content: Uint8Array): Promise<void>;

Writes exact file bytes in the Sandbox, creating missing parent directories.

`pathExists`

pathExists(path: string): Promise<boolean>;

Checks whether a file or directory path exists in the Sandbox.

`SandboxTransferOperations`

`upload`

upload(content: SandboxContent, targetPath: string): Promise<void>;

Transfers registered immutable content without exposing its host path.

`uploadFile`

uploadFile(source: string | URL, targetPath: string): Promise<void>;

`uploadDirectory`

uploadDirectory(
  sourceDir: string | URL,
  targetDir?: string,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

`downloadFile`

downloadFile(sourcePath: string, target: string | URL): Promise<void>;

`downloadDirectory`

downloadDirectory(
  sourceDir: string,
  targetDir: string | URL,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

`Sandbox`

`stop`

stop(): Promise<void>;

Destroys compute resources held by the Sandbox, such as a container or microVM. The Sandbox cannot be used after this call. Whether it is safe to call repeatedly depends on the Provider; do not rely on that.

`sandboxId`

readonly sandboxId: string;

The stable identifier for this Sandbox: a Provider-native ID such as a Docker container-ID prefix. It associates session state with the same Sandbox across calls and appears in logs.

`otlpHost`

readonly otlpHost: string | null;

The placement capability of the OTLP receiver.

string: the Sandbox can reach the host receiver through this hostname.
null: the Provider does not promise host callback access. The Runner tries to start an Attempt-scoped receiver inside the Sandbox. This does not guarantee tracing succeeds; when the image lacks the receiver runtime, NiceEval records only a supplemental diagnostic.

defineConfig({ telemetry: { host } }) can explicitly override it after the author provides a controlled tunnel.

`appendLog`

appendLog?(line: string): Promise<void>;

Optional. Writes a line into the container’s main log, which PID 1 tails, so docker logs and the Docker UI Logs tab can show Agent activity turn by turn. The docker Provider implements it; other Providers can omit it.

`CommandOptions`

`sensitiveValues`

readonly sensitiveValues?: readonly string[];

Known sensitive plaintext handled by this command, such as an API key, token, or HTTP-header value. The Runner still gives the original values to the Provider to execute, but replaces every exact value before persisting any timing, commands, execution, or error evidence. This array itself is not persisted or added to a fingerprint. Empty strings are ignored. This is explicit provenance, not a secret scanner. Free text that is not declared cannot be identified reliably. When a caller first encodes or splits a value, also register the encoded form that will actually appear in the command or output.

`env`

readonly env?: Readonly<globalThis.Record<string, string>>;

Adds or overrides environment variables for this command, layering them on Sandbox default environment rather than clearing defaults. PATH is Sandbox-managed: each Provider retains the PATH it computed and does not guarantee this option can override it. To extend PATH, use the Sandbox factory’s pathPrepend; see PATH: managed variable and pathPrepend in docs/feature/sandbox/library.md.

`cwd`

readonly cwd?: string;

The working directory for this command. When omitted, it is Sandbox.workdir. A relative path resolves from workdir, and an absolute path is used unchanged.

`stream`

readonly stream?: boolean;

Also sends this command’s output to the Sandbox’s native log stream, so docker logs and the Docker UI Logs tab show it live. Enable it for Agent commands such as codex exec, bub run, or claude to see the Agent’s raw output in container logs. Each Provider implements it differently: docker tees to a file tailed by PID 1, while an unsupported Provider ignores it. How logs surface is the Provider’s concern; the Adapter only declares intent.

`onStdout`

readonly onStdout?: (chunk: string) => void | Promise<void>;

Called for each stdout chunk from the command. The callback is only short-lived feedback while running; full stdout remains unchanged in the returned CommandResult. When a Provider cannot truly stream, it must at least call once with complete stdout after the command ends; it cannot silently drop it.

`onStderr`

readonly onStderr?: (chunk: string) => void | Promise<void>;

The stderr counterpart to onStdout; complete stderr remains in CommandResult.

`user`

readonly user?: string;

Overrides the execution identity for this command. When omitted, it uses the Sandbox default identity declared by the environment itself: Docker image USER, Compose service user:, or E2B template default user. See Execution identity in docs/feature/sandbox/library.md. The semantics are consistent across Providers, while each maps it to its native mechanism: Docker uses exec --user; E2B uses { user }; Vercel accepts only "root" and maps it to { sudo: true }, with other values erroring. A Provider that is root for the whole lifetime treats it as a no-op. A Provider that cannot change identity at all can reject it, but omission and an explicit value keep the same semantics across Providers.

`timeoutMs`

readonly timeoutMs?: number;

The upper bound for this command itself, in milliseconds. Omission is normal: when omitted, the limit is the Attempt deadline’s remaining time. See timeout ownership in docs/feature/sandbox/architecture.md. The Provider layer has no independent default. An explicitly shorter value is an intentional declaration and applies normally; when it expires, the attribution is command-explicit timeout.

`signal`

readonly signal?: AbortSignal;

Cancels this managed command tree. Before its Promise settles, a Provider must confirm that the command tree has ended. When it cannot terminate it precisely, it must retire the whole Sandbox rather than close only the transport and leave a process in the background.

ctx and t

ctx is the runtime context an Adapter sees. t is the test context an Eval author sees. They use the same runtime data, but have different responsibilities.

​defineAgent

​Direct Agent example

​input (TurnInput)

​defineSandboxAgent

​Register it

​Agent and Adapter Context fields

​DirectAgentDef

​name

​evidenceCoverage

​setup

​tracing

​spanMapper

​send

​classifySendFailure

​teardown

​SandboxAgentDef

​name

​evidenceCoverage

​ensure

​installers

​setup

​tracing

​spanMapper

​send

​classifySendFailure

​teardown

​AgentContext

​signal

​model

​reasoningEffort

​flags

​experimentId

​evalId

​evalGroup

​attempt

​session

​telemetry

​progress

​diagnostic

​log

​Sandbox interface

​SandboxOperations

​workdir

​runCommand

​runShell

​runCommandOrThrow

​runShellOrThrow

​readText

​writeText

​readBytes

​writeBytes

​pathExists

​SandboxTransferOperations

​upload

​uploadFile

​uploadDirectory

​downloadFile

​downloadDirectory

​Sandbox

​stop

​sandboxId

​otlpHost

​appendLog

​CommandOptions

​sensitiveValues

​env

​cwd

​stream

​onStdout

​onStderr

​user

​timeoutMs

​signal

​ctx and t

`defineAgent`

Direct Agent example

`input` (`TurnInput`)

`defineSandboxAgent`

Register it

Agent and Adapter Context fields

`DirectAgentDef`

`name`

`evidenceCoverage`

`setup`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`teardown`

`SandboxAgentDef`

`name`

`evidenceCoverage`

`ensure`

`installers`

`setup`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`teardown`

`AgentContext`

`signal`

`model`

`reasoningEffort`

`flags`

`experimentId`

`evalId`

`evalGroup`

`attempt`

`session`

`telemetry`

`progress`

`diagnostic`

`log`

Sandbox interface

`SandboxOperations`

`workdir`

`runCommand`

`runShell`

`runCommandOrThrow`

`runShellOrThrow`

`readText`

`writeText`

`readBytes`

`writeBytes`

`pathExists`

`SandboxTransferOperations`

`upload`

`uploadFile`

`uploadDirectory`

`downloadFile`

`downloadDirectory`

`Sandbox`

`stop`

`sandboxId`

`otlpHost`

`appendLog`

`CommandOptions`

`sensitiveValues`

`env`

`cwd`

`stream`

`onStdout`

`onStderr`

`user`

`timeoutMs`

`signal`

ctx and t