Define What You Evaluate: Adapter Reference

An Adapter connects the object you evaluate and provides its operations and evidence. Agents, games, and ordinary applications all use NiceEval’s Experiments, assertions, and result queries. A custom object provides native methods through defineAdapter and can bind domain checks in assertions; several implementations share an interface and assertions through defineAdapterContract. For the complete definition and how to run it, see Evaluate a custom application. defineAgent and defineSandboxAgent below are dedicated to conversational integration, using send to return the standard event stream.

`defineAgent`

Use this for direct agent integration:

import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

An Agent produced by defineAgent always has Agent.kind set to "direct" (an internal discriminator you do not declare). t has no capability bits that must be declared to unlock them; filesystem assertions such as t.sandbox exist only on Agents built with defineSandboxAgent (kind: "sandbox"). Everything else is determined by the events send actually returns and how ctx.session is used; see the Capabilities reference. For the complete field set, see “Agent and Adapter Context fields” below.

Direct agent example

export default defineAgent({
  name: "echo",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input) {
    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: input.text }],
    };
  },
});

`input` (`TurnInput`)

string

The text passed to the current t.send(...).

readonly InputFile[] | undefined

Files attached to this turn (images and so on). An adapter that does not support multimodal input can ignore them.

readonly InputResponse[] | undefined

Present only on response turns (t.respond / t.respondAll): structured answers per request, matched by requestId.

`defineSandboxAgent`

Use this for coding agent CLIs. The resulting Agent.kind is always "sandbox"; filesystem assertions such as t.sandbox and t.sandbox.fileChanged() are unlocked only on this kind of agent. The complete field set is also in “Agent and Adapter Context fields” below, under SandboxAgentDef. The complete methods of ctx.sandbox (the Sandbox interface) are in “Sandbox interface” further below.

import { completeEvidenceCoverage, defineSandboxAgent } from "niceeval/adapter";
import { command } from "niceeval/sandbox";

export default defineSandboxAgent({
  name: "my-cli-agent",
  evidenceCoverage: completeEvidenceCoverage,
  ensure: {
    identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" },
    probe: command("my-agent", ["--version"]),
  },
  async send(input, ctx) {
    await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
    return { status: "completed", events: [] };
  },
});

Register it

import { defineExperiment } from "niceeval";
import echo from "./agents/echo";

export default defineExperiment({
  agent: echo,
  attempts: 1,
});

Agent and Adapter Context fields

The construction parameters of defineAgent (DirectAgentDef) and defineSandboxAgent (SandboxAgentDef), plus the ctx (AgentContext) that both receive in send(input, ctx): A custom Adapter’s create(ctx) uses AdapterCreateContext, and its cleanup callback uses AdapterCleanupContext.

`AdapterAssertionsFactoryContext`

`app`

readonly app: MatchContext<Context>;

`check`

readonly check: AssertionCheck<"polymorphic", Context>;

`AdapterCleanupContext`

`timeoutMs`

readonly timeoutMs: number;

A fixed millisecond budget for the whole cleanup window; it is not renewed per callback.

`deadlineAt`

readonly deadlineAt: number;

The Unix millisecond deadline fixed when the cleanup window actually starts.

`signal`

readonly signal: AbortSignal;

Independent of the execution signal; aborts when the whole Adapter cleanup window reaches its deadline.

`AdapterCreateContext`

`models`

readonly models: Readonly<Record<Roles, ResolvedModelRole>>;

Frozen application model selections for this Attempt.

`recordUsage`

recordUsage(input: Omit<import("./adapter-usage.ts").AdapterUsageInput, "modelRole"> & { readonly modelRole?: Roles | null }): void;

`sealUsage`

sealUsage(input: import("./adapter-usage.ts").AdapterUsageSeal): void;

Close application usage admission and declare whether the recorded collection is complete.

`attach`

attach(input: import("./adapter-attachments.ts").AdapterAttachmentInput): Promise<import("./adapter-attachments.ts").AdapterAttachmentReceipt>;

`recordTrace`

recordTrace<Event extends import("./adapter-execution-trace.ts").ExecutionTraceEvent>(
  input: import("./adapter-execution-trace.ts").ExecutionTraceInput<Event>,
): Promise<import("./adapter-execution-trace.ts").ExecutionTraceReceipt>;

`evalId`

readonly evalId: string;

The public ID of the current Eval.

`experimentId`

readonly experimentId: string;

The public ID of the current Experiment.

`attempt`

readonly attempt: number;

The zero-based index of the current Attempt.

`signal`

readonly signal: AttemptSignal;

The Attempt execution signal; its reason distinguishes timeout from cancelled, and it is not used for the cleanup window.

`model`

readonly model?: string;

The model selected by the Experiment.

`reasoningEffort`

readonly reasoningEffort?: string;

The reasoning effort selected by the Experiment.

`flags`

readonly flags: Flags;

The read-only flags the Experiment passes to the Adapter.

`progress`

progress(update: ProgressUpdate): void;

Updates the human-readable progress of the current Attempt.

`diagnostic`

diagnostic(input: DiagnosticInput): void;

Appends a structured diagnostic to the current Attempt.

`log`

log(message: string): void;

Appends a log line to the current Attempt.

`onCleanup`

onCleanup(cleanup: (context: AdapterCleanupContext) => void | Promise<void>): void;

Registers release of an Attempt-local resource. Callbacks run in global LIFO order; one failure does not skip the rest. Zero-argument callbacks can still be passed directly. The context passed in is frozen, and callbacks in the same cleanup window share one signal.

`DirectAgentDef`

`name`

name: string;

The agent’s display name/identifier, copied verbatim into Agent.name. It is not a registry lookup key; it is used only for display, result attribution, and the dedupe fingerprint.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This Adapter’s normal evidence coverage declaration; use completeEvidenceCoverage for complete collection.

`before`

before?: DirectAgentBefore;

Once per attempt. A Direct Agent does not receive a Sandbox; commonly used for one-time preparation such as opening connections and authentication.

`tracing`

tracing?: Omit<AgentTracing, "configure">;

OTLP export configuration: how the system under test sends traces to the endpoint (env-based injection).

`spanMapper`

spanMapper?: SpanMapper;

A thin mapper from native spans → canonical; omitted uses the generic heuristic. Affects only the waterfall.

`send`

send(input: TurnInput, ctx: AgentSendContext): Promise<Turn>;

Once per turn: sends one turn’s prompt directly to a function, SDK, or service endpoint, and parses the response into events. The author callback keeps the Promise ABI.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional send execution failure classifier: see Agent.classifySendFailure.

`after`

after?: DirectAgentAfter;

Cleanup before the run ends, executed if and only if this attempt reached the before point (a before throw does not exempt it), run once in finally.

`SandboxAgentDef`

`name`

name: string;

The agent’s display name/identifier, copied verbatim into Agent.name. It is not a registry lookup key; it is used only for display, result attribution, and the dedupe fingerprint.

`evidenceCoverage`

evidenceCoverage: EvidenceCoverage;

This Adapter’s normal evidence coverage declaration; use completeEvidenceCoverage for complete collection.

`sandbox`

sandbox?: SandboxLayer<"command-only">;

The Adapter’s own Sandbox actions; an Agent cannot choose or contribute a template.

`install`

readonly install: AgentInstall | readonly AgentInstall[];

A single entry or an array is normalized into Agent layers in declaration order.

`installers`

installers?: readonly AgentInstaller[];

Paired installation layers; omitted means this adapter provides only the probe protocol, and on a miss the Runner names the exact missing identity at agent.install. The factory normalizes an omission into an empty array.

`before`

before?: AgentBefore;

Once per Attempt (not once per send): writes config.toml / authentication configuration (model/base/auth and other runtime configuration that does not change within this Attempt). Probing, installing, and rechecking the CLI belong to agent.install. The runner calls it once after the Sandbox is ready (after layer prepare/install/baseline) and before the first send; it returns no value.

`tracing`

tracing?: AgentTracing;

OTLP export configuration: how to make the CLI in the Sandbox send traces to the endpoint (env / config file), split out of before.

`spanMapper`

spanMapper?: SpanMapper;

A thin mapper from native spans → canonical; omitted uses the generic heuristic. Affects only the waterfall.

`send`

send(input: TurnInput, ctx: SandboxAgentSendContext): Promise<Turn>;

Once per turn: runs the prompt (fresh / resume) and parses it into events. The author callback keeps the Promise ABI.

`classifySendFailure`

classifySendFailure?: SendFailureClassifier;

Optional send execution failure classifier: see Agent.classifySendFailure.

`after`

after?: AgentAfter;

Cleanup before the Sandbox is destroyed, executed if and only if this attempt reached the before point (a before throw does not exempt it), run once in finally.

`AgentContext`

`signal`

readonly signal: AbortSignal;

Soft cancellation signal: combines the attempt timeout, run-level interruption (user Ctrl+C), and the Eval’s own interruption request (see src/runner/attempt.ts). An adapter can optionally check it (or pass it straight to fetch) to exit gracefully early, but it is not the only hard boundary: even if the adapter ignores it entirely, the runner uses Effect.timeoutTo as a backstop to force after (stopping the Sandbox container).

`model`

readonly model?: string;

The model name used by this attempt, passed through from the Experiment’s model field. Sandbox agents usually write it into configuration in before; direct agents usually use it in send to select the model.

`reasoningEffort`

readonly reasoningEffort?: string;

The model’s reasoning effort; owned the same way as model: the Experiment decides it, and when omitted the agent’s native default is not overridden.

`flags`

readonly flags: ExperimentFlags;

The Experiment’s flags field passed through verbatim; its content and structure are entirely defined by the Experiment author (such as { webResearch: true }, { systemPrompt: "..." }). The adapter reads its fields by its own convention; the framework itself neither interprets nor validates the content. The name deliberately avoids the CLI-parsed flag (run-level —timeout/—budget and so on); the two are unrelated concepts.

`experimentId`

readonly experimentId?: string;

The Experiment id derived from the path (the same source as result attribution runWho / AgentRun.experimentId); undefined when not running through an Experiment (such as outside the CLI, constructing AgentRun directly). Typical uses: a SandboxLayer command isolating cross-attempt infrastructure content per Experiment, or an adapter switching authentication / routing per Experiment. It is a different dimension from flags (the concrete values of Experiment conditions): this is only a stable identifier of “which Experiment is running” and carries no condition content.

`evalId`

readonly evalId?: string;

The eval ID of the current Attempt. The NiceEval Runner always fills it from the Eval identity after discovery; third parties constructing AgentContext directly can omit it. An Adapter can use it to locate read-only host assets in the same directory as the task, but must not use it to bypass the Sandbox’s hidden-criteria isolation.

`evalGroup`

readonly evalGroup?: {
  readonly id: string;
  readonly definitionHash: string;
};

The eval group of the current Attempt; omitted for ungrouped Evals.

`attempt`

readonly attempt?: AttemptRef;

The current Attempt reference filled in by the Runner; third parties constructing the context directly can omit it.

`session`

readonly session: AgentSession;

`telemetry`

readonly telemetry?: Telemetry;

Present only when OTel integration is configured (the agent’s tracing block / the config’s telemetry exists): this run’s OTLP traces receiver information (endpoint + env for env-based export). How to hand it to the CLI is declared by the agent’s tracing block: env-based spreads ctx.telemetry.env into send; file-based writes configuration in tracing.configure. A remote HTTP integration’s send only needs to spread headers into the request headers (a new traceparent per turn); the endpoint is fixed by startup configuration (defineConfig({ telemetry: { port } })) and is not passed from here.

`progress`

progress(update: ProgressUpdate): void;

Scoped feedback: reports what is happening right now (turn / tool / install progress). Short-lived state: the Human profile updates the active line, agent/ci do not print each one, and it does not enter the final result; do not call it for every token/delta. The runner attributes it by the lifecycle phase the callback is currently in (agent.before / agent.run / agent.after); the caller cannot impersonate another phase (see docs/feature/experiments/library/README.md).

`diagnostic`

diagnostic(input: DiagnosticInput): void;

Scoped feedback: reports problems that should remain after the run ends (protocol downgrade, incomplete data, cleanup issues). A permanent event that lands in the attempt’s diagnostics and each profile’s permanent output; deduplicated by dedupeKey. Even with level “error” it does not change Turn.status / verdict; throw when you cannot continue.

`log`

log(msg: string): void;

An explicit timeout breadcrumb. It also updates the Human active line, and the most recent entries are merged into the result’s error information on a timeout failure; do not use it to carry user messages, tool input, or other text meant only for short-lived display. ctx.session (AgentSession) stores the state for one session line: every send on the same session line gets the same ctx.session, and a new session line (the Eval’s first turn / after t.newSession()) gets a fresh one. Accessors:

id?: string / capture(id): void — for session continuation when the server keeps history: id is the session id this line has recorded (undefined on a new line), and capture records the returned id (it takes effect only if none has been recorded yet).
createSessionKey<T>(name) — creates a typed key at Adapter module scope. Keys are isolated by symbol identity.
get(key) / set(key, value) — store and read client-side history or Adapter-private state.
take(key) — reads and deletes the HITL pause state, consumed once.

For the complete contract, see Adapter concepts.

Sandbox interface

ctx.sandbox (Sandbox) of a Sandbox agent is the handle to the current isolated environment. Sandbox extends SandboxOperations and SandboxTransferOperations; all members are listed below by declaring interface. CommandOptions holds the options for runCommand / runShell:

`SandboxOperations`

`workdir`

readonly workdir: string;

The absolute path of the project/workspace root inside the Sandbox (the default cwd for agent commands, and where the git baseline is committed). Relative paths in every method resolve against it, and omitting cwd/targetDir also lands here.

`runCommand`

runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise<CommandResult>;

Runs a single command; args are passed as separate argv entries and are not interpreted by a shell (no &&, pipes, or glob expansion). Prefer it when you only want to run one executable, arguments come from external input, or you are worried about injection.

`runShell`

runShell(script: string, opts?: CommandOptions): Promise<CommandResult>;

Runs a whole script interpreted by a shell (bash), supporting &&, pipes, $(), redirection, and so on. Use it when you need to chain several commands or branch on conditions.

`runCommandOrThrow`

runCommandOrThrow(
  cmd: string,
  args?: readonly string[],
  opts?: CommandOptions,
): Promise<SuccessfulCommandResult>;

Runs a single command like runCommand, but throws SandboxCommandExitError on a non-zero exit. The error message includes a bounded, cleaned, and redacted stderr tail; it falls back to stdout when stderr is empty. The full output is kept in the exception’s result field. A successful result’s exitCode is fixed to 0 in its type.

`runShellOrThrow`

runShellOrThrow(script: string, opts?: CommandOptions): Promise<SuccessfulCommandResult>;

Runs a shell script like runShell, but throws SandboxCommandExitError on a non-zero exit. The error summary, full output, and successful result semantics are the same as runCommandOrThrow.

`readText`

readText(path: string): Promise<string>;

Reads the text content (UTF-8) of a file inside the Sandbox. Throws when the file does not exist.

`writeText`

writeText(path: string, content: string): Promise<void>;

Writes a UTF-8 text file inside the Sandbox; missing parent directories are created automatically.

`readBytes`

readBytes(path: string): Promise<Uint8Array>;

Reads a file’s exact bytes inside the Sandbox; the public contract is not bound to Node Buffer.

`writeBytes`

writeBytes(path: string, content: Uint8Array): Promise<void>;

Writes a file’s exact bytes inside the Sandbox; missing parent directories are created automatically.

`pathExists`

pathExists(path: string): Promise<boolean>;

Checks whether a file or directory path exists inside the Sandbox.

`SandboxTransferOperations`

`upload`

upload(content: SandboxContent, targetPath: string): Promise<void>;

Transfers registered immutable content without exposing its host path.

`uploadFile`

uploadFile(source: string | URL, targetPath: string): Promise<void>;

`uploadDirectory`

uploadDirectory(
  sourceDir: string | URL,
  targetDir?: string,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

`downloadFile`

downloadFile(sourcePath: string, target: string | URL): Promise<void>;

`downloadDirectory`

downloadDirectory(
  sourceDir: string,
  targetDir: string | URL,
  options?: { readonly ignore?: readonly string[] },
): Promise<void>;

`Sandbox`

`stop`

stop(): Promise<void>;

Destroys the compute resources the Sandbox occupies (container/microVM). The Sandbox cannot be used after the call; whether repeated calls are safe varies by provider, so do not rely on it.

`sandboxId`

readonly sandboxId: string;

This Sandbox’s stable identifier (each provider’s native ID, such as a Docker container ID prefix); used to correlate session state of the same Sandbox across calls, and for log display.

`nativeInstanceId`

readonly nativeInstanceId?: string;

Provider-native instance selector for read-only diagnostics; absent when unavailable.

`otlpHost`

readonly otlpHost: string | null;

The placement capability of the OTLP receiver.

string: the host receiver is reachable from inside the Sandbox through this hostname.
null: the provider does not promise a connection back to the host; the runner tries to start an attempt-scope receiver inside the Sandbox. This does not guarantee tracing succeeds; when the image lacks the runtime the receiver needs, only a supplemental diagnostic is recorded. defineConfig({ telemetry: { host } }) can override it explicitly when the author already provides a controlled tunnel.

`appendLog`

appendLog?(line: string): Promise<void>;

Optional: writes a line into the container’s “main log” (which PID1 tails), so docker logs / the Logs tab of the Docker UI shows the agent’s turn-by-turn activity live. Implemented by the docker provider; others can omit it.

`CommandOptions`

`sensitiveValues`

readonly sensitiveValues?: readonly string[];

Sensitive plaintext this command is known to handle (such as API keys, tokens, HTTP header values). The Runner still hands the original values to the provider for execution, but replaces these values exactly before any timing / commands / execution / error evidence is persisted; this array itself is never persisted and does not enter the fingerprint. Empty strings are ignored. This is explicit provenance, not a secret scanner: unregistered free text cannot be reliably identified; if the caller encodes or splits a value first, also register the encoded form that will actually appear in the command/output.

`env`

readonly env?: Readonly<globalThis.Record<string, string>>;

Adds/overrides environment variables for this command (layered on the Sandbox default environment without clearing defaults). PATH is a Sandbox managed variable; each provider keeps the PATH it computed and does not guarantee it can be overridden here; to extend PATH, use the Sandbox factory’s pathPrepend (see docs/feature/sandbox/library/README.md “PATH: managed variable and pathPrepend”).

`cwd`

readonly cwd?: string;

This command’s working directory; omitted lands on Sandbox.workdir. Relative paths resolve against the workdir, and absolute paths are used as is.

`stream`

readonly stream?: boolean;

Also sends this command’s output into the Sandbox’s “native log stream” (so docker logs / the Logs tab of the Docker UI shows it live). Turn it on for agent commands (codex exec / bub run / claude) to see the agent’s raw output in the container log. Each provider implements it (docker: tee into the file PID1 tails; unsupported providers ignore it); how the log surfaces is the provider’s business, and the adapter only declares intent.

`onStdout`

readonly onStdout?: (chunk: string) => void | Promise<void>;

Called once for each stdout chunk of the command. The callback is only for short-lived feedback while running; the full stdout still appears verbatim in the returned CommandResult. When the provider does not support real streaming, it calls it at least once with the full stdout after the command ends; it must not drop it silently.

`onStderr`

readonly onStderr?: (chunk: string) => void | Promise<void>;

The stderr counterpart of onStdout; the full stderr is still kept in CommandResult.

`user`

readonly user?: string;

Overrides the execution identity of this command; omitted = the Sandbox default identity (the identity the environment declares itself: the Docker image USER, the Compose service user:, the E2B template default user; see docs/feature/sandbox/library/README.md “Execution identity”). The semantics are consistent across providers, and each provider maps them to its own native mechanism (docker: exec --user; E2B: { user }; Vercel: accepts only "root", mapped to { sudo: true }, other values error). Providers that run as root throughout treat it as a no-op; providers that cannot switch identity at all may not support it (and throw), but the semantics of omission and explicit values stay consistent and do not vary by provider.

`timeoutMs`

readonly timeoutMs?: number;

This command’s own limit (milliseconds). Omitting it is the norm: when omitted, the limit = the remaining attempt deadline (see docs/feature/sandbox/architecture/README.md “Deadline ownership”); the provider layer has no independent default. Passing a shorter value explicitly is an intentional declaration and takes effect as usual; on hitting it, the attribution is recorded as “explicit command timeout”.

`signal`

readonly signal?: AbortSignal;

Cancels this managed command tree. The provider must confirm the command tree has terminated before the Promise settles; when it cannot terminate precisely, it should retire the whole Sandbox rather than only closing the transport and leaving processes in the background.

ctx and t

ctx is the run context the adapter side sees. t is the test context the Eval author sees. Both use the same run data, but have different responsibilities.

defineAgent

Direct agent example

input (TurnInput)

defineSandboxAgent

Register it

Agent and Adapter Context fields

AdapterAssertionsFactoryContext

app

check

AdapterCleanupContext

timeoutMs

deadlineAt

signal

AdapterCreateContext

models

recordUsage

sealUsage

attach

recordTrace

evalId

experimentId

attempt

signal

model

reasoningEffort

flags

progress

diagnostic

log

onCleanup

DirectAgentDef

name

evidenceCoverage

before

tracing

spanMapper

send

classifySendFailure

after

SandboxAgentDef

name

evidenceCoverage

sandbox

install

installers

before

tracing

spanMapper

send

classifySendFailure

after

AgentContext

signal

model

reasoningEffort

flags

experimentId

evalId

evalGroup

attempt

session

telemetry

progress

diagnostic

log

Sandbox interface

SandboxOperations

workdir

runCommand

runShell

runCommandOrThrow

runShellOrThrow

readText

writeText

readBytes

writeBytes

pathExists

SandboxTransferOperations

upload

uploadFile

`defineAgent`

`input` (`TurnInput`)

`defineSandboxAgent`

`AdapterAssertionsFactoryContext`

`app`

`check`

`AdapterCleanupContext`

`timeoutMs`

`deadlineAt`

`signal`

`AdapterCreateContext`

`models`

`recordUsage`

`sealUsage`

`attach`

`recordTrace`

`evalId`

`experimentId`

`attempt`

`signal`

`model`

`reasoningEffort`

`flags`

`progress`

`diagnostic`

`log`

`onCleanup`

`DirectAgentDef`

`name`

`evidenceCoverage`

`before`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`after`

`SandboxAgentDef`

`name`

`evidenceCoverage`

`sandbox`

`install`

`installers`

`before`

`tracing`

`spanMapper`

`send`

`classifySendFailure`

`after`

`AgentContext`

`signal`

`model`

`reasoningEffort`

`flags`

`experimentId`

`evalId`

`evalGroup`

`attempt`

`session`

`telemetry`

`progress`

`diagnostic`

`log`

`SandboxOperations`

`workdir`

`runCommand`

`runShell`

`runCommandOrThrow`

`runShellOrThrow`

`readText`

`writeText`

`readBytes`

`writeBytes`

`pathExists`

`SandboxTransferOperations`

`upload`

`uploadFile`

`uploadDirectory`

`downloadFile`

`downloadDirectory`

`Sandbox`