Adapter: connect an Agent to NiceEval

In NiceEval, an Adapter is the integration layer between the Runner and the subject under test. An Adapter implements send: it sends Eval-side actions such as t.send() to your application, then translates the application’s response into NiceEval’s standard Turn. For the next step, read Write Send.

Adapters

If the subject under test uses the standard OpenAI Chat Completions or Responses protocol, use an official Adapter directly. For a custom frontend/backend protocol—HTTP, gRPC, or WebSocket all work—write an Adapter. It knows how to authenticate, call your application, and translate the response into the standard event stream. One t.send() round trip crosses this boundary twice: On the way out, an Eval-side drive verb—t.send, t.sendFile, or t.respond—builds a TurnInput and the session line’s ctx, then calls the Adapter’s send once. On the way back, the Adapter translates the application’s raw response into a Turn. Its events are an array of event objects in their actual order; examples appear below. Pass the subject’s URL, authentication, and protocol details into an Adapter factory. defineExperiment receives an already configured Agent instance. The Adapter consumes that configuration in its send closure, then forwards dynamic per-turn values such as ctx.model, ctx.flags, and ctx.telemetry with the request:

// agents/web-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

interface WebAgentOptions {
  baseUrl: string;
  apiKey?: string;
}

export function webAgent(options: WebAgentOptions): Agent {
  const baseUrl = options.baseUrl.replace(/\/$/, "");

  return defineAgent({
    name: "web-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      ctx.progress({ message: "Waiting for Web Agent" });
      const response = await fetch(`${baseUrl}/api/turn`, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          ...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}),
          ...ctx.telemetry?.headers,
        },
        body: JSON.stringify({
          message: input.text,
          files: input.files,
          model: ctx.model,
          flags: ctx.flags,
        }),
        signal: ctx.signal,
      });

      if (!response.ok) {
        ctx.diagnostic({
          code: "agent-http-error",
          level: "error",
          message: `Web Agent returned HTTP ${response.status}`,
          data: { status: response.status },
        });
        return {
          status: "failed",
          events: [{ type: "error", message: `HTTP ${response.status}` }],
        };
      }

      const body = await response.json();
      return {
        status: "completed",
        events: body.events,
        data: body.data,
        usage: body.usage,
      };
    },
  });
}

// experiments/staging.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";

export default defineExperiment({
  agent: webAgent({
    baseUrl: "https://staging.example.com",
    apiKey: process.env.STAGING_AGENT_API_KEY,
  }),
  model: "gpt-5.4",
  flags: { promptVariant: "concise" },
});

Integration tiers

Integration has three tiers based on where the Adapter connects and what additional observability data it receives: Tier 1 connects only send—without changing application code, and with the complete assertion set already available; Tier 2 is send + OTel—the application sends spans to NiceEval in return for the call waterfall in niceeval view; and Tier 3 changes the application and adds Experiment flags for feature A/B testing. See Tier for each tier’s investment, capabilities, and when to move up. Every Eval-side drive API—t.send(), t.sendFile(), t.newSession(), and the HITL methods t.respond() / t.respondAll()—ultimately calls the Adapter’s send. For how send receives and responds, see Write Send.

The `send` function

Whether an Adapter connects to an HTTP service or a CLI inside a Sandbox, the interface it exposes to the Runner is identical:

interface Agent {
  readonly name: string;
  send(input: TurnInput, ctx: AgentContext): Promise<Turn>;
  // Optional: setup / teardown (the lifecycle that runs once per Sandbox).
  // Optional observability members do not participate in send; see “OTel Integration”.
}

progress is short-lived runtime state and is not persisted. diagnostic is a bounded warning or error that needs review after the run and is saved with the Attempt. Neither can specify a lifecycle phase, and neither automatically changes Turn.status. Throw an exception when the connection fails or the response cannot be parsed. The Runner records it as errored and gives the terminal an inspectable locator. send is the only function you need to implement. Its signature has three types, explained separately below.

The setup / teardown here are private to how an Agent connects itself, such as installing a CLI or writing authentication configuration. A Sandbox Agent can also contribute command-only .before() actions from its own sandbox field, such as writing the Adapter’s public .env. An Experiment or Eval must still select the one Provider template. See Sandbox providers — Prepare the Sandbox in a deterministic order for ordering and cache rules.

Input: `TurnInput`

interface TurnInput {
  readonly text: string;                          // Present on every turn: text to send. A response turn joins multiple answers with newlines.
  readonly files?: readonly InputFile[];          // Only t.sendFile turns: attached files (base64)
  readonly responses?: readonly InputResponse[];  // Only response turns (t.respond / t.respondAll): one structured answer per request
  readonly outputSchema?: JsonSchema;             // Only when this turn declared output: the structured-output requirement, lowered to JSON Schema
}

interface InputFile {
  readonly filename?: string;     // Optional, for an Adapter or model to inspect
  readonly mimeType: string;      // For example, image/png
  readonly dataBase64: string;    // Base64 content; JSON-friendly and ready for a request body
}

interface InputResponse {
  readonly requestId: string;    // Required: identifies the input.requested request; aligns several simultaneous requests
  readonly optionId?: string;    // Mutually exclusive with text: an id in the request's options (approve / deny …)
  readonly text?: string;        // Mutually exclusive with optionId: a free-text answer when the request has no options or no option matches
}

Whether the Eval calls t.send(), t.sendFile(), or t.respond(), the Adapter receives an ordinary send. The fields are what differ: If your application interface does not accept files, ignore files.

Eval side	`text`	`files`	`responses`	`outputSchema`
`t.send(text)`	The sent text	—	—	Present only when `output` was declared
`t.sendFile(path, text?)`	An optional note, which can be an empty string	Present	—	Same
`t.respond(...)` / `t.respondAll(...)`	Answer text	—	One per request	Same

Ignore files if the application interface does not accept files. Forward outputSchema when the interface accepts a schema, such as Chat Completions response_format or Responses text.format. It is also fine when it does not: the Runner validates the result in any case, as described in the Turn.data rules below.

Inputs for the different answers

In a HITL response turn, a person’s decision arrives structurally in input.responses. The Adapter does not need to parse text to guess which sentence answers which request or whether it approves something. Each answer has requestId. optionId and text are mutually exclusive: if an answer matches an id in the request’s options, it uses optionId; the Eval has already verified that it exists, so a typo cannot silently pass into the application. Otherwise the full answer is in text. The four typical forms are:

// ① One pending request; the answer matches an option (approve / deny differ only by optionId)
await t.respond("approve");
// → { text: "approve",
//     responses: [{ requestId: "req_1", optionId: "approve" }] }

// ② Several requests pause together; the object form aligns the response explicitly
await t.respond({ request, optionId: "deny" });
// → { text: "deny",
//     responses: [{ requestId: request.id, optionId: "deny" }] }

// ③ The answer matches no option → free text (a request for more information has this form)
await t.respond("Change the recipient to ceo@corp.com");
// → { text: "Change the recipient to ceo@corp.com",
//     responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] }

// ④ respondAll: one answer per pending request, all with the same optionId
await t.respondAll("approve");
// → { text: "approve\napprove",
//     responses: [{ requestId: "req_1", optionId: "approve" },
//                 { requestId: "req_2", optionId: "approve" }] }

On the Adapter side, return each decision to the application by requestId; do not guess by order. A call rejected by a person sets the tool operation.finished status to "rejected", not "failed". Rejection is a human decision, not a tool failure. That keeps noFailedActions() from being triggered incorrectly, while calledTool(toolMatch(..., { status: "rejected" })) can assert it precisely.

Context: `AgentContext`

interface AgentContext {
  // Feedback and runtime context present on every turn
  readonly signal: AbortSignal;    // Runner timeout and cancellation: forward it to every request you make
  readonly session: AgentSession;  // State slots for this session line
  progress(update: { message: string; current?: number; total?: number }): void;
  diagnostic(input: {
    code: string;
    level: "warning" | "error";
    message: string;
    data?: Readonly<Record<string, JsonValue>>;
    dedupeKey?: string;
  }): void;

  // Three fields that appear by integration tier (see “Integration tiers”)
  readonly model?: string;         // Tier 1 model comparison: experiment.model passed through. Forward it if the application interface accepts model selection.
  readonly reasoningEffort?: string;  // Reasoning effort: experiment.reasoningEffort passed through, with the same role as model
  readonly telemetry?: Telemetry;  // Tier 2: this turn's W3C trace context (headers, with a new traceparent each turn)
  readonly flags: Readonly<Record<string, JsonValue>>; // Tier 3 feature A/B: experiment.flags passed through. {} when unspecified
  readonly experimentId?: string;  // A stable ID derived from the path, shared with experimentId in results. undefined outside an Experiment

}

interface SandboxAgentContext extends AgentContext {
  readonly sandbox: Sandbox;       // Available only to an Agent constructed by defineSandboxAgent
}

interface AgentSession {
  // Server-side continuation: the shape for an interface that accepts a session id
  readonly id?: string;                      // The session id remembered for this line. undefined on a new session line
  capture(id: string | undefined): void;     // Save a returned id, only if none was saved already

  // Adapter-private state: client history, a paused HITL turn, and similar values
  get<T>(slot: SessionSlot<T>): T | undefined;
  set<T>(slot: SessionSlot<T>, value: T): void;
  take<T>(slot: SessionSlot<T>): T | undefined; // Read and clear: one consumption
}

The Runner binds a lifecycle scope separately for Agent setup, every send, and teardown. An Adapter reports progress and diagnostics only from the current callback; it cannot supply a phase, color, or output stream. progress is not persisted. diagnostic is published to the Record as an Attempt-owned diagnostic-channel event. Throw when work cannot continue; the Runner saves a structured error and produces an errored Verdict in the niceeval.verdict channel. An Attempt lifecycle state is only active, completed, or abandoned; a Verdict token is not an Attempt state. ctx has no flag for you to inspect. The three tier fields have pass-through semantics: model and flags are declared by the Experiment and passed through unchanged by the Runner. The Adapter only forwards them with the request; it does not interpret them. telemetry appears only when OTel integration is configured. In send, spread its headers into request headers. The receiving endpoint is fixed in defineConfig and pointed to when the application starts; it is not supplied here. See OTel Integration. experimentId is a stable identity derived from the path. A common use is to isolate state shared across Attempts by Experiment in a Sandbox callback, for example by partitioning a cache directory or checkpoint key. See Sandbox providers — Prepare the Sandbox in a deterministic order. session is the state owned by this session line. NiceEval makes one promise: every send on the same session line receives the same ctx.session; a new session line, either the Eval’s first turn or one created by t.newSession(), receives a new one. Continue server-side history with id / capture. For client-side history and paused HITL state, create a typed slot in Adapter module scope with createSessionSlot<T>(name), then use get / set / take. take clears as it reads. Slots are isolated by symbol identity even when their names match.

Return: `Turn`

interface Turn {
  readonly events: StreamEvent[];          // ★ The standard event stream: every Adapter's core result
  readonly data?: JsonValue;               // This turn's structured result. Fill it only when input.outputSchema exists; the Runner validates it against the declaration
  readonly status: "completed" | "failed" | "waiting";  // waiting = a paused turn awaiting a person (HITL)
  readonly usage?: Usage;
}

events is an ordinary JavaScript array. Each thing that happens in the turn is one object, in the order it actually happened. For example, in a turn that asks “What is the temperature in Beijing today?”, the Agent looks up weather and then answers:

[
  { type: "operation.started", operationId: "c1",
    operation: { kind: "tool", name: "get_weather", input: { city: "Beijing" } } },
  { type: "operation.finished", operationId: "c1", kind: "tool",
    output: { temp: 21 }, status: "completed" },
  { type: "message", role: "assistant", text: "Beijing is 21°C today." },
]

There are ten variants in total—message, operation.*, input.requested, and others. See Event Stream Reference for the full list. NiceEval derives managed logical tool occurrences and event occurrences from this array. t.calledTool("get_weather") matches the tool collection, while t.reply takes the last assistant message. Each send returns only this turn’s array. The Runner combines turns into a complete session line; see the next section. Most of the time you do not write these objects by hand: an official converter returns a complete Turn with events, usage, and status already populated. See Write Send for choosing a converter. data is not a pocket for arbitrary values. It has one rule: the Eval declares what it needs through a schema on send. data exists only after that declaration, and it has the declared type when you receive it.

// On the Eval side, output is both an application requirement and the type source for turn.data
const turn = await t.send("What is the total amount on this invoice?", {
  output: z.object({ amount: z.number(), currency: z.string() }),
});
turn.data.amount;   // type: number, not unknown; no manual narrowing needed

The declaration is lowered to JSON Schema and reaches the Adapter as input.outputSchema. After the Adapter receives an application response, it puts the structured result in data, for example JSON.parse(reply.content). The Runner enforces validation: if data does not match the declaration, the turn is immediately failed and reports the difference. There is no path where an Adapter puts in the wrong object and an assertion silently passes. Conversely, a turn that did not declare output has no data. An application that returns only text never needs this field; do not copy the raw response body into it to fill space.

After `send` returns: where the four `Turn` fields go

Once send returns, the Adapter’s work is done. The Runner owns everything after that. Each field has a clear destination:

`Turn` field	What the Runner does with it	Where an Eval author sees it
`events`	Appends it to this session line and normalizes messages, logical tool occurrences, and supported event occurrences. The last assistant `message` also updates `t.reply`.	Managed `toolCalls` / `eventOccurrences` Matches; raw `events` remains directly available as a Value subject
`status`	Records this turn’s terminal state. When it is `"waiting"`, it collects the turn’s `input.requested` items as pending answers.	`t.succeeded()` or `t.check(turn.status, equals("waiting"))`; use `t.respond()` after `"waiting"`
`usage`	Adds it per turn to the session line and the whole run.	`maxTokens` / `maxCost` assertions and usage in reports
`data`	Validates it against the `output` schema declared on `send`; a mismatch immediately makes the turn `failed`. It saves only a passing value.	`turn.data`, strongly typed by the declaration; `t.check(turn.data, equals(...))` or `matches(...)`

The handle returned by await t.send() is your view of that turn. Turn-level assertions see only that turn’s events and fields; root t assertions see all events accumulated on the session line. The Adapter does not need to cooperate with individual assertions: correctly fill these four fields and assertions, scoring, and reports happen downstream automatically.

Where capabilities come from: proof by construction, not a questionnaire

defineAgent constructs a Direct Agent. The Runner calls its function, SDK, or service endpoint directly; the target can be local or remote. A Direct Agent’s setup, send, and teardown do not receive a Sandbox. defineSandboxAgent constructs a Sandbox Agent; only its callbacks receive a SandboxAgentContext with a real sandbox.

What becomes available on `t`	What proves it	What you write
`t.send`, `t.sendFile`, `t.check`, …	Every Agent has it	Nothing
Multiple `t.send()` calls, `t.reply`, `t.newSession()`	`send` uses a `ctx.session` continuation accessor, either a typed slot or `id` + `capture()`. Without it, each turn is a new conversation.	Nothing—the accessor is already on `ctx`
`t.respond()`, status checks, and other HITL APIs	`send` has returned `"waiting"` plus an `input.requested` event	Nothing—doing it is having it
Positive assertions such as `turn.calledTool()` / `turn.toolOrder()`	`events` contains paired `operation.started` / `operation.finished` items	Nothing—events make the assertion possible
Negative assertions such as `t.notCalledTool()` / `t.usedNoTools()` are trustworthy	Events come from an official converter that proves completeness	Nothing—the proof travels with the converter result
`t.sandbox`, `t.sandbox.fileChanged()`, and similar APIs	Construct with `defineSandboxAgent`	Nothing
Trace channel and the `niceeval view` waterfall	A `tracing` block exists; spans go only to the waterfall, not assertions	Nothing—the block already needs to exist

The negative-assertion row is the only one with levels. notCalledTool claims that something did not happen, which requires complete events. That is trustworthy only when the source has a completeness contract, such as direct SDK event forwarding, AI SDK result.steps, or Responses output. A handwritten mapping has no proof. A negative assertion then warns at runtime that it is untrustworthy instead of silently passing. There is therefore no mismatch where an author declares a capability but cannot provide it: trust follows the event source, not author intent. See Capability Reference for exact obligations by capability.

Write Send — The practical tutorial: from sending one message to a complete integration in seven incremental steps, each unlocking a set of assertions.
Connect Your Agent — The integration overview: minimal integration, parameter channels, and the incremental map.
Drive — The Eval-side view: how to use t.send(), t.newSession(), and HITL.
Assert — The full assertion vocabulary driven by the standard event stream.
Architecture Overview — The four-layer architecture and its boundaries.

Core Model

Interaction & Assertions

Execution Model

Adapter: connect an Agent to NiceEval

Adapters

Integration tiers

The `send` function

Input: `TurnInput`

Inputs for the different answers

Context: `AgentContext`

Return: `Turn`

After `send` returns: where the four `Turn` fields go

Where capabilities come from: proof by construction, not a questionnaire

​Adapters

​Integration tiers

​The send function

​Input: TurnInput

​Inputs for the different answers

​Context: AgentContext

​Return: Turn

​After send returns: where the four Turn fields go

​Where capabilities come from: proof by construction, not a questionnaire

​Related reading

Adapters

Integration tiers

The `send` function

Input: `TurnInput`

Inputs for the different answers

Context: `AgentContext`

Return: `Turn`

After `send` returns: where the four `Turn` fields go

Where capabilities come from: proof by construction, not a questionnaire

Related reading