Adapter: connect an Agent to NiceEval

In NiceEval, an Adapter is the integration layer between the Runner and the system under test. An Adapter implements the send function: it sends eval-side actions such as t.send() to your application, then translates the application’s response into NiceEval’s standard Turn. Read next: Write send

Adapters

If the system under test uses the standard OpenAI Chat Completions or Responses protocol, use an official Adapter directly. For your own frontend/backend protocol (HTTP, gRPC, or WebSocket all work), write an Adapter. It knows how to authenticate, how to call your application, and how to translate the response into the standard event stream. One complete t.send() round trip crosses this boundary twice: On the way out: a drive verb in the eval case (t.send / t.sendFile / t.respond) builds the TurnInput and this session line’s ctx, then calls the Adapter’s send once. On the way back: the Adapter translates the application’s raw response into a Turn. Its events are an array of event objects in the order they happened (real values appear below). The URL, authentication, and protocol details of the system under test are passed in through the Adapter factory’s configuration. defineExperiment holds an already configured agent instance. The Adapter consumes this configuration in the send closure, then forwards per-turn dynamic values (such as ctx.model, ctx.flags, and ctx.telemetry) to the application with the request:

// agents/web-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

interface WebAgentOptions {
  baseUrl: string;
  apiKey?: string;
}

export function webAgent(options: WebAgentOptions): Agent {
  const baseUrl = options.baseUrl.replace(/\/$/, "");

  return defineAgent({
    name: "web-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      ctx.progress({ message: "Waiting for Web Agent" });
      const response = await fetch(`${baseUrl}/api/turn`, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          ...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}),
          ...ctx.telemetry?.headers,
        },
        body: JSON.stringify({
          message: input.text,
          files: input.files,
          model: ctx.model,
          flags: ctx.flags,
        }),
        signal: ctx.signal,
      });

      if (!response.ok) {
        ctx.diagnostic({
          code: "agent-http-error",
          level: "error",
          message: `Web Agent returned HTTP ${response.status}`,
          data: { status: response.status },
        });
        return {
          status: "failed",
          events: [{ type: "error", message: `HTTP ${response.status}` }],
        };
      }

      const body = await response.json();
      return {
        status: "completed",
        events: body.events,
        data: body.data,
        usage: body.usage,
      };
    },
  });
}

// experiments/staging.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";

export default defineExperiment({
  agent: webAgent({
    baseUrl: "https://staging.example.com",
    apiKey: process.env.STAGING_AGENT_API_KEY,
  }),
  model: "gpt-5.4",
  flags: { promptVariant: "concise" },
});

Integration tiers

Integration has three tiers based on where the Adapter connects and what extra observability data it gets: Tier 1 connects only send (no application code changes, and the full assertion set is already available at this tier), Tier 2 is send + OTel (the application sends spans to NiceEval in exchange for the call waterfall in niceeval view), and Tier 3 changes the application and adds experiment flags (feature A/B). For what each tier costs, what it provides, and when to move up, see Tier. The eval-side drive APIs, t.send(), t.sendFile(), t.newSession(), and the HITL methods t.respond() / t.respondAll(), all call the Adapter’s send. For how this converges and how to wire it inside send, see Write send.

The send function

Whether the Adapter connects to an HTTP service or a CLI inside a Sandbox, the interface it exposes to the Runner is exactly the same:

interface Agent {
  readonly name: string;
  send(input: TurnInput, ctx: AgentContext): Promise<Turn>;
  // Optional: setup / teardown (the lifecycle that runs once per Sandbox).
  // Optional observability members do not take part in send; see "OTel integration"
}

progress is short-lived runtime state and is not saved to disk. diagnostic is a bounded warning/error that needs review after the run, and it is saved with the Attempt. Neither can specify a lifecycle phase, and neither changes Turn.status automatically. Throw an exception when the connection fails or the response cannot be parsed. The Runner records it as errored and prints a locator in the terminal, which you can use to see the details of this failure. send is the only function you need to implement. Its signature has only three types, covered separately below.

The setup / teardown here are the lifecycle of how the Agent sets up its own connection. A Sandbox Agent can also contribute command-only .before() actions through its own sandbox field, for example to write the Adapter’s public .env. The Experiment or eval case selects the one Provider template. For ordering and caching, see Sandbox · Prepare the Sandbox in a deterministic order.

Input: `TurnInput`

interface TurnInput {
  readonly text: string;                          // Present on every turn: the text sent. On a response turn = the answer text (multiple joined with newlines)
  readonly files?: readonly InputFile[];          // Only on t.sendFile turns: attached files (base64)
  readonly responses?: readonly InputResponse[];  // Only on response turns (t.respond / t.respondAll): one structured answer per request
  readonly outputSchema?: JsonSchema;             // Only when this turn declared output: the structured-output requirement, lowered to JSON Schema
}

interface InputFile {
  readonly filename?: string;     // Optional, for the adapter / model to refer to
  readonly mimeType: string;      // For example, image/png
  readonly dataBase64: string;    // Base64 content, JSON-friendly, can go straight into a request body
}

interface InputResponse {
  readonly requestId: string;    // Required: which input.requested request this answers; aligns answers when several requests pause together
  readonly optionId?: string;    // Either this or text: the answer matched an id in the request's options (approve / deny…)
  readonly text?: string;        // Either this or optionId: a free-text answer (the request has no options, or the answer is not any option)
}

Whether the eval side calls t.send(), t.sendFile(), or t.respond(), the Adapter receives one ordinary send. The difference is all in the fields: If your application interface does not accept files, ignore files.

Eval side	`text`	`files`	`responses`	`outputSchema`
`t.send(text)`	The sent text	—	—	Present only when `output` was declared
`t.sendFile(path, text?)`	A note (can be an empty string)	Present	—	Same as above
`t.respond(...)` / `t.respondAll(...)`	Answer text	—	One per request	Same as above

Ignore files if the interface does not accept files. Forward outputSchema if the application interface accepts a schema (Chat Completions response_format, Responses text.format). It is fine if it does not: validation happens in the Runner anyway (see the Turn.data rules below).

Inputs for the different answers

In a HITL response turn, a person’s decision arrives in structured form with input.responses. The Adapter does not need to parse text to guess which answer belongs to which request, or whether it counts as approval. Every answer carries requestId. optionId and text are mutually exclusive: when the answer matches an id in the request’s options, it is optionId (the eval side has already checked that it exists, so a typo cannot slip through silently). Otherwise the whole answer is free text in text. The four typical forms:

// ① One pending request, the answer matches an option (approve / deny are the same, only optionId differs)
await t.respond("approve");
// → { text: "approve",
//     responses: [{ requestId: "req_1", optionId: "approve" }] }

// ② Several requests pause together; the object form names the request explicitly
await t.respond({ request, optionId: "deny" });
// → { text: "deny",
//     responses: [{ requestId: request.id, optionId: "deny" }] }

// ③ The answer is not any option → free text (requests waiting for more information look like this)
await t.respond("Change the recipient to ceo@corp.com");
// → { text: "Change the recipient to ceo@corp.com",
//     responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] }

// ④ respondAll: one answer per pending request, all with the same optionId
await t.respondAll("approve");
// → { text: "approve\napprove",
//     responses: [{ requestId: "req_1", optionId: "approve" },
//                 { requestId: "req_2", optionId: "approve" }] }

On the Adapter side, hand each decision back to the application by requestId; do not guess by order. A call rejected by a person sets the tool operation.finished status to "rejected", not "failed". Rejection is a human decision, not a tool failure. This way noFailedActions() is not tripped by mistake, and calledTool(toolMatch(..., { status: "rejected" })) can assert it precisely.

Context: `AgentContext`

interface AgentContext {
  // Feedback and runtime context present on every turn
  readonly signal: AbortSignal;    // Runner timeout and cancellation: pass it to every request you make
  readonly session: AgentSession;  // State keys for this session line
  progress(update: { message: string; current?: number; total?: number }): void;
  diagnostic(input: {
    code: string;
    level: "warning" | "error";
    message: string;
    data?: Readonly<Record<string, JsonValue>>;
    dedupeKey?: string;
  }): void;

  // Three fields that appear by integration tier (see "Integration tiers")
  readonly model?: string;         // Tier 1 model comparison: experiment.model passed through. Forward it if the application interface accepts model selection
  readonly reasoningEffort?: string;  // Reasoning effort: experiment.reasoningEffort passed through, owned the same way as model
  readonly telemetry?: Telemetry;  // Tier 2: this turn's W3C trace context (headers, a new traceparent each turn)
  readonly flags: Readonly<Record<string, string | number | boolean>>; // Tier 3 feature A/B: experiment.flags passed through. {} when not configured
  readonly experimentId?: string;  // Experiment id derived from the path, same source as experimentId in results. undefined when not run through an experiment

}

interface SandboxAgentContext extends AgentContext {
  readonly sandbox: Sandbox;       // Available only to Agents built with defineSandboxAgent
}

interface AgentSession {
  // Session continuation: the server keeps history (the shape for interfaces that accept a session id)
  readonly id?: string;                      // The session id recorded on this line. undefined on a new session line
  capture(id: string | undefined): void;     // Record a returned id. Only takes effect if none was recorded yet

  // Adapter-private state: client-side history, a paused HITL turn, and so on
  get<T>(key: SessionKey<T>): T | undefined;
  set<T>(key: SessionKey<T>, value: T): void;
  take<T>(key: SessionKey<T>): T | undefined; // Clears on read, consumed once
}

The Runner binds a separate lifecycle scope for the Agent’s setup, each send, and teardown. The Adapter only reports progress/diagnostic from inside the current callback, and cannot pass a phase, color, or output stream. progress is not saved to disk. diagnostic is committed to Record as an Attempt-owned diagnostic-channel event. When it cannot continue, throw; the Runner saves a structured error and forms an errored verdict on the niceeval.verdict channel. The Attempt lifecycle state is only active / completed / abandoned; a verdict token is not an Attempt state. ctx has no flags for you to check. The three tier fields are all pass-through: model and flags are declared by the Experiment and handed over unchanged by the Runner. The Adapter only forwards them to the application with the request and does not interpret their meaning. telemetry appears only when OTel integration is configured; in send, just spread headers into the request headers. The receiving endpoint is fixed in defineConfig and pointed to when the application starts, not passed from here. See OTel integration. experimentId is a stable identifier derived from the path. You can use it in dynamic Sandbox callbacks to isolate cross-Attempt state per experiment. See Sandbox · Prepare the Sandbox in a deterministic order. session is the state owned by this session line. NiceEval promises only one thing about it: every send on the same session line gets the same ctx.session, and a new session line (the eval case’s first turn, or after t.newSession()) gets a brand new one. Continue server-side history with id / capture. For client-side history and paused HITL turns, the Adapter creates a typed key at module scope with createSessionKey<T>(name), then reads and writes it with get / set / take. take clears the value as it reads it. Keys are isolated by symbol identity, so they never collide even with the same name.

Return: `Turn`

interface Turn {
  readonly events: StreamEvent[];          // ★ The standard event stream: every Adapter's core output
  readonly data?: JsonValue;               // This turn's structured output. Fill it only when input.outputSchema is present; the Runner validates it against the declaration
  readonly status: "completed" | "failed" | "waiting";  // waiting = the turn paused for a person (HITL)
  readonly usage?: Usage;
}

events is an ordinary JS array: one object for each thing that happened in this turn, in the real order it happened. For example, in a turn asking “What’s the temperature in Beijing today?”, the agent looks up the weather and then answers, and events looks like this:

[
  { type: "operation.started", operationId: "c1",
    operation: { kind: "tool", name: "get_weather", input: { city: "Beijing" } } },
  { type: "operation.finished", operationId: "c1", kind: "tool",
    output: { temp: 21 }, status: "completed" },
  { type: "message", role: "assistant", text: "Beijing is 21°C today." },
]

There are ten object variants in total (message, operation.*, input.requested, and so on; see the Event stream reference for the full list). Assertions read exactly this array: t.calledTool("get_weather") counts started tool operations, and t.reply takes the last assistant message. Note that each send returns only this turn’s array. Joining turns into a whole session line is the Runner’s job; see the next section. Most of the time you do not write these objects by hand either: an official converter returns a complete Turn with events, usage, and status filled in. For choosing a converter, see Write send. data is not a pocket for “whatever”. It has only one rule: the eval case declares what it wants with a schema on send. data exists only when declared, and it has the declared type when you get it.

// Eval side: the output declaration is both a requirement for the application and the type source of turn.data
const turn = await t.send("What is the total on this invoice?", {
  output: z.object({ amount: z.number(), currency: z.string() }),
});
turn.data.amount;   // Type is number, not unknown; no manual narrowing

The declaration is lowered to JSON Schema and reaches the Adapter through input.outputSchema. After the Adapter gets the application’s response, it puts the structured output in data (for example, JSON.parse(reply.content)). The Runner enforces validation: if data does not match the declaration, the turn is immediately failed and reports the difference. There is no path where a wrong object slips in and an assertion silently passes. Conversely, a turn that did not declare output has no data: an application that only returns text never touches this field. Do not copy the raw response body into it to fill space.

After send returns: where the four Turn fields go

Once send returns, the Adapter’s work is done; everything else is the Runner’s job. Each of the four fields has a clear destination:

Turn field	What the Runner does with it	Where the eval author sees it
`events`	Appends it to this session line and normalizes it into messages and logical tool occurrences. The last assistant `message` also updates `t.reply`	Scoped assertions such as `turn.calledTool()` and `turn.event(...)`; `t.events` can be inspected directly
`status`	Records this turn’s final state. On `"waiting"`, it collects this turn’s `input.requested` into the pending list	`t.succeeded()` or `t.check(turn.status, equals("waiting"))`. After `"waiting"`, continue with `t.respond()`
`usage`	Adds it up per turn for the session line and the whole run	`maxTokens` / `maxCost` assertions and usage in reports
`data`	Validates it against the `output` schema declared on send; a mismatch makes the turn `failed` immediately. Saved only if it passes	`turn.data` (strongly typed by the declaration), `t.check(turn.data, equals(...))` or `matches(...)`

The handle you get from await t.send() is a view of this one Turn: turn-level assertions see only this turn’s events and fields, and t-level assertions see what has accumulated on the whole session line. So the Adapter does not need to “cooperate” with any assertion. Fill these four fields correctly, and assertions, scoring, and reports all happen downstream automatically.

Where capabilities come from: proof by construction, not a questionnaire

defineAgent builds a Direct Agent: the Runner calls a function, SDK, or service endpoint directly, and the target can be local or remote. A Direct Agent’s setup, send, and teardown never get a Sandbox. defineSandboxAgent builds a Sandbox Agent, and only its callbacks get a SandboxAgentContext with a real sandbox.

What it unlocks on `t`	What proves it	What you write
`t.send`, `t.sendFile`, `t.check`…	Every Agent has them	Nothing
Multiple `t.send()` calls, `t.reply`, `t.newSession()`	`send` uses the `ctx.session` continuation accessors (typed key or `id` + `capture()`); without them every turn is a new conversation	Nothing; the accessors are already on `ctx`
`t.respond()`, status checks, and other HITL	send has returned `"waiting"` + an `input.requested` event	Nothing; doing it means having it
Positive assertions such as `turn.calledTool()` / `turn.toolOrder()`	events contain paired `operation.started` / `operation.finished`	Nothing; if the events exist, you can assert
Negative assertions such as `t.notCalledTool()` / `t.usedNoTools()` being trustworthy	Events come from an official converter with a completeness proof	Nothing; the proof travels with the converter’s return value
`t.sandbox`, `t.sandbox.fileChanged()`, and similar	Built with `defineSandboxAgent`	Nothing
Trace channel and the `niceeval view` waterfall	A `tracing` block exists (spans only go into the waterfall, not into assertions)	Nothing; you write the block anyway

The negative-assertion row is the only one with levels. notCalledTool asserts that something “did not happen”, which requires “all events”. That holds only when the source itself has a completeness contract (direct SDK native event streams, AI SDK result.steps, Responses output). A handwritten mapping has no proof, so a negative assertion warns at runtime that it is not trustworthy instead of silently passing. So there is no gap of “declared but cannot deliver” either: trust follows the source of the events, not the author’s diligence. For exact obligations per capability, see the Capability reference.

Write send — The hands-on tutorial: from sending one message to a full integration in seven steps, each unlocking a set of assertions.
Connect your Agent — The integration overview: minimal integration, parameter channels, and the incremental map.
Drive — The eval-side view: how to use t.send(), t.newSession(), and HITL.
Assert — The full assertion vocabulary driven by the standard event stream.
Overall model — How eval cases, Experiments, Adapters, the Runner, and Record connect into one run.

Core Model

Interaction and Scoring

Execution Model

Adapter: connect an Agent to NiceEval

Adapters

Integration tiers

The send function

Input: `TurnInput`

Inputs for the different answers

Context: `AgentContext`

Return: `Turn`

After send returns: where the four Turn fields go

Where capabilities come from: proof by construction, not a questionnaire

Adapters

Integration tiers

The send function

Input: TurnInput

Inputs for the different answers

Context: AgentContext

Return: Turn

After send returns: where the four Turn fields go

Where capabilities come from: proof by construction, not a questionnaire

Related reading

Input: `TurnInput`

Context: `AgentContext`

Return: `Turn`