Adapter: Connect an Agent to NiceEval

In NiceEval, an Adapter is the adaptation layer between the runner and the system under test. The Adapter implements the send function: it forwards eval-side actions such as t.send() to your application, then translates the application’s response into NiceEval’s standard Turn. Follow-up reading: Write Send

The adapter

If the system under test speaks the standard OpenAI Chat Completions or Responses protocol, use an official adapter directly. For a homegrown frontend-backend protocol (HTTP, gRPC, WebSocket — anything works), write an Adapter: it knows how to authenticate, how to call your application, and how to translate the response into the standard event stream. One full round trip of t.send() crosses this boundary twice:

Outbound: the driving verbs in the eval (t.send / t.sendFile / t.respond) assemble a TurnInput and the ctx for this conversation line, then call the Adapter’s send once. Inbound: the Adapter translates the application’s raw response into a Turn — where events is an array of event objects ordered by when they happened (real values below). The system under test’s URL, auth, and protocol details are passed in through the Adapter factory’s configuration. defineExperiment receives an already configured agent instance; the Adapter consumes that configuration inside the send closure, and forwards per-turn dynamic values (such as ctx.model, ctx.flags, ctx.telemetry) to the application with each request:

// agents/web-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

interface WebAgentOptions {
  baseUrl: string;
  apiKey?: string;
}

export function webAgent(options: WebAgentOptions): Agent {
  const baseUrl = options.baseUrl.replace(/\/$/, "");

  return defineAgent({
    name: "web-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      ctx.progress({ message: "Waiting for Web Agent" });
      const response = await fetch(`${baseUrl}/api/turn`, {
        method: "POST",
        headers: {
          "content-type": "application/json",
          ...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}),
          ...ctx.telemetry?.headers,
        },
        body: JSON.stringify({
          message: input.text,
          files: input.files,
          model: ctx.model,
          flags: ctx.flags,
        }),
        signal: ctx.signal,
      });

      if (!response.ok) {
        ctx.diagnostic({
          code: "agent-http-error",
          level: "error",
          message: `Web Agent returned HTTP ${response.status}`,
          data: { status: response.status },
        });
        return {
          status: "failed",
          events: [{ type: "error", message: `HTTP ${response.status}` }],
        };
      }

      const body = await response.json();
      return {
        status: "completed",
        events: body.events,
        data: body.data,
        usage: body.usage,
      };
    },
  });
}

// experiments/staging.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";

export default defineExperiment({
  agent: webAgent({
    baseUrl: "https://staging.example.com",
    apiKey: process.env.STAGING_AGENT_API_KEY,
  }),
  model: "gpt-5.4",
  flags: { promptVariant: "concise" },
});

Integration tiers

Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: Tier 1 is send only (not a single line of application code changes, and the full assertion set is already complete at this tier), Tier 2 is send + OTel (the application sends spans to NiceEval, buying the call waterfall in niceeval view), Tier 3 is application changes + experiment flags (feature A/B). What each tier costs, what it buys, and when to move up: see Tier. The eval-side driving API — t.send(), t.sendFile(), t.newSession(), and HITL’s t.respond() / t.respondAll() — all uniformly call the Adapter’s send. How they converge and how to handle them in send: see Write Send.

The send function

Whether the Adapter connects to an HTTP service or a CLI in a Sandbox, the interface exposed to the runner is exactly the same:

interface Agent {
  readonly name: string;
  send(input: TurnInput, ctx: AgentContext): Promise<Turn>;
  // Optional: setup / teardown (once-per-Sandbox lifecycle);
  // optional observability members are not part of send — see "OTel Integration"
}

progress is short-lived status during the run and is never persisted. diagnostic is a bounded warning or error saved in an Attempt-owned diagnostics channel. Neither can specify a lifecycle phase or change Turn.status on its own. When the connection fails or the response cannot be parsed, throw: the Runner writes a structured error and forms an errored Verdict in the niceeval.verdict channel. send is the only function you must implement. There are only three types in its signature; take them one at a time.

The setup / teardown here are the Agent’s own business of “how it connects itself” (installing a CLI, writing auth config). Environment setup that varies per experiment (installing a binary specific to one experiment, warming up, carrying state across attempts) does not belong on the Agent — it hangs off the .setup() / .teardown() chain methods of the Sandbox spec in the sandbox field; see Sandbox provider · Lifecycle.

Input: `TurnInput`

interface TurnInput {
  readonly text: string;                          // Every turn: the text being sent; on answer turns it's the answer text (multiple answers joined by newlines)
  readonly files?: readonly InputFile[];          // t.sendFile turns only: attached files (base64)
  readonly responses?: readonly InputResponse[];  // Answer turns only (t.respond / t.respondAll): one structured answer per request
  readonly outputSchema?: JsonSchema;             // Only when this turn declares output: the structured output requirement, lowered to JSON Schema
}

interface InputFile {
  readonly filename?: string;     // Optional, for the adapter / model to reference
  readonly mimeType: string;      // e.g. image/png
  readonly dataBase64: string;    // base64 content, JSON-friendly, can go straight into the request body
}

interface InputResponse {
  readonly requestId: string;    // Always present: which input.requested request this answers; used to match up when multiple requests are paused
  readonly optionId?: string;    // Either this or text: the answer hit one of the request's option ids (approve / deny...)
  readonly text?: string;        // Either this or optionId: a free-text answer (the request has no options, or the answer matches none)
}

Whether the eval side calls t.send(), t.sendFile(), or t.respond(), it arrives at the Adapter as one ordinary send — the entire difference is in the fields: If your application interface does not accept files, ignore files.

Eval side	`text`	`files`	`responses`	`outputSchema`
`t.send(text)`	Sent text	—	—	Only when `output` is declared
`t.sendFile(path, text?)`	Caption (may be empty string)	Present	—	Same
`t.respond(...)` / `t.respondAll(...)`	Answer text	—	One per request	Same

files: ignore it if the interface does not accept files; outputSchema: forward it if the application interface accepts a schema (Chat Completions’ response_format, Responses’ text.format); it’s fine if it doesn’t — validation happens in the runner anyway (see the Turn.data rule below).

Inputs for the different answers

On HITL answer turns, the human verdict arrives in structured form via input.responses — the Adapter never needs to parse text to guess which sentence answers which request or whether it counts as approval. Every answer carries a requestId; optionId and text are mutually exclusive: if the answer hits one of the request’s option ids it becomes optionId (the eval side has already validated it exists, so no typo can slip through silently); otherwise the whole sentence lands in text as free text. The four typical shapes:

// 1) A single pending request, answer hits an option (approve / deny work the same, only the optionId differs)
await t.respond("approve");
// → { text: "approve",
//     responses: [{ requestId: "req_1", optionId: "approve" }] }

// 2) Multiple requests paused at once — the object form matches them up explicitly
await t.respond({ request, optionId: "deny" });
// → { text: "deny",
//     responses: [{ requestId: request.id, optionId: "deny" }] }

// 3) The answer is none of the options → free text (requests waiting for extra info look like this)
await t.respond("Change the recipient to ceo@corp.com");
// → { text: "Change the recipient to ceo@corp.com",
//     responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] }

// 4) respondAll: one answer per pending request, same optionId
await t.respondAll("approve");
// → { text: "approve\napprove",
//     responses: [{ requestId: "req_1", optionId: "approve" },
//                 { requestId: "req_2", optionId: "approve" }] }

The Adapter’s corresponding obligations: hand the verdict back to the application by requestId (don’t guess by order); for calls a human rejected, set the tool operation.finished event’s status to "rejected" rather than "failed" — rejection is a human decision, not a tool failure, so noFailedActions() doesn’t misfire and calledTool(..., { status: "rejected" }) can assert it precisely.

Context: `AgentContext`

interface AgentContext {
  // Feedback and run context, present on every turn
  readonly signal: AbortSignal;    // The runner's timeout and cancellation: pass it through to every request you make
  readonly session: AgentSession;  // The state slot for this conversation line
  progress(update: { message: string; current?: number; total?: number }): void;
  diagnostic(input: {
    code: string;
    level: "warning" | "error";
    message: string;
    data?: Readonly<Record<string, JsonValue>>;
    dedupeKey?: string;
  }): void;

  // The ones that appear per integration tier (see "Integration tiers")
  readonly model?: string;         // Tier 1 model comparison: experiment.model passed through; forward it if the application interface accepts model selection
  readonly reasoningEffort?: string;  // Reasoning effort: experiment.reasoningEffort passed through, same ownership as model
  readonly telemetry?: Telemetry;  // Tier 2: this turn's W3C trace context (headers, a fresh traceparent per turn)
  readonly flags: Readonly<Record<string, JsonValue>>; // Tier 3 feature A/B: experiment.flags passed through; {} when unset
  readonly experimentId?: string;  // The path-derived experiment id, the same source as the experimentId in results; undefined when not run through an experiment

  readonly sandbox: Sandbox;       // Non-sandbox Agents also receive a Sandbox (a no-op stub for Direct Agents)
}

interface AgentSession {
  // Session continuation: server keeps the history (the shape for interfaces that accept a session id)
  readonly id?: string;                      // The session id recorded for this line; undefined on a new conversation line
  capture(id: string | undefined): void;     // Record the returned id; only lands if none has been recorded yet

  // Adapter-private typed slots: create a slot with createSessionSlot<T>() first.
  // take clears the slot on read — use it for a paused HITL scene.
  get<T>(slot: SessionSlot<T>): T | undefined;
  set<T>(slot: SessionSlot<T>, value: T): void;
  take<T>(slot: SessionSlot<T>): T | undefined;
}

The Runner binds a lifecycle scope to the Agent’s setup, each send, and teardown separately. The Adapter only reports progress and diagnostics from the current callback; it cannot pass a phase, color, or output stream. progress is never persisted. A diagnostic is committed as an event in an Attempt-owned diagnostics channel. When it cannot continue, throw: the Runner saves the structured error and forms an errored Verdict in the niceeval.verdict channel. Attempt lifecycle state is only active / completed / abandoned; a Verdict token is not Attempt state. There are no flags to check on ctx. All three tier fields have “pass-through” semantics: model and flags are declared by the experiment and handed over verbatim by the runner — the Adapter’s only job is to forward them to the application with the request, never to interpret them; telemetry only appears when OTel integration is configured, and inside send you only need to spread its headers into the request headers — the receiving endpoint is fixed in defineConfig and pointed at by the application at startup, not passed from here; see OTel Integration. experimentId is a stable identifier derived from the path; the typical use is isolating cross-attempt state per experiment inside a Sandbox’s environment hooks (partitioning cache directory names and sandbox snapshot tags by it); see Sandbox provider · Lifecycle. session is the private state of one conversation line, and NiceEval promises exactly one thing about it: every send on the same conversation line receives the same ctx.session, and a new conversation line (the first turn of an eval, or after t.newSession()) receives a brand-new one. The continuation accessors (id/capture) and private typed slots all live on it. Create each slot once with createSessionSlot<T>(), then call set(slot, value) and take(slot) for a paused HITL scene; take clears the value on read. On a new line, id and every slot are unset.

Return: `Turn`

interface Turn {
  readonly events: StreamEvent[];          // ★ The standard event stream — every Adapter's core product
  readonly data?: JsonValue;               // This turn's structured output; only filled when input.outputSchema is present, validated by the runner against the declaration
  readonly status: "completed" | "failed" | "waiting";  // waiting = paused for a human (HITL)
  readonly usage?: Usage;
}

events is just a plain JS array: one object per thing that happened this turn, in the order it actually happened. For a turn like “How warm is Beijing today?”, where the agent checks the weather before answering, events looks like this:

[
  { type: "operation.started", operationId: "c1",
    operation: { kind: "tool", name: "get_weather", input: { city: "Beijing" } } },
  { type: "operation.finished", operationId: "c1", kind: "tool",
    output: { temp: 21 }, status: "completed" },
  { type: "message", role: "assistant", text: "Beijing is 21°C today." },
]

There are ten object variants in total (message, operation.*, input.requested… full list in the Events Reference). Assertions read this exact array: t.calledTool("get_weather") counts started tool operations, while t.reply takes the last assistant message. Note that each send returns only this turn’s array — stitching turns into a whole conversation line is the runner’s job, covered in the next section. In most cases you don’t hand-write these objects either: an official converter’s return value is already a complete Turn with events, usage, and status filled in — how to pick a converter is covered in Write Send. data is not a “put whatever” pocket; it has exactly one rule: what goes in it is declared by the eval with a schema on the send; only a declaration produces data, and what you get is the declared type.

// Eval side: the output declaration is both the requirement given to the application and the type source for turn.data
const turn = await t.send("What's the total on this invoice?", {
  output: z.object({ amount: z.number(), currency: z.string() }),
});
turn.data.amount;   // Typed as number — not unknown, no manual narrowing

The declaration, lowered to JSON Schema, reaches the Adapter via input.outputSchema; the Adapter puts the structured output from the application’s response into data (e.g. JSON.parse(reply.content)). Validation is enforced by the runner: if data doesn’t match the declaration, the turn fails immediately with the diff reported — there is no path where “a wrong object got stuffed in and the assertion silently passed”. Conversely, turns with no output declaration have no data: applications that only return text never touch this field — don’t copy the raw response body into it to fill space.

After send returns: where the four Turn fields go

The moment send returns, the Adapter’s work is done — everything after that is the runner’s job. Each of the four fields has a definite destination:

Turn field	What the runner does with it	Where the eval author feels it
`events`	Appended to this conversation line’s event stream; `deriveRunFacts` folds it into facts (`toolCalls`, `parked`, `messageCount`…); the last assistant `message` also updates `t.reply`	All scoped assertions: `t.calledTool()`, `t.messageIncludes()`…; `t.events` is directly queryable
`status`	Records the turn’s outcome; on `"waiting"`, collects this turn’s `input.requested` into the pending list	`t.succeeded()` / `t.parked()`; after `"waiting"`, continue with `t.respond()`
`usage`	Accumulated per turn into the conversation line and the whole run	`maxTokens` / `maxCost` scorers, usage in reports
`data`	Validated against the `output` schema declared on the send — mismatch fails the turn immediately; saved only if it passes	`turn.data` (strongly typed by the declaration), `outputEquals`

The handle you get from await t.send() is a view onto this one Turn: turn-level assertions (things like draft.parked()) look only at this turn’s events, while t-level assertions look at the whole conversation line’s accumulation. So the Adapter never needs to “cooperate” with any assertion — fill in these four fields correctly, and assertion evaluation, verdict folding, and reporting all happen automatically downstream.

Where capabilities come from: proof by construction, not a questionnaire

What unlocks on `t`	What the evidence is	What you write
`t.send`, `t.sendFile`, `t.check`, `t.judge`…	Every Agent has them	Nothing
Multiple `t.send()`, `t.reply`, `t.newSession()`	`send` wired up `ctx.session` through a typed slot or `id` + `capture()` — without them every turn is a fresh conversation	Nothing — the accessors are right on `ctx`
`t.respond()` / `t.parked()` and other HITL	send has returned `"waiting"` + an `input.requested` event	Nothing — doing it is having it
`t.calledTool()` / `t.toolOrder()` and other positive assertions	paired operation events present	Nothing — having events means you can assert
`t.notCalledTool()` / `t.usedNoTools()` and other negative assertions being trustworthy	Events come from an official converter with a completeness proof	Nothing — the proof travels with the converter’s return value
`t.sandbox`, `t.sandbox.fileChanged()`, etc.	Constructed by `defineSandboxAgent`	Nothing
trace channel and the `niceeval view` waterfall	A `tracing` block exists (spans only feed the waterfall, never assertions)	Nothing — you were writing the block anyway

The negative-assertion row is the only tiered one: notCalledTool asserts “it did not happen”, which presupposes “the events are complete” — and that only holds when the source itself carries a completeness contract (SDK-native event stream pass-through, AI SDK’s result.steps, Responses’ output). Hand-written mappings carry no proof, so negative assertions warn at runtime that they are untrustworthy instead of silently passing. This also means there is no “declared but can’t deliver” drift: trustworthiness follows where the events came from, not the author’s self-assessment. The precise per-capability obligations are in the Capabilities Reference.

Write Send — The hands-on tutorial: from sending one message to a full integration, seven steps, each unlocking a set of assertions.
Connect Your Agent — The integration overview: minimal integration, parameter channels, and the incremental map.
Drive — The eval-side view: how to use t.send(), t.newSession(), and HITL.
Assert — The complete assertion vocabulary driven by the standard event stream.
Architecture Overview — The four layers and their boundaries.

Core Model

Interaction & Assertions

Execution Model

Adapter: Connect an Agent to NiceEval

The adapter

Integration tiers

The send function

Input: `TurnInput`

Inputs for the different answers

Context: `AgentContext`

Return: `Turn`

After send returns: where the four Turn fields go

Where capabilities come from: proof by construction, not a questionnaire

​The adapter

​Integration tiers

​The send function

​Input: TurnInput

​Inputs for the different answers

​Context: AgentContext

​Return: Turn

​After send returns: where the four Turn fields go

​Where capabilities come from: proof by construction, not a questionnaire

​Related reading

The adapter

Integration tiers

The send function

Input: `TurnInput`

Inputs for the different answers

Context: `AgentContext`

Return: `Turn`

After send returns: where the four Turn fields go

Where capabilities come from: proof by construction, not a questionnaire

Related reading