> ## Documentation Index > Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Adapter: Connect an Agent to NiceEval > The Adapter is the adapter you write. This page explains what the send function receives and returns, how system-under-test configuration is passed in, the three integration tiers, and where the capabilities on t come from. In [NiceEval](https://niceeval.com/), an `Adapter` is the adaptation layer between the runner and the system under test. The Adapter implements the `send` function: it forwards eval-side actions such as `t.send()` to your application, then translates the application's response into [NiceEval](https://niceeval.com/)'s standard `Turn`. Follow-up reading: [Write Send](/docs/tutorials/write-send) ## The adapter If the system under test speaks the standard OpenAI Chat Completions or Responses protocol, use an official adapter directly. For a homegrown frontend-backend protocol (HTTP, gRPC, WebSocket — anything works), write an Adapter: it knows how to authenticate, how to call your application, and how to translate the response into the standard event stream. One full round trip of `t.send()` crosses this boundary twice: Full round trip of t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter calls your application and returns a Turn with the standard event stream. Outbound: the driving verbs in the eval (`t.send` / `t.sendFile` / `t.respond`) assemble a `TurnInput` and the `ctx` for this conversation line, then call the Adapter's `send` once. Inbound: the Adapter translates the application's raw response into a `Turn` — where `events` is an array of event objects ordered by when they happened (real values below). The system under test's URL, auth, and protocol details are passed in through the Adapter factory's configuration. `defineExperiment` receives an already configured agent instance; the Adapter consumes that configuration inside the `send` closure, and forwards per-turn dynamic values (such as `ctx.model`, `ctx.flags`, `ctx.telemetry`) to the application with each request: ```ts theme={null} // agents/web-agent.ts import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter"; import type { Agent } from "niceeval/adapter"; interface WebAgentOptions { baseUrl: string; apiKey?: string; } export function webAgent(options: WebAgentOptions): Agent { const baseUrl = options.baseUrl.replace(/\/$/, ""); return defineAgent({ name: "web-agent", evidenceCoverage: completeEvidenceCoverage, async send(input, ctx) { ctx.progress({ message: "Waiting for Web Agent" }); const response = await fetch(`${baseUrl}/api/turn`, { method: "POST", headers: { "content-type": "application/json", ...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}), ...ctx.telemetry?.headers, }, body: JSON.stringify({ message: input.text, files: input.files, model: ctx.model, flags: ctx.flags, }), signal: ctx.signal, }); if (!response.ok) { ctx.diagnostic({ code: "agent-http-error", level: "error", message: `Web Agent returned HTTP ${response.status}`, data: { status: response.status }, }); return { status: "failed", events: [{ type: "error", message: `HTTP ${response.status}` }], }; } const body = await response.json(); return { status: "completed", events: body.events, data: body.data, usage: body.usage, }; }, }); } // experiments/staging.ts import { defineExperiment } from "niceeval"; import { webAgent } from "../agents/web-agent.ts"; export default defineExperiment({ agent: webAgent({ baseUrl: "https://staging.example.com", apiKey: process.env.STAGING_AGENT_API_KEY, }), model: "gpt-5.4", flags: { promptVariant: "concise" }, }); ``` ## Integration tiers Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: **Tier 1 is send only** (not a single line of application code changes, and the full assertion set is already complete at this tier), **Tier 2 is send + OTel** (the application sends spans to [NiceEval](https://niceeval.com/), buying the call waterfall in `niceeval view`), **Tier 3 is application changes + experiment flags** (feature A/B). What each tier costs, what it buys, and when to move up: see [Tier](/docs/explanation/tier). The eval-side driving API — `t.send()`, `t.sendFile()`, `t.newSession()`, and HITL's `t.respond()` / `t.respondAll()` — all uniformly call the Adapter's `send`. How they converge and how to handle them in `send`: see [Write Send](/docs/tutorials/write-send). ## The send function Whether the Adapter connects to an HTTP service or a CLI in a Sandbox, the interface exposed to the runner is exactly the same: ```ts theme={null} interface Agent { readonly name: string; send(input: TurnInput, ctx: AgentContext): Promise; // Optional: setup / teardown (once-per-Sandbox lifecycle); // optional observability members are not part of send — see "OTel Integration" } ``` `progress` is short-lived status during the run and is never persisted. `diagnostic` is a bounded warning or error saved in an Attempt-owned diagnostics channel. Neither can specify a lifecycle phase or change `Turn.status` on its own. When the connection fails or the response cannot be parsed, throw: the Runner writes a structured error and forms an `errored` Verdict in the `niceeval.verdict` channel. `send` is the only function you must implement. There are only three types in its signature; take them one at a time. The `setup` / `teardown` here are the Agent's own business of "how it connects itself" (installing a CLI, writing auth config). Environment setup that varies per experiment (installing a binary specific to one experiment, warming up, carrying state across attempts) does not belong on the Agent — it hangs off the `.setup()` / `.teardown()` chain methods of the Sandbox spec in the `sandbox` field; see [Sandbox provider · Lifecycle](/docs/tutorials/sandbox-providers#lifecycle). ### Input: `TurnInput` ```ts theme={null} interface TurnInput { readonly text: string; // Every turn: the text being sent; on answer turns it's the answer text (multiple answers joined by newlines) readonly files?: readonly InputFile[]; // t.sendFile turns only: attached files (base64) readonly responses?: readonly InputResponse[]; // Answer turns only (t.respond / t.respondAll): one structured answer per request readonly outputSchema?: JsonSchema; // Only when this turn declares output: the structured output requirement, lowered to JSON Schema } interface InputFile { readonly filename?: string; // Optional, for the adapter / model to reference readonly mimeType: string; // e.g. image/png readonly dataBase64: string; // base64 content, JSON-friendly, can go straight into the request body } interface InputResponse { readonly requestId: string; // Always present: which input.requested request this answers; used to match up when multiple requests are paused readonly optionId?: string; // Either this or text: the answer hit one of the request's option ids (approve / deny...) readonly text?: string; // Either this or optionId: a free-text answer (the request has no options, or the answer matches none) } ``` Whether the eval side calls `t.send()`, `t.sendFile()`, or `t.respond()`, it arrives at the Adapter as one ordinary `send` — the entire difference is in the fields: If your application interface does not accept files, ignore `files`. | Eval side | `text` | `files` | `responses` | `outputSchema` | | -------------------------------------- | ----------------------------- | ------- | --------------- | ------------------------------ | | `t.send(text)` | Sent text | — | — | Only when `output` is declared | | `t.sendFile(path, text?)` | Caption (may be empty string) | Present | — | Same | | `t.respond(...)` / `t.respondAll(...)` | Answer text | — | One per request | Same | `files`: ignore it if the interface does not accept files; `outputSchema`: forward it if the application interface accepts a schema (Chat Completions' `response_format`, Responses' `text.format`); it's fine if it doesn't — validation happens in the runner anyway (see the `Turn.data` rule below). #### Inputs for the different answers On HITL answer turns, the human verdict arrives in structured form via `input.responses` — the Adapter never needs to parse `text` to guess which sentence answers which request or whether it counts as approval. Every answer carries a `requestId`; `optionId` and `text` are mutually exclusive: if the answer hits one of the request's option ids it becomes `optionId` (the eval side has already validated it exists, so no typo can slip through silently); otherwise the whole sentence lands in `text` as free text. The four typical shapes: ```ts theme={null} // 1) A single pending request, answer hits an option (approve / deny work the same, only the optionId differs) await t.respond("approve"); // → { text: "approve", // responses: [{ requestId: "req_1", optionId: "approve" }] } // 2) Multiple requests paused at once — the object form matches them up explicitly await t.respond({ request, optionId: "deny" }); // → { text: "deny", // responses: [{ requestId: request.id, optionId: "deny" }] } // 3) The answer is none of the options → free text (requests waiting for extra info look like this) await t.respond("Change the recipient to ceo@corp.com"); // → { text: "Change the recipient to ceo@corp.com", // responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] } // 4) respondAll: one answer per pending request, same optionId await t.respondAll("approve"); // → { text: "approve\napprove", // responses: [{ requestId: "req_1", optionId: "approve" }, // { requestId: "req_2", optionId: "approve" }] } ``` The Adapter's corresponding obligations: hand the verdict back to the application by `requestId` (don't guess by order); for calls a human rejected, set the tool `operation.finished` event's `status` to `"rejected"` rather than `"failed"` — rejection is a human decision, not a tool failure, so `noFailedActions()` doesn't misfire and `calledTool(..., { status: "rejected" })` can assert it precisely. ### Context: `AgentContext` ```ts theme={null} interface AgentContext { // Feedback and run context, present on every turn readonly signal: AbortSignal; // The runner's timeout and cancellation: pass it through to every request you make readonly session: AgentSession; // The state slot for this conversation line progress(update: { message: string; current?: number; total?: number }): void; diagnostic(input: { code: string; level: "warning" | "error"; message: string; data?: Readonly>; dedupeKey?: string; }): void; // The ones that appear per integration tier (see "Integration tiers") readonly model?: string; // Tier 1 model comparison: experiment.model passed through; forward it if the application interface accepts model selection readonly reasoningEffort?: string; // Reasoning effort: experiment.reasoningEffort passed through, same ownership as model readonly telemetry?: Telemetry; // Tier 2: this turn's W3C trace context (headers, a fresh traceparent per turn) readonly flags: Readonly>; // Tier 3 feature A/B: experiment.flags passed through; {} when unset readonly experimentId?: string; // The path-derived experiment id, the same source as the experimentId in results; undefined when not run through an experiment readonly sandbox: Sandbox; // Non-sandbox Agents also receive a Sandbox (a no-op stub for Direct Agents) } interface AgentSession { // Session continuation: server keeps the history (the shape for interfaces that accept a session id) readonly id?: string; // The session id recorded for this line; undefined on a new conversation line capture(id: string | undefined): void; // Record the returned id; only lands if none has been recorded yet // Adapter-private typed slots: create a slot with createSessionSlot() first. // take clears the slot on read — use it for a paused HITL scene. get(slot: SessionSlot): T | undefined; set(slot: SessionSlot, value: T): void; take(slot: SessionSlot): T | undefined; } ``` The Runner binds a lifecycle scope to the Agent's `setup`, each `send`, and `teardown` separately. The Adapter only reports progress and diagnostics from the current callback; it cannot pass a phase, color, or output stream. `progress` is never persisted. A diagnostic is committed as an event in an Attempt-owned diagnostics channel. When it cannot continue, throw: the Runner saves the structured error and forms an `errored` Verdict in the `niceeval.verdict` channel. Attempt lifecycle state is only `active` / `completed` / `abandoned`; a Verdict token is not Attempt state. There are no flags to check on `ctx`. All three tier fields have "pass-through" semantics: `model` and `flags` are declared by the experiment and handed over verbatim by the runner — the Adapter's only job is to forward them to the application with the request, never to interpret them; `telemetry` only appears when OTel integration is configured, and inside `send` you only need to spread its `headers` into the request headers — the receiving endpoint is fixed in `defineConfig` and pointed at by the application at startup, not passed from here; see [OTel Integration](/docs/tutorials/connect-otel). `experimentId` is a stable identifier derived from the path; the typical use is isolating cross-attempt state per experiment inside a Sandbox's environment hooks (partitioning cache directory names and sandbox snapshot tags by it); see [Sandbox provider · Lifecycle](/docs/tutorials/sandbox-providers#lifecycle). `session` is the private state of one conversation line, and [NiceEval](https://niceeval.com/) promises exactly one thing about it: **every `send` on the same conversation line receives the same `ctx.session`, and a new conversation line (the first turn of an eval, or after `t.newSession()`) receives a brand-new one.** The continuation accessors (`id`/`capture`) and private typed slots all live on it. Create each slot once with `createSessionSlot()`, then call `set(slot, value)` and `take(slot)` for a paused HITL scene; `take` clears the value on read. On a new line, `id` and every slot are unset. ### Return: `Turn` ```ts theme={null} interface Turn { readonly events: StreamEvent[]; // ★ The standard event stream — every Adapter's core product readonly data?: JsonValue; // This turn's structured output; only filled when input.outputSchema is present, validated by the runner against the declaration readonly status: "completed" | "failed" | "waiting"; // waiting = paused for a human (HITL) readonly usage?: Usage; } ``` `events` is just a **plain JS array**: one object per thing that happened this turn, in the order it actually happened. For a turn like "How warm is Beijing today?", where the agent checks the weather before answering, `events` looks like this: ```ts theme={null} [ { type: "operation.started", operationId: "c1", operation: { kind: "tool", name: "get_weather", input: { city: "Beijing" } } }, { type: "operation.finished", operationId: "c1", kind: "tool", output: { temp: 21 }, status: "completed" }, { type: "message", role: "assistant", text: "Beijing is 21°C today." }, ] ``` There are ten object variants in total (`message`, `operation.*`, `input.requested`… full list in the [Events Reference](/docs/reference/events)). Assertions read this exact array: `t.calledTool("get_weather")` counts started tool operations, while `t.reply` takes the last assistant `message`. Note that each `send` returns only **this turn's** array — stitching turns into a whole conversation line is the runner's job, covered in the next section. In most cases you don't hand-write these objects either: an official converter's return value is already a complete `Turn` with `events`, `usage`, and `status` filled in — how to pick a converter is covered in [Write Send](/docs/tutorials/write-send). `data` is not a "put whatever" pocket; it has exactly one rule: **what goes in it is declared by the eval with a schema on the send; only a declaration produces data, and what you get is the declared type.** ```ts theme={null} // Eval side: the output declaration is both the requirement given to the application and the type source for turn.data const turn = await t.send("What's the total on this invoice?", { output: z.object({ amount: z.number(), currency: z.string() }), }); turn.data.amount; // Typed as number — not unknown, no manual narrowing ``` The declaration, lowered to JSON Schema, reaches the Adapter via `input.outputSchema`; the Adapter puts the structured output from the application's response into `data` (e.g. `JSON.parse(reply.content)`). **Validation is enforced by the runner**: if `data` doesn't match the declaration, the turn fails immediately with the diff reported — there is no path where "a wrong object got stuffed in and the assertion silently passed". Conversely, turns with no `output` declaration have no `data`: applications that only return text never touch this field — don't copy the raw response body into it to fill space. ### After send returns: where the four Turn fields go The moment `send` returns, the Adapter's work is done — everything after that is the runner's job. Each of the four fields has a definite destination: | Turn field | What the runner does with it | Where the eval author feels it | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `events` | Appended to this conversation line's event stream; `deriveRunFacts` folds it into facts (`toolCalls`, `parked`, `messageCount`…); the last assistant `message` also updates `t.reply` | All scoped assertions: `t.calledTool()`, `t.messageIncludes()`…; `t.events` is directly queryable | | `status` | Records the turn's outcome; on `"waiting"`, collects this turn's `input.requested` into the pending list | `t.succeeded()` / `t.parked()`; after `"waiting"`, continue with `t.respond()` | | `usage` | Accumulated per turn into the conversation line and the whole run | `maxTokens` / `maxCost` scorers, usage in reports | | `data` | Validated against the `output` schema declared on the send — mismatch fails the turn immediately; saved only if it passes | `turn.data` (strongly typed by the declaration), `outputEquals` | The handle you get from `await t.send()` is a view onto this one Turn: turn-level assertions (things like `draft.parked()`) look only at this turn's `events`, while `t`-level assertions look at the whole conversation line's accumulation. So the Adapter never needs to "cooperate" with any assertion — fill in these four fields correctly, and assertion evaluation, verdict folding, and reporting all happen automatically downstream. ## Where capabilities come from: proof by construction, not a questionnaire | What unlocks on `t` | What the evidence is | What you write | | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `t.send`, `t.sendFile`, `t.check`, `t.judge`… | Every Agent has them | Nothing | | Multiple `t.send()`, `t.reply`, `t.newSession()` | `send` wired up `ctx.session` through a typed slot or `id` + `capture()` — without them every turn is a fresh conversation | Nothing — the accessors are right on `ctx` | | `t.respond()` / `t.parked()` and other HITL | send has returned `"waiting"` + an `input.requested` event | Nothing — doing it is having it | | `t.calledTool()` / `t.toolOrder()` and other positive assertions | paired operation events present | Nothing — having events means you can assert | | `t.notCalledTool()` / `t.usedNoTools()` and other negative assertions being **trustworthy** | Events come from an official converter with a completeness proof | Nothing — the proof travels with the converter's return value | | `t.sandbox`, `t.sandbox.fileChanged()`, etc. | Constructed by `defineSandboxAgent` | Nothing | | trace channel and the `niceeval view` waterfall | A `tracing` block exists (spans only feed the waterfall, never assertions) | Nothing — you were writing the block anyway | The negative-assertion row is the only tiered one: `notCalledTool` asserts "it did not happen", which presupposes "the events are complete" — and that only holds when the source itself carries a completeness contract (SDK-native event stream pass-through, AI SDK's `result.steps`, Responses' `output`). Hand-written mappings carry no proof, so negative assertions warn at runtime that they are untrustworthy instead of silently passing. This also means there is no "declared but can't deliver" drift: trustworthiness follows where the events came from, not the author's self-assessment. The precise per-capability obligations are in the [Capabilities Reference](/docs/reference/capabilities). ## Related reading * [Write Send](/docs/tutorials/write-send) — The hands-on tutorial: from sending one message to a full integration, seven steps, each unlocking a set of assertions. * [Connect Your Agent](/docs/tutorials/connect-your-agent) — The integration overview: minimal integration, parameter channels, and the incremental map. * [Drive](/docs/explanation/drive) — The eval-side view: how to use `t.send()`, `t.newSession()`, and HITL. * [Assert](/docs/explanation/assert) — The complete assertion vocabulary driven by the standard event stream. * [Architecture Overview](/docs/explanation/overview) — The four layers and their boundaries.