Write Send

Adapter defines the contract for send: it receives TurnInput and AgentContext and returns a Turn. This tutorial starts from sending one message and adds, step by step, the capabilities a complete integration needs. Each step adds only a little code; parts already shown are marked with // …omitted; each step ends by listing the assertions the newly added data supports. After finishing any step, write the corresponding assertions into an eval, rerun npx niceeval exp, and check the multi-turn trajectory, usage, tool events, or pending requests in niceeval view. Three principles run through the whole page:

Connect at the interface your frontend already uses. The Adapter calls the same endpoint and receives the same format; do not open a new endpoint for evals, and do not import app internals to call functions directly. For why, see Connect your agent.
Only hand-write the transport. URL, auth, and request body depend on the app. niceeval/adapter provides converters from the raw response to the standard event stream; ctx.session provides the state API needed for session continuation and HITL pause/resume.
Runtime feedback goes through ctx, never straight to the terminal. Use ctx.progress(...) for long steps; use ctx.diagnostic(...) for degradations or unusual context you want to review after the run; throw when you cannot continue. Do not call console.log/error from an Adapter, and do not write to process.stdout/stderr.

Full round trip of one t.send: the eval calls t.send, the runner assembles TurnInput and ctx, and the adapter calls your app and returns a Turn with the standard event stream.

Confirm your app’s interface shape

NiceEval does not define a new application protocol. Existing apps usually use one of the protocols below, or a variant of one, and the built-in converters are provided for exactly these response shapes:

Protocol	Who defines it	In one sentence	Reference
Chat Completions	OpenAI; the industry’s de facto standard — nearly every model provider offers a compatible endpoint	Stateless question-and-answer: the client sends the full `messages` list every turn and one JSON comes back	API reference
Responses / Open Responses	OpenAI (the Responses API); Open Responses is an open specification built on it, initiated by OpenAI and co-developed with Hugging Face and the community for multi-provider interoperability	Stateful and agent-oriented: requests carry `previous_response_id` so the server continues the history, and the `output` array promises a record of the full process	API reference · Open Responses spec
AI SDK (Vercel)	Vercel; the de facto standard of the TypeScript ecosystem	Not a wire protocol but an SDK: `generateText` returns a complete result, and `useChat` speaks the UI Message Stream protocol it defines	generateText reference · UI Message Stream protocol

The steps below use a Chat Completions-shaped interface. Replacement implementations for a Responses-shaped interface and a streaming interface are given in steps two and four respectively.

Step 1: send a message, get the reply

The minimal send does exactly three things: send input.text to the app’s interface, put the reply into one message event, and report this turn’s status:

// agents/chat-app.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

const BASE_URL = "http://localhost:8080";   // to switch the URL per experiment, make it a factory option: see the integration guide

export default defineAgent({
  name: "chat-app",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    ctx.progress({ message: "Waiting for the Chat API" });
    const res = await fetch(`${BASE_URL}/v1/chat/completions`, {  // ← the interface the frontend already uses
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({
        messages: [{ role: "user", content: input.text }],
        model: ctx.model,                                         // ← experiment.model reaches here via ctx.model; delete this line if the app's interface doesn't support model selection
      }),
      signal: ctx.signal,                                         // ← runner timeout and cancellation
    }).then((r) => r.json());

    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: res.choices[0].message.content }],
    };
  },
});

If the interface returns a response you can keep working with but whose evidence is incomplete, report a diagnostic instead of printing the whole raw response:

if (!res.requestId) {
  ctx.diagnostic({
    code: "missing-request-id",
    level: "warning",
    message: "the response has no request id, so it cannot be correlated with server logs",
  });
}

progress is not persisted. A diagnostic is written to an Attempt-owned channel and can be reviewed on an Attempt detail page within a selected Run. If the HTTP connection fails or the response cannot be parsed, throw: the Runner records the error in agent.run and forms an errored Verdict in the niceeval.verdict channel. This step unlocks: t.reply, t.messageIncludes(), all the conversation material for the Judge, and model comparison on the Experiment side. ctx.model comes from the Experiment’s model; the runner passes it through verbatim and the Adapter only forwards it. For integration tiers, see Tier. It has two obvious limitations: every turn is a brand-new conversation (a second t.send cannot continue from the first), and tool calls are completely invisible. The next two steps solve one each.

Step 2: continue from earlier messages

The runner promises exactly one thing about sessions: every send on the same session line gets the same ctx.session; a new session line (the eval’s first turn, or after t.newSession()) gets a brand-new one. How to continue a session depends on the shape of the app’s interface. The app’s interface follows one of two common patterns, and ctx.session provides a pair of accessors for each:

Client carries the full history (stateless server; the complete message list is sent every turn: the Chat Completions shape is the typical case) → create an adapter-private slot with createSessionSlot<TMsg[]>(), then use ctx.session.get(slot) / set(slot, messages)
Server keeps the history (the interface takes a session id: the Responses shape’s previous_response_id, the native sessions / threads of various SDKs) → ctx.session.id + ctx.session.capture(id)

The storyline’s interface is the former:

// agents/chat-app.ts
import { completeEvidenceCoverage, createSessionSlot, defineAgent } from "niceeval/adapter";

const BASE_URL = "http://localhost:8080";

interface Msg { role: "user" | "assistant"; content: string | null }
const historySlot = createSessionSlot<Msg[]>("chat-app/history");

export default defineAgent({
  name: "chat-app",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    const history = ctx.session.get(historySlot) ?? []; // ← unwritten on a new session line
    const messages = [...history, { role: "user" as const, content: input.text }];

    const res = await fetch(`${BASE_URL}/v1/chat/completions`, {
      // …method, headers, signal same as step one, omitted
      body: JSON.stringify({ messages, model: ctx.model }),        // ← the change: send the full history; model forwarded same as step one
    }).then((r) => r.json());

    const reply = res.choices[0].message;
    ctx.session.set(historySlot, [...messages, reply]); // ← the next turn on this line reads it back

    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: reply.content }],
    };
  },
});

Notice there is no “first turn” branch: ctx.session.get(historySlot) naturally returns undefined on a new session line, and ?? [] normalizes that to empty history. Nothing needs declaring beyond using the typed session slot: connect it and multi-turn continues; skip it and every turn is a new conversation. If the app’s interface takes a session id, the history lives on the server and the Adapter only records the id — just two changes inside send:

// Inside send:
//   the request carries the previous turn's id … previous_response_id: ctx.session.id   ← naturally undefined on a new session line
//   write back whatever id the response returns … ctx.session.capture(res.id)           ← the next turn continues from it

capture only lands when no id has been recorded yet; a backend re-sending the id (or even changing it due to a fork) will not overwrite the line being continued. This step unlocks: multi-turn conversations, and t.newSession() session isolation.

Step 3: record usage

An agent that answers correctly but burns ten times the tokens should not get the same score as one that is frugal. Usage is usage, the fourth field on Turn alongside events and status: if the app’s interface returns usage, report it truthfully, and the runner accumulates it turn by turn into the session line and the whole run. Chat Completions-shaped responses come with usage — copy it over; the rest of send is exactly the same as step two:

    // …transport and session same as step two, omitted
    return {
      status: "completed",
      events: [{ type: "message", role: "assistant", text: reply.content }],
      usage: {                                        // ← the change: report the usage the interface returned, as-is
        inputTokens: res.usage.prompt_tokens,
        outputTokens: res.usage.completion_tokens,
      },
    };

The full Usage fields also include the optional cacheReadTokens / cacheCreationTokens, the reasoning token count reasoningTokens, the request count requests, and costUSD — if a gateway returns measured cost, fill that in; it takes precedence over price-table estimation. Each field is only filled when the interface actually reports it; if the interface returns no usage, leave usage out entirely, and other assertions are unaffected. This hand-copying is also transitional: the official converter in the next step fills in usage along with everything else. This step unlocks: the t.maxTokens() / t.maxCost() scorers (maxCost uses costUSD or the price table in the config), plus usage in reports and niceeval view.

Step 4: parse tools into events

The app’s response contains more than reply text — the tool_calls in a Chat Completions-shaped response record which tools this turn called. The Adapter’s most important job is normalizing the interface’s response into the standard event stream: one object per thing that happened this turn, ordered by actual occurrence in Turn.events, each object one of the ten types below (for the actual field values, the contract page has a complete one-turn example):

type StreamEvent =
  | { type: "message"; role: "assistant" | "user"; text: string }
  | { type: "operation.started"; operationId: string; operation:
      | { kind: "tool"; name: string; input: JsonValue; tool?: ToolName }
      | { kind: "subagent"; name: string; remoteUrl?: string } }
  | { type: "operation.finished"; operationId: string; kind: "tool";
      output?: JsonValue; status: "completed" | "failed" | "rejected" }
  | { type: "operation.finished"; operationId: string; kind: "subagent";
      output?: JsonValue; status: "completed" | "failed" }
  | { type: "skill.loaded"; skill: string; operationId?: string }
  | { type: "input.requested"; request: InputRequest }
  | { type: "thinking"; text: string }
  | { type: "compaction"; reason?: string }
  | { type: "error"; message: string };

Parsing is one “response field → event” mapping. Hand-written it looks like this — the rest of send is exactly the same as step two:

    // …transport and session same as step two, usage same as step three, omitted
    const reply = res.choices[0].message;

    const events: StreamEvent[] = [];
    for (const call of reply.tool_calls ?? []) {
      events.push({
        type: "operation.started",
        operationId: call.id,
        operation: {
          kind: "tool",
          name: call.function.name,
          input: JSON.parse(call.function.arguments),
        },
      });
    }
    if (reply.content) events.push({ type: "message", role: "assistant", text: reply.content });

    return { events, status: "completed", usage };

But you usually do not have to write this loop. When the response is a standard shape, one line of official converter replaces all of the above — events, status, even step three’s hand-copied usage are all in the return value; just return it:

    // …transport and session same as step two, omitted
    return turnFromChatCompletion(res);            // ← the mapping above plus step three's usage, official edition

If the interface is not this shape, swap in the matching piece:

Interface response shape	Built-in piece	What you need to write
Chat Completions-shaped response	`turnFromChatCompletion(res)`	No mapping to hand-write
Responses-shaped response	`turnFromResponses(res)`	No mapping to hand-write
AI SDK `generateText`’s complete result	`turnFromAiSdk(result)`	No mapping to hand-write
Streaming: SDK-native events passed through (one complete unit per frame)	`createClaudeSdkEventStream()` / `createPiAgentEventStream()` / `createCodexThreadEventStream()`	No mapping to hand-write
Streaming: the AI SDK’s UI Message Stream (the `useChat` protocol)	`uiMessageStreamAgent()`	No `send` or event mapping to hand-write
Streaming: token-by-token / argument-by-argument deltas	The protocol’s official reducer; if there is none, `deltaStream({ toOps })`	One “frame type → operation” mapping table
Final answer only	A few hand-written events	Start with one `message` (step one is this tier)

The built-in converters work by response shape, not by assuming a specific application protocol. Only delta streams with no ready-made reducer from the protocol side require you to write a mapping; the mapping only declares which operation each frame corresponds to — concatenation, pairing, and landing timing are handled by deltaStream. Once normalized, whichever events you emit determine which family of assertions eval authors can write:

Events you emit	Assertions unlocked
`message`	`t.reply`, `t.messageIncludes()`, Judge conversation material
tool `operation.started` / `operation.finished` (paired by `operationId`)	`t.calledTool()` / `t.toolOrder()` / `t.maxToolCalls()` / `t.noFailedActions()`…
`input.requested` (together with status `"waiting"`)	`t.parked()`, `t.requireInputRequest()`, `t.respond()` (step five)
`thinking` / `compaction` / `error`	The matching assertions and o11y counts

The Chat Completions response does not guarantee a complete process record. The app may finish the tool loop server-side and return only the final answer. Because of this, turnFromChatCompletion’s return carries no completeness proof: positive assertions like calledTool work, but negative assertions like notCalledTool will flag that the evidence is incomplete. The Responses protocol requires the output array to record the complete process, so turnFromResponses’s return carries a completeness proof and negative assertions are trustworthy. The difference in trustworthiness between the two comes from the interface contract. This step unlocks: the whole family of tool assertions — t.calledTool(), t.toolOrder(), t.maxToolCalls(), t.noFailedActions(), and more.

Step 5: HITL

When the app stops mid-turn to wait for a human (tool approval, missing information), send has obligations on both sides:

The pausing turn: return status: "waiting", and emit one input.requested event with a stable id per pending question — t.parked() and t.requireInputRequest() read them, and answers are matched by this id.
The answer turn: t.respond(...) in the eval reaches the Adapter as just another ordinary send (still the same session line, the same state); the human verdict arrives in structured form via input.responses, each entry carrying requestId and optionId or text (for the shapes, see Inputs for the different answers). The Adapter hands the verdict back to the app first, then continues fetching the result. For a call a human rejected, set the tool operation.finished event’s status to "rejected" rather than "failed" — a rejection is a human decision, not a tool failure, so noFailedActions() does not misfire.

The “scene read halfway through when the turn paused” (say, an SSE stream read halfway through) also lives on ctx.session: create an adapter-private createSessionSlot<Pending>(), call ctx.session.set(slot, scene) when pausing, and ctx.session.take(slot) at the start of the answer turn to get it back — taking it clears it, a single consumption. HITL almost always happens on streaming interfaces (pausing mid-stream), so this step’s example switches to an app that passes native events through over SSE — it exercises everything from the earlier steps together (complete runnable version in the tier1 example):

// agents/my-app.ts
import { completeEvidenceCoverage, createSessionSlot, defineAgent, sseJsonFrames, createPiAgentEventStream, driveFrameStream } from "niceeval/adapter";
import type { AgentContext, PiAgentStream, SseFrameCursor } from "niceeval/adapter";
import type { Turn, TurnInput } from "niceeval";

const BASE_URL = "http://localhost:5299";               // the app's own port; evals do not manage the process

interface Pending { cursor: SseFrameCursor<Frame>; stream: PiAgentStream; callId: string }
const pendingSlot = createSessionSlot<Pending>("my-app/pending");

function readStream(cursor: SseFrameCursor<Frame>, ctx: AgentContext, stream: PiAgentStream): Promise<Turn> {
  return driveFrameStream(cursor, stream, ctx, (frame) => {
    if (frame.type === "session") {
      ctx.session.capture(frame.sessionId);             // ← step two: write back the returned id; the next turn continues from it
      return;
    }
    if (frame.type === "approval_request") {            // ← this step: the app stops for approval; hold the scene
      ctx.session.set(pendingSlot, { cursor, stream, callId: frame.toolCallId });
      return { pause: { id: frame.toolCallId, action: frame.toolName,
                        options: [{ id: "approve" }, { id: "deny" }] } };
      // driveFrameStream on pause: emits an input.requested, sets status to "waiting", stops reading the stream
    }
    if (frame.type === "server_error") return { fail: frame.message };
  });
}

export default defineAgent({
  name: "my-app",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input: TurnInput, ctx: AgentContext): Promise<Turn> {
    const held = ctx.session.take(pendingSlot);
    if (held) {                                         // ← t.respond("approve"): an ordinary send; deliver the verdict first
      const approved = input.responses?.[0]?.optionId === "approve";  // with multiple pending requests, match by requestId
      if (!approved) held.stream.markRejected(held.callId);
      await fetch(`${BASE_URL}/api/chat/approve`, { method: "POST", signal: ctx.signal,
        body: JSON.stringify({ toolUseId: held.callId, approved }) });
      return readStream(held.cursor, ctx, held.stream); //    then keep reading the same stream; no new request
    }

    const res = await fetch(`${BASE_URL}/api/chat`, {   // ← transport: the only part you truly hand-write
      method: "POST",
      body: JSON.stringify({
        message: input.text,                            // ← step one: send the message
        model: ctx.model,                               // ← step one: experiment.model forwarded
        sessionId: ctx.session.id,                      // ← step two: absent on the first send, present automatically afterward
      }),
      signal: ctx.signal,
    });
    return readStream(sseJsonFrames<Frame>(res.body!), ctx, createPiAgentEventStream());  // ← step four: official converter, zero mapping
  },
});

For interfaces that don’t need HITL, delete the three parts related to the held pause-scene (Pending, its session slot, and the opening take branch) — the rest stays the same. For the complete mental model of pausing / answering / resuming, see HITL. This step unlocks: t.parked(), t.requireInputRequest(), t.respond() / t.respondAll(), and the precise assertion calledTool(..., { status: "rejected" }).

Step 6: hook up OTel traces

If the app is already instrumented (standard OTel HTTP server instrumentation is enough), the integration splits into two halves — one is startup-time configuration, the other lives in send. Telling them apart is telling apart “what never changes” from “what changes every turn”. The endpoint is startup-time configuration; it is not passed from send. NiceEval’s OTLP receiver address is the same on every run, so it does not go through ctx: pin the receiver port in niceeval.config.ts, point the app’s OTel exporter at that fixed URL on startup, and you never need to touch it again no matter how many eval runs follow:

// niceeval.config.ts
import { defineConfig } from "niceeval";

export default defineConfig({
  telemetry: { port: 4318 },   // the receiver always listens on http://localhost:4318/v1/traces
});

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces node server.js

What send passes is this turn’s trace context, not the endpoint. ctx.telemetry.headers is a W3C traceparent header the runner generates fresh every turn — spread it into the request, and the spans your app produces this turn attach precisely to this turn’s trace, with no misattribution when multiple evals run concurrently. Back on the storyline’s chat-app, send gains exactly one line:

    const res = await fetch(`${BASE_URL}/v1/chat/completions`, {
      // …rest same as step four, omitted
      headers: { "content-type": "application/json", ...ctx.telemetry?.headers },  // ← the only added line
    }).then((r) => r.json());

ctx.telemetry only appears when OTel integration is configured; spreading an undefined when it isn’t configured is safe, so this line can stay permanently. Spans still arrive without this header, but attribution degrades to time windows and that agent’s turns fall back to running serially — carrying it is what makes attribution accurate under concurrency. This step unlocks: the call waterfall shown per turn in niceeval view, including model calls, tool execution, duration, and tokens. Assertions still read the events produced by the earlier steps; spans are only used for the waterfall. For receiver configuration and span attribution rules, see OTel integration.

Step 7: pass through the experiment’s flags (A/B comparison)

Once the app exposes variants as switchable configuration, the experiment declares flags, and the runner hands them to send verbatim via ctx.flags every turn; the Adapter does not interpret their meaning, it only forwards them with the request — the app switches variants based on the parameter:

      // …rest of send same as before, omitted
      body: JSON.stringify({
        messages,
        model: ctx.model,        // ← experiment.model, already forwarded since step one; listed here alongside flags
        flags: ctx.flags,      // ← experiment.flags passed through as-is; {} when not configured
      }),

// experiments/concise.ts
import { defineExperiment } from "niceeval";
import chatApp from "../agents/chat-app.ts";

export default defineExperiment({
  agent: chatApp,
  model: "gpt-5.4",                      // ← reaches send via ctx.model
  flags: { promptVariant: "concise" },   // ← reaches send via ctx.flags every turn
});

Two experiment files, each declaring its own flags, running the same set of evals via npx niceeval exp separately — that’s an A/B comparison. This is Tier 3 of the three integration tiers (it requires the app to cooperate by exposing switches); for the cost and payoff, see Tier; flags alongside model, attempts, and the rest of the experiment fields are covered in Write experiments. This step unlocks: score comparison across variants over the same set of evals.

Reference: the five t APIs as send sees them

With the seven steps written, look back at the five driving APIs on the eval side — reaching send, they are just the same function receiving different fields; there is no second method to implement:

Eval-side API	Calls to the Adapter	What `send` receives
`t.send(text)`	One `send`	`input.text`; `ctx.session` is this session line’s own state
`t.sendFile(path, text?)`	One `send`	Same as above, plus `input.files` (base64 `InputFile[]`); put them into the request however the app’s interface expects, or ignore them if it is not multimodal
`t.newSession()`	Does not trigger a `send` by itself	Opens a second session line; that line’s next first `send` gets completely empty state — exactly the same form as the eval’s very first `send`
`t.respond(...answers)`	One ordinary `send`	`input.text` = the answer text; `input.responses` has one `{ requestId, optionId }` per request (`{ requestId, text }` for free-text answers); still the same line, the same state
`t.respondAll(optionId)`	One ordinary `send`	Same as above, with one answer per pending request; `optionId` is validated eval-side against every request — a typo throws instead of silently reaching the app

Adapter — send’s inputs and outputs, the three integration tiers, and where capabilities come from.
Connect your agent — minimal integration, parameter passing, and optional capabilities.
HITL — the complete concept of pausing to wait for a human: handshake timing and both sides’ obligations.
Drive — the eval side’s usage of t.send(), t.newSession(), and HITL.
Assert — the complete assertion vocabulary driven by the standard event stream.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Confirm your app’s interface shape

Step 1: send a message, get the reply

Step 2: continue from earlier messages

Step 3: record usage

Step 4: parse tools into events

Step 5: HITL

Step 6: hook up OTel traces

Step 7: pass through the experiment’s flags (A/B comparison)

Reference: the five t APIs as send sees them

​Confirm your app’s interface shape

​Step 1: send a message, get the reply

​Step 2: continue from earlier messages

​Step 3: record usage

​Step 4: parse tools into events

​Step 5: HITL

​Step 6: hook up OTel traces

​Step 7: pass through the experiment’s flags (A/B comparison)

​Reference: the five t APIs as send sees them

​Related reading

Confirm your app’s interface shape

Step 1: send a message, get the reply

Step 2: continue from earlier messages

Step 3: record usage

Step 4: parse tools into events

Step 5: HITL

Step 6: hook up OTel traces

Step 7: pass through the experiment’s flags (A/B comparison)

Reference: the five t APIs as send sees them

Related reading