> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Write Send
> In seven steps, write the Adapter's send function: send a message, continue the session, record usage, map tool events, handle human-in-the-loop (HITL), hook up OTel traces, and pass through Experiment flags.
[Adapter](/docs/explanation/adapter) defines the contract for `send`: it receives `TurnInput` and `AgentContext` and returns a `Turn`. This tutorial starts from sending one message and adds, step by step, the capabilities a complete integration needs. Each step adds only a little code; parts already shown are marked with `// …omitted`; each step ends by listing the assertions the newly added data supports. After finishing any step, write the corresponding assertions into an eval, rerun `npx niceeval exp`, and check the multi-turn trajectory, usage, tool events, or pending requests in `niceeval view`.
Three principles run through the whole page:
* **Connect at the interface your frontend already uses.** The Adapter calls the same endpoint and receives the same format; do not open a new endpoint for evals, and do not import app internals to call functions directly. For why, see [Connect your agent](/docs/tutorials/connect-your-agent).
* **Only hand-write the transport.** URL, auth, and request body depend on the app. `niceeval/adapter` provides converters from the raw response to the standard event stream; `ctx.session` provides the state API needed for session continuation and HITL pause/resume.
* **Runtime feedback goes through `ctx`, never straight to the terminal.** Use `ctx.progress(...)` for long steps; use `ctx.diagnostic(...)` for degradations or unusual context you want to review after the run; throw when you cannot continue. Do not call `console.log/error` from an Adapter, and do not write to `process.stdout/stderr`.
## Confirm your app's interface shape
[NiceEval](https://niceeval.com/) does not define a new application protocol. Existing apps usually use one of the protocols below, or a variant of one, and the built-in converters are provided for exactly these response shapes:
| Protocol | Who defines it | In one sentence | Reference |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Chat Completions** | OpenAI; the industry's de facto standard — nearly every model provider offers a compatible endpoint | Stateless question-and-answer: the client sends the full `messages` list every turn and one JSON comes back | [API reference](https://platform.openai.com/docs/api-reference/chat) |
| **Responses / Open Responses** | OpenAI (the Responses API); Open Responses is an open specification built on it, initiated by OpenAI and co-developed with Hugging Face and the community for multi-provider interoperability | Stateful and agent-oriented: requests carry `previous_response_id` so the server continues the history, and the `output` array promises a record of the full process | [API reference](https://platform.openai.com/docs/api-reference/responses) · [Open Responses spec](https://www.openresponses.org/specification) |
| **AI SDK (Vercel)** | Vercel; the de facto standard of the TypeScript ecosystem | Not a wire protocol but an SDK: `generateText` returns a complete result, and `useChat` speaks the UI Message Stream protocol it defines | [generateText reference](https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text) · [UI Message Stream protocol](https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol) |
The steps below use a Chat Completions-shaped interface. Replacement implementations for a Responses-shaped interface and a streaming interface are given in steps two and four respectively.
## Step 1: send a message, get the reply
The minimal `send` does exactly three things: send `input.text` to the app's interface, put the reply into one `message` event, and report this turn's `status`:
```ts theme={null}
// agents/chat-app.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
const BASE_URL = "http://localhost:8080"; // to switch the URL per experiment, make it a factory option: see the integration guide
export default defineAgent({
name: "chat-app",
evidenceCoverage: completeEvidenceCoverage,
async send(input, ctx) {
ctx.progress({ message: "Waiting for the Chat API" });
const res = await fetch(`${BASE_URL}/v1/chat/completions`, { // ← the interface the frontend already uses
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
messages: [{ role: "user", content: input.text }],
model: ctx.model, // ← experiment.model reaches here via ctx.model; delete this line if the app's interface doesn't support model selection
}),
signal: ctx.signal, // ← runner timeout and cancellation
}).then((r) => r.json());
return {
status: "completed",
events: [{ type: "message", role: "assistant", text: res.choices[0].message.content }],
};
},
});
```
If the interface returns a response you can keep working with but whose evidence is incomplete, report a diagnostic instead of printing the whole raw response:
```ts theme={null}
if (!res.requestId) {
ctx.diagnostic({
code: "missing-request-id",
level: "warning",
message: "the response has no request id, so it cannot be correlated with server logs",
});
}
```
`progress` is not persisted. A diagnostic is written to an Attempt-owned channel and can be reviewed on an Attempt detail page within a selected Run. If the HTTP connection fails or the response cannot be parsed, throw: the Runner records the error in `agent.run` and forms an `errored` Verdict in the `niceeval.verdict` channel.
**This step unlocks**: `t.reply`, `t.messageIncludes()`, all the conversation material for the Judge, and **model comparison** on the Experiment side. `ctx.model` comes from the Experiment's `model`; the runner passes it through verbatim and the Adapter only forwards it. For integration tiers, see [Tier](/docs/explanation/tier).
It has two obvious limitations: every turn is a brand-new conversation (a second `t.send` cannot continue from the first), and tool calls are completely invisible. The next two steps solve one each.
## Step 2: continue from earlier messages
The runner promises exactly one thing about sessions: **every `send` on the same session line gets the same `ctx.session`; a new session line (the eval's first turn, or after `t.newSession()`) gets a brand-new one.**
How to continue a session depends on the shape of the app's interface. The app's interface follows one of two common patterns, and `ctx.session` provides a pair of accessors for each:
* **Client carries the full history** (stateless server; the complete message list is sent every turn: the Chat Completions shape is the typical case) → create an adapter-private slot with `createSessionSlot()`, then use `ctx.session.get(slot)` / `set(slot, messages)`
* **Server keeps the history** (the interface takes a session id: the Responses shape's `previous_response_id`, the native sessions / threads of various SDKs) → `ctx.session.id` + `ctx.session.capture(id)`
The storyline's interface is the former:
```ts theme={null}
// agents/chat-app.ts
import { completeEvidenceCoverage, createSessionSlot, defineAgent } from "niceeval/adapter";
const BASE_URL = "http://localhost:8080";
interface Msg { role: "user" | "assistant"; content: string | null }
const historySlot = createSessionSlot("chat-app/history");
export default defineAgent({
name: "chat-app",
evidenceCoverage: completeEvidenceCoverage,
async send(input, ctx) {
const history = ctx.session.get(historySlot) ?? []; // ← unwritten on a new session line
const messages = [...history, { role: "user" as const, content: input.text }];
const res = await fetch(`${BASE_URL}/v1/chat/completions`, {
// …method, headers, signal same as step one, omitted
body: JSON.stringify({ messages, model: ctx.model }), // ← the change: send the full history; model forwarded same as step one
}).then((r) => r.json());
const reply = res.choices[0].message;
ctx.session.set(historySlot, [...messages, reply]); // ← the next turn on this line reads it back
return {
status: "completed",
events: [{ type: "message", role: "assistant", text: reply.content }],
};
},
});
```
Notice there is **no "first turn" branch**: `ctx.session.get(historySlot)` naturally returns `undefined` on a new session line, and `?? []` normalizes that to empty history. Nothing needs declaring beyond using the typed session slot: connect it and multi-turn continues; skip it and every turn is a new conversation.
If the app's interface takes a session id, the history lives on the server and the Adapter only records the id — just two changes inside `send`:
```ts theme={null}
// Inside send:
// the request carries the previous turn's id … previous_response_id: ctx.session.id ← naturally undefined on a new session line
// write back whatever id the response returns … ctx.session.capture(res.id) ← the next turn continues from it
```
`capture` only lands when no id has been recorded yet; a backend re-sending the id (or even changing it due to a fork) will not overwrite the line being continued.
**This step unlocks**: multi-turn conversations, and `t.newSession()` session isolation.
## Step 3: record usage
An agent that answers correctly but burns ten times the tokens should not get the same score as one that is frugal. Usage is `usage`, the fourth field on `Turn` alongside `events` and `status`: if the app's interface returns usage, report it truthfully, and the runner accumulates it turn by turn into the session line and the whole run. Chat Completions-shaped responses come with `usage` — copy it over; the rest of `send` is exactly the same as step two:
```ts theme={null}
// …transport and session same as step two, omitted
return {
status: "completed",
events: [{ type: "message", role: "assistant", text: reply.content }],
usage: { // ← the change: report the usage the interface returned, as-is
inputTokens: res.usage.prompt_tokens,
outputTokens: res.usage.completion_tokens,
},
};
```
The full `Usage` fields also include the optional `cacheReadTokens` / `cacheCreationTokens`, the reasoning token count `reasoningTokens`, the request count `requests`, and `costUSD` — if a gateway returns measured cost, fill that in; it takes precedence over price-table estimation. Each field is only filled when the interface actually reports it; if the interface returns no usage, leave `usage` out entirely, and other assertions are unaffected. This hand-copying is also transitional: the official converter in the next step fills in `usage` along with everything else.
**This step unlocks**: the `t.maxTokens()` / `t.maxCost()` scorers (`maxCost` uses `costUSD` or the price table in the config), plus usage in reports and `niceeval view`.
## Step 4: parse tools into events
The app's response contains more than reply text — the `tool_calls` in a Chat Completions-shaped response record which tools this turn called. The Adapter's most important job is **normalizing the interface's response into the standard event stream**: one object per thing that happened this turn, ordered by actual occurrence in `Turn.events`, each object one of the ten types below (for the actual field values, [the contract page has a complete one-turn example](/docs/explanation/adapter)):
```ts theme={null}
type StreamEvent =
| { type: "message"; role: "assistant" | "user"; text: string }
| { type: "operation.started"; operationId: string; operation:
| { kind: "tool"; name: string; input: JsonValue; tool?: ToolName }
| { kind: "subagent"; name: string; remoteUrl?: string } }
| { type: "operation.finished"; operationId: string; kind: "tool";
output?: JsonValue; status: "completed" | "failed" | "rejected" }
| { type: "operation.finished"; operationId: string; kind: "subagent";
output?: JsonValue; status: "completed" | "failed" }
| { type: "skill.loaded"; skill: string; operationId?: string }
| { type: "input.requested"; request: InputRequest }
| { type: "thinking"; text: string }
| { type: "compaction"; reason?: string }
| { type: "error"; message: string };
```
Parsing is one "response field → event" mapping. Hand-written it looks like this — the rest of `send` is exactly the same as step two:
```ts theme={null}
// …transport and session same as step two, usage same as step three, omitted
const reply = res.choices[0].message;
const events: StreamEvent[] = [];
for (const call of reply.tool_calls ?? []) {
events.push({
type: "operation.started",
operationId: call.id,
operation: {
kind: "tool",
name: call.function.name,
input: JSON.parse(call.function.arguments),
},
});
}
if (reply.content) events.push({ type: "message", role: "assistant", text: reply.content });
return { events, status: "completed", usage };
```
But you usually **do not have to write this loop**. When the response is a standard shape, one line of official converter replaces all of the above — `events`, `status`, even step three's hand-copied `usage` are all in the return value; just `return` it:
```ts theme={null}
// …transport and session same as step two, omitted
return turnFromChatCompletion(res); // ← the mapping above plus step three's usage, official edition
```
If the interface is not this shape, swap in the matching piece:
| Interface response shape | Built-in piece | What you need to write |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
| Chat Completions-shaped response | `turnFromChatCompletion(res)` | No mapping to hand-write |
| Responses-shaped response | `turnFromResponses(res)` | No mapping to hand-write |
| AI SDK `generateText`'s complete result | `turnFromAiSdk(result)` | No mapping to hand-write |
| Streaming: SDK-native events passed through (one complete unit per frame) | `createClaudeSdkEventStream()` / `createPiAgentEventStream()` / `createCodexThreadEventStream()` | No mapping to hand-write |
| Streaming: the AI SDK's UI Message Stream (the `useChat` protocol) | `uiMessageStreamAgent()` | No `send` or event mapping to hand-write |
| Streaming: token-by-token / argument-by-argument deltas | The protocol's official reducer; if there is none, `deltaStream({ toOps })` | One "frame type → operation" mapping table |
| Final answer only | A few hand-written events | Start with one `message` (step one is this tier) |
The built-in converters work by response shape, not by assuming a specific application protocol. Only delta streams with no ready-made reducer from the protocol side require you to write a mapping; the mapping only declares which operation each frame corresponds to — concatenation, pairing, and landing timing are handled by `deltaStream`.
Once normalized, whichever events you emit determine which family of assertions eval authors can write:
| Events you emit | Assertions unlocked |
| ------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `message` | `t.reply`, `t.messageIncludes()`, Judge conversation material |
| tool `operation.started` / `operation.finished` (paired by `operationId`) | `t.calledTool()` / `t.toolOrder()` / `t.maxToolCalls()` / `t.noFailedActions()`… |
| `input.requested` (together with status `"waiting"`) | `t.parked()`, `t.requireInputRequest()`, `t.respond()` (step five) |
| `thinking` / `compaction` / `error` | The matching assertions and o11y counts |
The Chat Completions response **does not guarantee a complete process record**. The app may finish the tool loop server-side and return only the final answer. Because of this, `turnFromChatCompletion`'s return carries no completeness proof: positive assertions like `calledTool` work, but negative assertions like `notCalledTool` will flag that the evidence is incomplete. The Responses protocol requires the `output` array to record the complete process, so `turnFromResponses`'s return carries a completeness proof and negative assertions are trustworthy. The difference in trustworthiness between the two comes from the interface contract.
**This step unlocks**: the whole family of tool assertions — `t.calledTool()`, `t.toolOrder()`, `t.maxToolCalls()`, `t.noFailedActions()`, and more.
## Step 5: HITL
When the app stops mid-turn to wait for a human (tool approval, missing information), `send` has obligations on both sides:
* **The pausing turn**: return `status: "waiting"`, and emit one `input.requested` event with a stable `id` per pending question — `t.parked()` and `t.requireInputRequest()` read them, and answers are matched by this `id`.
* **The answer turn**: `t.respond(...)` in the eval reaches the Adapter as **just another ordinary `send`** (still the same session line, the same state); the human verdict arrives in structured form via `input.responses`, each entry carrying `requestId` and `optionId` or `text` (for the shapes, see [Inputs for the different answers](/docs/explanation/adapter)). The Adapter hands the verdict back to the app first, then continues fetching the result. For a call a human rejected, set the tool `operation.finished` event's `status` to `"rejected"` rather than `"failed"` — a rejection is a human decision, not a tool failure, so `noFailedActions()` does not misfire.
The "scene read halfway through when the turn paused" (say, an SSE stream read halfway through) also lives on `ctx.session`: create an adapter-private `createSessionSlot()`, call `ctx.session.set(slot, scene)` when pausing, and `ctx.session.take(slot)` at the start of the answer turn to get it back — taking it clears it, a single consumption.
HITL almost always happens on streaming interfaces (pausing mid-stream), so this step's example switches to an app that passes native events through over SSE — it exercises everything from the earlier steps together (complete runnable version in the [tier1 example](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1)):
```ts theme={null}
// agents/my-app.ts
import { completeEvidenceCoverage, createSessionSlot, defineAgent, sseJsonFrames, createPiAgentEventStream, driveFrameStream } from "niceeval/adapter";
import type { AgentContext, PiAgentStream, SseFrameCursor } from "niceeval/adapter";
import type { Turn, TurnInput } from "niceeval";
const BASE_URL = "http://localhost:5299"; // the app's own port; evals do not manage the process
interface Pending { cursor: SseFrameCursor; stream: PiAgentStream; callId: string }
const pendingSlot = createSessionSlot("my-app/pending");
function readStream(cursor: SseFrameCursor, ctx: AgentContext, stream: PiAgentStream): Promise {
return driveFrameStream(cursor, stream, ctx, (frame) => {
if (frame.type === "session") {
ctx.session.capture(frame.sessionId); // ← step two: write back the returned id; the next turn continues from it
return;
}
if (frame.type === "approval_request") { // ← this step: the app stops for approval; hold the scene
ctx.session.set(pendingSlot, { cursor, stream, callId: frame.toolCallId });
return { pause: { id: frame.toolCallId, action: frame.toolName,
options: [{ id: "approve" }, { id: "deny" }] } };
// driveFrameStream on pause: emits an input.requested, sets status to "waiting", stops reading the stream
}
if (frame.type === "server_error") return { fail: frame.message };
});
}
export default defineAgent({
name: "my-app",
evidenceCoverage: completeEvidenceCoverage,
async send(input: TurnInput, ctx: AgentContext): Promise {
const held = ctx.session.take(pendingSlot);
if (held) { // ← t.respond("approve"): an ordinary send; deliver the verdict first
const approved = input.responses?.[0]?.optionId === "approve"; // with multiple pending requests, match by requestId
if (!approved) held.stream.markRejected(held.callId);
await fetch(`${BASE_URL}/api/chat/approve`, { method: "POST", signal: ctx.signal,
body: JSON.stringify({ toolUseId: held.callId, approved }) });
return readStream(held.cursor, ctx, held.stream); // then keep reading the same stream; no new request
}
const res = await fetch(`${BASE_URL}/api/chat`, { // ← transport: the only part you truly hand-write
method: "POST",
body: JSON.stringify({
message: input.text, // ← step one: send the message
model: ctx.model, // ← step one: experiment.model forwarded
sessionId: ctx.session.id, // ← step two: absent on the first send, present automatically afterward
}),
signal: ctx.signal,
});
return readStream(sseJsonFrames(res.body!), ctx, createPiAgentEventStream()); // ← step four: official converter, zero mapping
},
});
```
For interfaces that don't need HITL, delete the three parts related to the held pause-scene (`Pending`, its session slot, and the opening `take` branch) — the rest stays the same. For the complete mental model of pausing / answering / resuming, see [HITL](/docs/explanation/hitl).
**This step unlocks**: `t.parked()`, `t.requireInputRequest()`, `t.respond()` / `t.respondAll()`, and the precise assertion `calledTool(..., { status: "rejected" })`.
## Step 6: hook up OTel traces
If the app is already instrumented (standard OTel HTTP server instrumentation is enough), the integration splits into two halves — one is startup-time configuration, the other lives in `send`. Telling them apart is telling apart "what never changes" from "what changes every turn".
**The endpoint is startup-time configuration; it is not passed from `send`.** [NiceEval](https://niceeval.com/)'s OTLP receiver address is the same on every run, so it does not go through `ctx`: pin the receiver port in `niceeval.config.ts`, point the app's OTel exporter at that fixed URL on startup, and you never need to touch it again no matter how many eval runs follow:
```ts theme={null}
// niceeval.config.ts
import { defineConfig } from "niceeval";
export default defineConfig({
telemetry: { port: 4318 }, // the receiver always listens on http://localhost:4318/v1/traces
});
```
```bash theme={null}
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces node server.js
```
**What `send` passes is this turn's trace context, not the endpoint.** `ctx.telemetry.headers` is a W3C `traceparent` header the runner generates fresh every turn — spread it into the request, and the spans your app produces this turn attach precisely to this turn's trace, with no misattribution when multiple evals run concurrently. Back on the storyline's chat-app, `send` gains exactly one line:
```ts theme={null}
const res = await fetch(`${BASE_URL}/v1/chat/completions`, {
// …rest same as step four, omitted
headers: { "content-type": "application/json", ...ctx.telemetry?.headers }, // ← the only added line
}).then((r) => r.json());
```
`ctx.telemetry` only appears when OTel integration is configured; spreading an `undefined` when it isn't configured is safe, so this line can stay permanently. Spans still arrive without this header, but attribution degrades to time windows and that agent's turns fall back to running serially — carrying it is what makes attribution accurate under concurrency.
**This step unlocks**: the call waterfall shown per turn in `niceeval view`, including model calls, tool execution, duration, and tokens. Assertions still read the events produced by the earlier steps; spans are only used for the waterfall. For receiver configuration and span attribution rules, see [OTel integration](/docs/tutorials/connect-otel).
## Step 7: pass through the experiment's flags (A/B comparison)
Once the app exposes variants as switchable configuration, the experiment declares `flags`, and the runner hands them to `send` verbatim via `ctx.flags` every turn; the Adapter does not interpret their meaning, it only forwards them with the request — the app switches variants based on the parameter:
```ts theme={null}
// …rest of send same as before, omitted
body: JSON.stringify({
messages,
model: ctx.model, // ← experiment.model, already forwarded since step one; listed here alongside flags
flags: ctx.flags, // ← experiment.flags passed through as-is; {} when not configured
}),
```
```ts theme={null}
// experiments/concise.ts
import { defineExperiment } from "niceeval";
import chatApp from "../agents/chat-app.ts";
export default defineExperiment({
agent: chatApp,
model: "gpt-5.4", // ← reaches send via ctx.model
flags: { promptVariant: "concise" }, // ← reaches send via ctx.flags every turn
});
```
Two experiment files, each declaring its own `flags`, running the same set of evals via `npx niceeval exp` separately — that's an A/B comparison. This is Tier 3 of the three integration tiers (it requires the app to cooperate by exposing switches); for the cost and payoff, see [Tier](/docs/explanation/tier); `flags` alongside `model`, `attempts`, and the rest of the experiment fields are covered in [Write experiments](/docs/tutorials/write-experiment).
**This step unlocks**: score comparison across variants over the same set of evals.
## Reference: the five t APIs as send sees them
With the seven steps written, look back at the five driving APIs on the eval side — reaching `send`, they are just the same function receiving different fields; there is no second method to implement:
| Eval-side API | Calls to the Adapter | What `send` receives |
| ------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `t.send(text)` | One `send` | `input.text`; `ctx.session` is this session line's own state |
| `t.sendFile(path, text?)` | One `send` | Same as above, plus `input.files` (base64 `InputFile[]`); put them into the request however the app's interface expects, or ignore them if it is not multimodal |
| `t.newSession()` | Does not trigger a `send` by itself | Opens a second session line; that line's next first `send` gets **completely empty** state — exactly the same form as the eval's very first `send` |
| `t.respond(...answers)` | One **ordinary** `send` | `input.text` = the answer text; `input.responses` has one `{ requestId, optionId }` per request (`{ requestId, text }` for free-text answers); still the same line, the same state |
| `t.respondAll(optionId)` | One **ordinary** `send` | Same as above, with one answer per pending request; `optionId` is validated eval-side against every request — a typo throws instead of silently reaching the app |
## Related reading
* [Adapter](/docs/explanation/adapter) — `send`'s inputs and outputs, the three integration tiers, and where capabilities come from.
* [Connect your agent](/docs/tutorials/connect-your-agent) — minimal integration, parameter passing, and optional capabilities.
* [HITL](/docs/explanation/hitl) — the complete concept of pausing to wait for a human: handshake timing and both sides' obligations.
* [Drive](/docs/explanation/drive) — the eval side's usage of `t.send()`, `t.newSession()`, and HITL.
* [Assert](/docs/explanation/assert) — the complete assertion vocabulary driven by the standard event stream.