How NiceEval drives agents: send, sessions, and HITL

Before assertions or Judge evaluation, you need to make the agent do something. Drive is the part of test(t) that sends input and gets results back: t.send(), t.sendFile(), t.newSession(), and HITL through t.respond() / t.respondAll(). Every drive API produces a Turn. All assertions and Judge checks in NiceEval read from Turn data. See Assert and Judge for the assertion side.

`t.send()` and the `Turn` it returns

t.send(input) is the core verb. Under the hood it calls the Adapter’s send(input, ctx) and normalizes the return value into a Turn:

const turn = await t.send("What's the weather like in Brooklyn today?");

turn.message;    // last assistant message
turn.data;       // structured output: only when this send declared output; typed by the declaration (see below)
turn.status;     // "completed" | "failed" | "waiting"
turn.events;     // StreamEvent[] for this turn
turn.usage;      // { inputTokens, outputTokens, ... } when the adapter reports it
turn.succeeded(); // scored assertion: this turn didn't fail and isn't parked on HITL

t.reply is shorthand for the last assistant message in the main session, equivalent to turn.message from the most recent t.send(). t.events is the full accumulated event stream for the main session. When you need structured output, declare output on the send call (any Standard Schema, so zod works). The declaration is both a requirement for the app and the type source for turn.data:

const turn = await t.send("What is the total on this invoice?", {
  output: z.object({ amount: z.number(), currency: z.string() }),
});
t.check(turn.data.amount, equals(3200));   // turn.data is { amount: number; currency: string }

The runner validates data against that declaration immediately after the adapter returns: if data does not match, the turn goes straight to failed and the diff is reported. A turn that did not declare output has no turn.data — it is not a pocket for “whatever the agent happens to return”, but a field that exists only when you declare it, and is strongly typed when it does.

turn.succeeded() scopes to this turn only: it checks this turn’s status and whether it’s parked on an unanswered HITL request. Call it right after a t.send() that later turns depend on, so a failure surfaces here instead of a pile of confusing follow-on failures.

One turn with a file: `t.sendFile()`

t.sendFile(path, text?) reads a local file relative to the eval directory, infers the MIME type from the extension, and attaches it to the turn input as a data URL:

const turn = await t.sendFile("fixtures/invoice.png", "What is the total on this invoice?");
t.check(turn.message, includes("$"));

Multi-turn conversations

Every await t.send(...) is a new turn on the same conversation. Keep each return value in a local variable when you want to assert on that turn specifically:

const draft = await t.send("Draft a follow-up email.");
draft.succeeded();
t.check(draft.message, includes("Best"));
draft.judge.autoevals.closedQA("Is the tone professional?").atLeast(0.6);

await t.send("Looks good, send it.");
t.calledTool("send_email");

Whether multi-turn calls actually continue the same conversation depends on whether the adapter’s send uses the continuation helpers from ctx.session (history() or id + capture()). If it does not, each turn becomes a fresh conversation. See Adapter and How to write send.

Independent sessions: `t.newSession()`

t.newSession() opens a second conversation line that runs in parallel with the main one and should not share state. The returned handle has the same drive APIs (send, sendFile, respond, respondAll) and the same scoped assertions, but only sees its own events:

await t.send("My name is Alice.");
await t.send("What is my name?");
t.check(t.reply, includes("Alice"));         // the main session remembered

const fresh = t.newSession();
await fresh.send("What is my name?");
t.check(fresh.reply, satisfies((r) => !r.includes("Alice"), "no memory leak"));

Do not assume t.newSession() guarantees isolation by itself. The runner only guarantees that the new session line starts with empty state. Actually opening a fresh conversation against the application is the adapter’s job. An adapter that ignores its own session state and always resumes the same underlying context will make t.newSession() quietly share state, without ever raising an error. When you write an adapter, verify isolation with an eval like the one above before trusting it.

Human-in-the-loop (HITL)

Some agents stop in the middle of a turn and wait for approval or missing information instead of finishing immediately. In that case the turn ends with status: "waiting" and one or more input.requested events that describe what it is waiting for. The full model is in HITL; here the focus is how evals drive it.

const draft = await t.send("Draft a follow-up email, but don't send it until I confirm.");
draft.parked();                              // t.parked() asserts status === "waiting"

const request = t.requireInputRequest({
  prompt: /send it/,
  optionIds: ["approve", "reject"],
});

await t.respond({ request, optionId: "approve" });
t.calledTool("send_email");

t.requireInputRequest(filter) turns a pending HITL request into a concrete value you can inspect and answer. It throws when 0 or more than 1 requests match, so fill as many filter fields as you can (id, prompt, display, action, optionIds, input) to disambiguate. t.respond(...) answers it and emits the next turn. String arguments answer requests in order; object arguments such as { request, optionId } target a specific request directly. Under the hood this is still just another send: the answer text goes into input.text, and structured answers go into input.responses one per request — an answer that hits one of the request’s options carries { requestId, optionId }, and a free-text answer carries { requestId, text }. The adapter never has to parse text to figure out which answer belongs to which request; the requestId already lines them up. See Inputs for the different answers for what each answer shape looks like once it reaches the adapter. When the current turn has multiple pending requests of the same kind and they should all get the same answer, use t.respondAll(optionId) instead of resolving them one by one. The optionId is validated against every pending request first — one that is not in a request’s options throws immediately instead of silently being sent as the wrong answer:

await t.send("Ask for approval on each of these file changes.");
t.requireInputRequest({ display: /approval/ });

await t.respondAll("approve");
t.succeeded();

HITL — The full concept for paused turns, request/response handshakes, and adapter obligations.
Assert — Assertions that read from Turn.events and Turn.data.
Judge — What t.judge, session.judge, and turn.judge score by default.
Adapter — Where capabilities come from: what unlocks t.newSession(), HITL, and tool assertions.

Core Model

Interaction & Assertions

Execution Model

How NiceEval drives agents: send, sessions, and HITL

`t.send()` and the `Turn` it returns

One turn with a file: `t.sendFile()`

Multi-turn conversations

Independent sessions: `t.newSession()`

Human-in-the-loop (HITL)

​t.send() and the Turn it returns

​One turn with a file: t.sendFile()

​Multi-turn conversations

​Independent sessions: t.newSession()

​Human-in-the-loop (HITL)

​Related reading

`t.send()` and the `Turn` it returns

One turn with a file: `t.sendFile()`

Multi-turn conversations

Independent sessions: `t.newSession()`

Human-in-the-loop (HITL)

Related reading