How to drive an Agent in NiceEval: send, sessions, and HITL

Before writing an assertion or Judge check, make the Agent do something. Drive is the part of test(t) that sends input and receives results: t.send(), t.sendFile(), t.newSession(), and the HITL methods t.respond() / t.respondAll(). Drive produces a Turn. Every assertion and Judge check in NiceEval reads data from a Turn; see Assert and Judge.

`t.send()` and the `Turn` it returns

t.send(input) is the Adapter’s basic execution verb. Under the hood, it calls the Adapter’s send(input, ctx) and normalizes the result into a Turn:

const turn = await t.send("What's the weather like in Brooklyn today?");

turn.reply;   // The last assistant message
turn.data;      // Structured output: present only if this send declared output; its type follows that declaration (below)
turn.status;    // "completed" | "failed" | "waiting"
turn.events;    // This turn's StreamEvent[]
turn.usage;     // { inputTokens, outputTokens, ... }, only if the Adapter reported it
turn.succeeded().label("Turn completed"); // This turn did not fail or remain on an unanswered HITL request

t.reply is shorthand for the main session’s last assistant message. It is equivalent to the turn.reply from the latest t.send(). t.events is the full accumulated event stream for the main session. When you need structured output, declare output on send using a Standard Schema such as zod. The declaration both asks the application for that output and supplies the type of turn.data:

const turn = await t.send("What is the total amount on this invoice?", {
  output: z.object({ amount: z.number(), currency: z.string() }),
});
t.check(turn.data.amount, equals(3200));   // turn.data is { amount: number; currency: string }

The Runner validates the value against the declaration immediately after the Adapter returns. If data does not match, the turn is immediately failed and reports the difference. A turn that did not declare output has no turn.data. It is not a pocket for “whatever the Agent happened to return”; the field exists only when you declare it and is strongly typed when it does.

turn.succeeded() looks only at this turn: its status and whether it is waiting for an unanswered HITL request. If later turns depend on an earlier one succeeding, use await turn.succeeded().orStop().

One turn with a file: `t.sendFile()`

t.sendFile(path, text?) reads a local file relative to the eval case’s directory, infers a MIME type from the extension, and attaches it to this turn’s input as a data URL:

const turn = await t.sendFile("fixtures/invoice.png", "What is the total amount on this invoice?");
t.check(turn.reply, includes("$"));

Multi-turn conversations

Each await t.send(...) is a new turn in the same conversation. Assign each response to a local variable so you can assert that individual turn as well as use run-level t assertions:

const draft = await t.send("Help me draft a follow-up email.");
draft.succeeded().label("Draft completed");
t.check(draft.reply, includes("Best regards"));
t.judge({ request: draft.input, draft: draft.reply }, toneQuality)
  .gate(0.6).label("Professional tone");

await t.send("Okay, send it.");
t.calledTool("send_email").label("Sent");

Whether multi-turn t.send() calls truly continue their context depends on whether the Adapter’s send uses ctx.session session-continuation storage, either a typed key or id + capture(). Without it, every turn is a new conversation. See Adapter and Write Send.

Independent sessions: `t.newSession()`

t.newSession() opens a second conversation line parallel to the main session, without interfering with it. Its returned session handle has the same drive API (send / sendFile / respond / respondAll) and scoped-assertion vocabulary, but sees only that session’s events:

await t.send("My name is Xiaoming.");
await t.send("What is my name?");
t.check(t.reply, includes("Xiaoming"));         // The main session remembers

const fresh = t.newSession();
await fresh.send("What is my name?");
t.check(fresh.reply, satisfies("No memory leak", (reply) => !reply.includes("Xiaoming")));

Do not assume that t.newSession() inherently guarantees isolation. The Runner only guarantees that the new session line starts with empty state. Opening a fresh conversation with the application is the Adapter’s job. An Adapter that ignores its session state and always resumes the same underlying context makes t.newSession() silently share state without an error. When you write an Adapter, verify isolation with an eval case like the one above before you trust it.

Human-in-the-loop (HITL)

Some Agents stop during a turn to wait for approval or missing information instead of finishing immediately. That turn ends with status: "waiting" and has one or more input.requested events that say what it needs. For the full mental model—handshake sequence, Adapter obligations, and rejected semantics—see HITL. This section explains how to use it on the eval side.

import { equals } from "niceeval/expect";

const draft = await t.send("Draft a follow-up email, but do not send it until I confirm.");
t.check(draft.status, equals("waiting")).label("Paused for approval");

const request = t.requireInputRequest({
  prompt: /send/,
  optionIds: ["approve", "reject"],
});

await t.respond({ request, optionId: "approve" });
t.calledTool("send_email").label("Sent");

t.requireInputRequest(filter) turns a pending HITL request into a concrete value you can inspect and answer. It throws if zero or more than one pending request matches, so use as many filter fields as you can—id, prompt, display, action, optionIds, or input—to disambiguate. t.respond(...) answers the request and emits the next turn. Each argument is one answer. The string form is only for when exactly one request is pending. When several requests pause together, name each one explicitly with { request, optionId } or { request, text }; argument order is never used to guess. Under the hood it is another ordinary send: answer text goes to input.text, while structured answers go to input.responses one per request. A response that selects a request option carries { requestId, optionId }; a free-text response carries { requestId, text }. The Adapter does not need to parse the text: requestId identifies which answer belongs to which request. See Inputs for the different answers for how each response looks to the Adapter. If the current turn has multiple pending requests of the same kind that should all receive the same answer, such as approving a batch of file changes one by one, use t.respondAll(optionId) to answer them at once. It validates optionId against every pending request first. If the option is absent from a request’s options, it throws instead of silently sending the wrong answer:

await t.send("Submit this batch of changes for approval one by one.");
t.requireInputRequest({ display: /approval/ });

await t.respondAll("approve");
t.succeeded().label("All completed");

HITL — The complete concept of paused turns waiting for people: handshake sequence and obligations on both sides.
Assert — The assertion vocabulary that reads Turn.events and Turn.data.
Judge — Assess open-ended output with a rubric and material that has meaningful fields.
Adapter — Where capabilities come from: what unlocks t.newSession(), HITL, and tool-related assertions.

Core Model

Interaction and Scoring

Execution Model

How to drive an Agent in NiceEval: send, sessions, and HITL

`t.send()` and the `Turn` it returns

One turn with a file: `t.sendFile()`

Multi-turn conversations

Independent sessions: `t.newSession()`

Human-in-the-loop (HITL)

t.send() and the Turn it returns

One turn with a file: t.sendFile()

Multi-turn conversations

Independent sessions: t.newSession()

Human-in-the-loop (HITL)

Related reading

`t.send()` and the `Turn` it returns

One turn with a file: `t.sendFile()`

Independent sessions: `t.newSession()`