Write Evals: Single-Turn, Multi-Turn, and Datasets

The eval in the quickstart only checked whether the reply contained “30 days”. A real Agent does more: it calls the right tools, remembers context across a multi-turn conversation, and edits code correctly. This page starts from the simplest eval and adds those checks step by step.

Structure of an eval

Evals live under evals/, with file names ending in .eval.ts, and default-export defineEval(...):

// evals/billing/refund.eval.ts → ID is billing/refund
import { defineEval } from "niceeval";

export default defineEval({
  description: "Answers the refund policy correctly",
  tags: ["billing"],
  async test(t) {
    // Send a message, then check the result
  },
});

The file path decides the ID: the ID of evals/billing/refund.eval.ts is billing/refund. Only .eval.ts and .eval.tsx files are discovered.
description appears in results. Write it as one sentence about what this eval verifies.
tags are for selecting by tag, for example niceeval exp local --tag billing. The Experiment’s evals function can read them too.
test(t) is the eval body. t has every method for sending messages, asserting, and working with the Sandbox.

For other fields, such as judge, timeoutMs, sandbox, and metadata, see the defineEval reference.

Single-turn: send one message and check the result

import { defineEval } from "niceeval";
import { includes, jsonMatch, toolMatch } from "niceeval/expect";

export default defineEval({
  description: "Calls the tool when asked about the weather",
  async test(t) {
    await t.send("What's the weather in Brooklyn today?");

    t.succeeded();
    t.calledTool(toolMatch("get_weather", { input: jsonMatch({ city: "Brooklyn" }) }).exactly(1));
    t.check(t.reply, includes("sunny"));
  },
});

This uses three kinds of checks:

t.succeeded(): this turn completed normally, without failing or stopping in a waiting state.
t.calledTool(...): it called get_weather, the city in the arguments is Brooklyn, and it called it exactly once. Tool calls are only visible when the Adapter reports the matching events; see Write send.
t.check(value, match): checks one specific value, here the reply text.

A failed check does not stop the checks after it. When a run ends, you see every unmet condition at once, instead of fixing one and running again. When later code really depends on a result, use await t.check(...).orStop() to stop on failure. Add .label("...") to a check, and results show your description instead of a piece of code, which is easier to read:

t.calledTool("get_weather").label("Called the weather tool");

t.send() returns this turn’s Turn. Save it when you need to check one turn on its own:

Property	Content
`turn.reply`	The assistant’s text reply
`turn.toolCalls`	This turn’s tool calls
`turn.data`	Structured output
`turn.status`	`"completed"`, `"failed"`, or `"waiting"`
`turn.usage`	Tokens and cost
`turn.events`	The full standard event stream; every other property comes from it

Multi-turn: check that context carries over

export default defineEval({
  description: "Drafts an email first, sends it after confirmation",
  async test(t) {
    const draft = await t.send("Draft a follow-up email for me.");
    draft.succeeded();
    t.check(draft.reply, includes("Best regards"));

    await t.send("Looks good, send it.");
    t.calledTool("send_email");
  },
});

Consecutive send calls on the same t form one conversation. draft.succeeded() only checks the first turn; t.calledTool checks whether the tool was called anywhere in the eval. When you need two unrelated conversations, for example to simulate two users, open a new session with t.newSession(). Multi-turn conversation requires the Adapter to carry the previous turn’s context to the application; see Write send.

Give open-ended answers to the Judge

Questions like “is the answer grounded” or “is the tone right for elementary school students” cannot be written as a regex. Hand them to the Judge model to score:

import { defineEval } from "niceeval";
import { defineJudge } from "niceeval/expect";

const grounded = defineJudge({ rubric: "Given the question, rate whether the answer is based on facts rather than invented or vague.",
});

export default defineEval({
  async test(t) {
    const turn = await t.send("What's the weather in Beijing today?");
    t.judge({ question: turn.input, answer: turn.reply }, grounded).gate(0.7).label("Answer is grounded");
  },
});

The Judge gives a score from 0 to 1, and .gate(0.7) means anything below 0.7 fails. When a deterministic rule can check something, such as which tool was called or whether a file changed, prefer the deterministic rule; save the Judge for truly open-ended quality. To configure which model the Judge uses, see Configure and Verify the Judge.

Generate many evals from one dataset

When dozens of evals share the same logic and differ only in input, you do not need dozens of files. Export an array from one file:

// evals/sql.eval.ts
import { defineEval } from "niceeval";
import { equals } from "niceeval/expect";

const rows = [
  { task: "Count orders", prompt: "How many orders are there?", expected: "42" },
  { task: "Most expensive product", prompt: "What is the most expensive product?", expected: "MacBook Pro" },
];

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

The generated IDs are sql/0000 and sql/0001. To read a dataset from YAML or JSON and give each row a stable ID, see Data-Driven Tests.

Prepare a workspace for a Coding Agent

When evaluating a Coding Agent, the eval is still an ordinary .eval.ts. You just put a project into the Sandbox first, then let the Agent change it:

export default defineEval({
  description: "Create a Button component",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
    const turn = await t.send("Create src/components/Button.tsx that accepts label and onClick props.");
    turn.succeeded();
    t.sandbox.fileChanged("src/components/Button.tsx");
  },
});

For where the Sandbox comes from and how to score with the project’s own tests, see Fixture and Sandbox Agent.

Report progress and issues during a run

When preparation is slow, use t.progress to show what is happening in the terminal. When something is worth recording but does not affect the conclusion, record a diagnostic with t.diagnostic:

export default defineEval({
  async test(t) {
    t.progress({ message: "Uploading project", current: 1, total: 2 });
    await t.sandbox.uploadDirectory("../fixtures/project");

    const preflight = await inspectFixture();
    if (preflight.usedFallback) {
      t.diagnostic({
        code: "fixture-check-degraded",
        level: "warning",
        message: "Fixture preflight used the fallback checker",
        data: { checker: preflight.checker },
      });
    }

    await t.send("Complete the task");
  },
});

progress is only shown during the run and is not saved. diagnostic is saved with this Attempt and visible in the results afterwards, but it does not change pass or fail. Write business conclusions as assertions. When preparation fails and the run cannot continue, throw an exception directly, and this run is recorded as errored.

Prepare and clean up external resources

Some evals create temporary resources outside the Sandbox, for example a temporary repository on a shared Git service. Use setup to prepare and teardown to clean up; each runs once per Attempt:

// evals/pr-review/close-outdated.eval.ts
import { defineEval } from "niceeval";
import type { Sandbox } from "niceeval/sandbox";

// Multiple Attempts of the same eval run concurrently and share this module, so store per sandbox instead of in a plain module variable
const fixtures = new WeakMap<Sandbox, { repoUrl: string; destroy(): Promise<void> }>();

export default defineEval({
  async setup(sandbox, ctx) {
    ctx.progress({ message: "Creating temporary repository" });
    const fixture = await createFixtureRepo("pr-review/close-outdated");
    fixtures.set(sandbox, fixture);
    await sandbox.runCommand("git", ["clone", fixture.repoUrl, "workspace"]);
  },
  async teardown(sandbox) {
    await fixtures.get(sandbox)?.destroy(); // May not exist if setup failed partway
  },
  async test(t) { /* Have the Agent close outdated PRs, then assert */ },
});

Key points:

Most evals do not need teardown. Files written into the Sandbox and installed dependencies disappear when the Sandbox is destroyed. Only resources outside the Sandbox need manual cleanup.
Once setup has started, teardown always runs, even if setup throws, so guard against resources that were never created.
When teardown throws or exceeds 30 seconds, it only records a teardown-failed diagnostic and does not change the conclusion this Attempt already reached. To make a check affect the conclusion, put it in setup or test.
When attempts is greater than 1, multiple Attempts of the same eval run concurrently. Store handles in a WeakMap keyed by sandbox, one per Attempt, so they do not overwrite each other.

The full order of these steps in one Attempt: By default each Attempt has its own Sandbox, so “once per Sandbox” and “every Attempt” mean the same thing. They only diverge when reuse is enabled; see Reuse Sandboxes.

Next steps

When a task has partial completion and you want to score by how many steps were done: read Pass/Fail and Scored Evals.
To write clear criteria that do not misjudge: read NiceEval Taste.
For every matcher: see the expect reference.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting