Authoring evals: single-turn, multi-turn, and dataset patterns

The `defineEval` shape

import { defineEval, type JsonValue } from "niceeval";

export default defineEval({
  description?: string;
  tags?: string[];
  judge?: JudgeConfig;
  reporters?: Reporter[];
  timeoutMs?: number;
  metadata?: Record<string, JsonValue>;
  async setup(sandbox, ctx) { /* task fixture + progress/diagnostic */ },
  async test(t) { /* interactions + assertions */ },
});

Single-turn evals

import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
    t.check(t.reply, includes("sunny"));
  },
});

t.send() drives one interaction, t.succeeded() and t.calledTool() are scoped assertions, and t.check() is a value assertion recorded immediately.

The `Turn` object

Property	Meaning
`turn.events`	Standard event stream, the main fact source
`turn.data`	Structured output
`turn.status`	`"completed"`, `"failed"`, or `"waiting"`
`turn.usage`	usage such as tokens and cost
`turn.message`	assistant text reply
`turn.toolCalls`	tool calls in this turn

Multi-turn evals

export default defineEval({
  description: "Draft an email, then send it on confirmation",
  async test(t) {
    const draft = await t.send("Draft a follow-up email.");
    draft.succeeded();
    t.check(draft.message, includes("Best"));

    await t.send("Looks good, send it.");
    t.calledTool("send_email");
  },
});

Use t.newSession() when you need parallel, independent sessions.

Dataset fan-out

One file can export an array of evals:

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

That generates IDs such as sql/0000, sql/0001, and so on. See Dataset fan-out.

Sandbox workspace

Coding-agent evals are still normal .eval.ts files. test just prepares a Sandbox workspace, sends a task, and inspects the resulting files:

import { defineEval } from "niceeval";

export default defineEval({
  description: "Create a Button component",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
    await t.send("Create src/components/Button.tsx with label and onClick props.").then((turn) => turn.succeeded());
    t.sandbox.fileChanged("src/components/Button.tsx");
  },
});

See Fixtures.

Report progress and diagnostics

setup handles this eval’s Fixture. Its second argument is bound to the eval’s setup phase; feedback from inside test(t) is bound to the eval’s run phase:

export default defineEval({
  async setup(sandbox, ctx) {
    ctx.progress({ message: "Installing fixture dependencies" });
    await sandbox.runCommand("npm", ["install"]);
  },

  async test(t) {
    t.progress({ message: "Uploading hidden tests", current: 1, total: 2 });
    await t.sandbox.uploadDirectory("../fixtures/project");

    const preflight = await inspectFixture();
    if (preflight.usedFallback) {
      t.diagnostic({
        code: "fixture-check-degraded",
        level: "warning",
        message: "Fixture preflight fell back to the backup checker",
        data: { checker: preflight.checker },
      });
    }

    await t.send("Finish the task");
  },
});

progress only updates short-lived status while the run is in flight; it never enters the result. diagnostic is committed with the current Attempt into the Record, but it does not replace an assertion or change the verdict on its own: business conclusions still come from t.check / t.require / gates, and you throw when the infrastructure cannot continue.

Eval lifecycle

setup(sandbox, ctx) prepares the Fixture; pair it with a teardown(sandbox, ctx) to clean up. Together they form this eval’s Fixture, and each runs once per Attempt. Execution order: setup runs after the Sandbox lifecycle hooks and the git baseline anchor, and before test(t); teardown is the first link in the Attempt’s teardown chain (eval teardown first, then agent teardown, then Sandbox teardown) — the Sandbox is still alive at this point, so teardown code can read it as usual. Most Fixtures do not need teardown — starting files written into the Sandbox and dependencies installed there disappear automatically when the Sandbox is destroyed. teardown is for Fixtures outside the Sandbox: temporary resources this Attempt created in a shared external service (a temporary repo, bucket, queue topic) that leak unless you clean them up. Multiple Attempts of the same eval (attempts greater than 1, or several experiments in the same batch running the same eval) run concurrently and share the same module, so a setup handle cannot live in a plain module variable — a later concurrent Attempt would overwrite it. Key it off the sandbox instance instead (WeakMap): a sandbox maps one-to-one to an Attempt, so it is a natural per-attempt key:

// evals/pr-review/close-outdated.eval.ts
import { defineEval } from "niceeval";
import type { Sandbox } from "niceeval/sandbox";

// Concurrent Attempts share this module: key the handle off sandbox, not a plain module variable
const fixtures = new WeakMap<Sandbox, { repoUrl: string; destroy(): Promise<void> }>();

export default defineEval({
  async setup(sandbox, ctx) {
    ctx.progress({ message: "seeding fixture repo" });
    const fixture = await createFixtureRepo("pr-review/close-outdated"); // a temporary resource outside the sandbox
    fixtures.set(sandbox, fixture);
    await sandbox.runCommand("git", ["clone", fixture.repoUrl, "workspace"]);
  },
  async teardown(sandbox) {
    await fixtures.get(sandbox)?.destroy(); // also reached when setup throws: skip if nothing was created
  },
  async test(t) { /* drive the agent to clean up the outdated PR, assert */ },
});

teardown runs if and only if this Attempt reached setup — a thrown error in setup does not exempt it, so teardown code must guard against resources that may not have been created. If teardown throws, or exceeds its 30-second cleanup budget, that only records a teardown-failed diagnostic; it does not change the verdict this Attempt already produced. To make a cleanup step affect the outcome, throw from setup or test — do not expect teardown to change the verdict.

Naming conventions

Filename

Only .eval.ts files are discovered by the runner.

Directory grouping

evals/billing/refund.eval.ts becomes the ID billing/refund.

Dataset

Good for many cases with the same structure and different inputs.

Sandbox workspace

Good for coding agents that need a real file system, commands, and diffs.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Authoring evals: single-turn, multi-turn, and dataset patterns

The `defineEval` shape

Single-turn evals

The `Turn` object

Multi-turn evals

Dataset fan-out

Sandbox workspace

Report progress and diagnostics

Eval lifecycle

Naming conventions

Filename

Directory grouping

Dataset

Sandbox workspace

​The defineEval shape

​Single-turn evals

​The Turn object

​Multi-turn evals

​Dataset fan-out

​Sandbox workspace

​Report progress and diagnostics

​Eval lifecycle

​Naming conventions

Filename

Directory grouping

Dataset

Sandbox workspace

The `defineEval` shape

Single-turn evals

The `Turn` object

Multi-turn evals

Dataset fan-out

Sandbox workspace

Report progress and diagnostics

Eval lifecycle

Naming conventions