> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Authoring evals: single-turn, multi-turn, and dataset patterns

> Write evals with defineEval, covering single-turn conversations, multi-turn conversations, data-driven testing, Sandbox workspaces, and the eval lifecycle and fixtures.

## The `defineEval` shape

```ts theme={null}
import { defineEval, type JsonValue } from "niceeval";

export default defineEval({
  description?: string;
  tags?: string[];
  judge?: JudgeConfig;
  reporters?: Reporter[];
  timeoutMs?: number;
  metadata?: Record<string, JsonValue>;
  async setup(sandbox, ctx) { /* task fixture + progress/diagnostic */ },
  async test(t) { /* interactions + assertions */ },
});
```

## Single-turn evals

```ts theme={null}
import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
    t.check(t.reply, includes("sunny"));
  },
});
```

`t.send()` drives one interaction, `t.succeeded()` and `t.calledTool()` are scoped assertions, and `t.check()` is a value assertion recorded immediately.

### The `Turn` object

| Property         | Meaning                                     |
| ---------------- | ------------------------------------------- |
| `turn.events`    | Standard event stream, the main fact source |
| `turn.data`      | Structured output                           |
| `turn.status`    | `"completed"`, `"failed"`, or `"waiting"`   |
| `turn.usage`     | usage such as tokens and cost               |
| `turn.message`   | assistant text reply                        |
| `turn.toolCalls` | tool calls in this turn                     |

## Multi-turn evals

```ts theme={null}
export default defineEval({
  description: "Draft an email, then send it on confirmation",
  async test(t) {
    const draft = await t.send("Draft a follow-up email.");
    draft.succeeded();
    t.check(draft.message, includes("Best"));

    await t.send("Looks good, send it.");
    t.calledTool("send_email");
  },
});
```

Use `t.newSession()` when you need parallel, independent sessions.

## Dataset fan-out

One file can export an array of evals:

```ts theme={null}
export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);
```

That generates IDs such as `sql/0000`, `sql/0001`, and so on. See [Dataset fan-out](/docs/tutorials/dataset-fanout).

## Sandbox workspace

Coding-agent evals are still normal `.eval.ts` files. `test` just prepares a Sandbox workspace, sends a task, and inspects the resulting files:

```ts theme={null}
import { defineEval } from "niceeval";

export default defineEval({
  description: "Create a Button component",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
    await t.send("Create src/components/Button.tsx with label and onClick props.").then((turn) => turn.succeeded());
    t.sandbox.fileChanged("src/components/Button.tsx");
  },
});
```

See [Fixtures](/docs/tutorials/fixtures).

## Report progress and diagnostics

`setup` handles this eval's Fixture. Its second argument is bound to the eval's setup phase; feedback from inside `test(t)` is bound to the eval's run phase:

```ts theme={null}
export default defineEval({
  async setup(sandbox, ctx) {
    ctx.progress({ message: "Installing fixture dependencies" });
    await sandbox.runCommand("npm", ["install"]);
  },

  async test(t) {
    t.progress({ message: "Uploading hidden tests", current: 1, total: 2 });
    await t.sandbox.uploadDirectory("../fixtures/project");

    const preflight = await inspectFixture();
    if (preflight.usedFallback) {
      t.diagnostic({
        code: "fixture-check-degraded",
        level: "warning",
        message: "Fixture preflight fell back to the backup checker",
        data: { checker: preflight.checker },
      });
    }

    await t.send("Finish the task");
  },
});
```

`progress` only updates short-lived status while the run is in flight; it never enters the result. `diagnostic` is committed with the current Attempt into the Record, but it does not replace an assertion or change the verdict on its own: business conclusions still come from `t.check` / `t.require` / gates, and you throw when the infrastructure cannot continue.

## Eval lifecycle

`setup(sandbox, ctx)` prepares the Fixture; pair it with a `teardown(sandbox, ctx)` to clean up. Together they form this eval's Fixture, and each runs once per Attempt. Execution order: `setup` runs after the Sandbox lifecycle hooks and the git baseline anchor, and before `test(t)`; `teardown` is the first link in the Attempt's teardown chain (eval teardown first, then agent teardown, then Sandbox teardown) — the Sandbox is still alive at this point, so teardown code can read it as usual.

Most Fixtures do not need `teardown` — starting files written into the Sandbox and dependencies installed there disappear automatically when the Sandbox is destroyed. `teardown` is for Fixtures **outside the Sandbox**: temporary resources this Attempt created in a shared external service (a temporary repo, bucket, queue topic) that leak unless you clean them up.

Multiple Attempts of the same eval (`attempts` greater than 1, or several experiments in the same batch running the same eval) run concurrently and share the same module, so a `setup` handle cannot live in a plain module variable — a later concurrent Attempt would overwrite it. Key it off the `sandbox` instance instead (`WeakMap`): a sandbox maps one-to-one to an Attempt, so it is a natural per-attempt key:

```ts theme={null}
// evals/pr-review/close-outdated.eval.ts
import { defineEval } from "niceeval";
import type { Sandbox } from "niceeval/sandbox";

// Concurrent Attempts share this module: key the handle off sandbox, not a plain module variable
const fixtures = new WeakMap<Sandbox, { repoUrl: string; destroy(): Promise<void> }>();

export default defineEval({
  async setup(sandbox, ctx) {
    ctx.progress({ message: "seeding fixture repo" });
    const fixture = await createFixtureRepo("pr-review/close-outdated"); // a temporary resource outside the sandbox
    fixtures.set(sandbox, fixture);
    await sandbox.runCommand("git", ["clone", fixture.repoUrl, "workspace"]);
  },
  async teardown(sandbox) {
    await fixtures.get(sandbox)?.destroy(); // also reached when setup throws: skip if nothing was created
  },
  async test(t) { /* drive the agent to clean up the outdated PR, assert */ },
});
```

`teardown` runs if and only if this Attempt reached `setup` — a thrown error in `setup` does not exempt it, so teardown code must guard against resources that may not have been created. If `teardown` throws, or exceeds its 30-second cleanup budget, that only records a `teardown-failed` diagnostic; it does not change the verdict this Attempt already produced. To make a cleanup step affect the outcome, throw from `setup` or `test` — do not expect `teardown` to change the verdict.

## Naming conventions

<CardGroup cols={2}>
  <Card title="Filename" icon="file">
    Only `.eval.ts` files are discovered by the runner.
  </Card>

  <Card title="Directory grouping" icon="folder">
    `evals/billing/refund.eval.ts` becomes the ID `billing/refund`.
  </Card>

  <Card title="Dataset" icon="database">
    Good for many cases with the same structure and different inputs.
  </Card>

  <Card title="Sandbox workspace" icon="box">
    Good for coding agents that need a real file system, commands, and diffs.
  </Card>
</CardGroup>