> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# defineEval: declare, configure, and run NiceEval evals

> Reference for defineEval options, the test context t, Turn return values, Sandbox helpers, and array and keyed-record dataset exports.

`defineEval` is the main entry point for authoring evals. Each eval file calls it once, passing a description and `test(t)`, and default-exports the result.

```ts theme={null}
import { defineEval } from "niceeval";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    const turn = await t.send("What's the weather like in Brooklyn today?");
    await turn.succeeded().stopOnFailure();
  },
});
```

<Note>
  Do not provide `id` or `name`. [NiceEval](https://niceeval.com/) derives the eval ID from the file path.
</Note>

## `defineEval` options

#### `id`

```ts theme={null}
id?: string;
```

Derived from the file path; hand-writing it in the definition is not allowed.

#### `description`

```ts theme={null}
description?: string;
```

One-line description, shown in `niceeval list` and the view; purely informational — it does not affect scheduling or assertion evaluation.

#### `tags`

```ts theme={null}
tags?: string[];
```

Tags, for the CLI's `--tag` filter and view categorization; a filtering dimension independent from ID-prefix filtering.

#### `environment`

```ts theme={null}
environment?: string;
```

The environment profile id this eval needs (provider-neutral, e.g. `"python-3.9-astropy-4.2"`); translated into that provider's prebuilt artifact via the Sandbox spec's `environments` table.

#### `judge`

```ts theme={null}
judge?: JudgeConfig;
```

Overrides the project-level `Config.judge`, taking effect only for this one eval (e.g. to switch to a more expensive judge model).

#### `reporters`

```ts theme={null}
reporters?: Reporter[];
```

Overrides / adds to the project-level `Config.reporters`, taking effect only for this one eval.

#### `timeoutMs`

```ts theme={null}
timeoutMs?: number;
```

Overrides the project-level / CLI timeout for a single attempt (milliseconds), taking effect only for this one eval.

#### `metadata`

```ts theme={null}
metadata?: Record<string, JsonValue>;
```

Arbitrary extra metadata, saved as Attempt Provenance; does not participate in scheduling or assertion evaluation, and is meant for custom reporters to consume.

#### `diff`

```ts theme={null}
diff?: { include?: string[]; ignore?: string[] };
```

Adjusts the attribution exclusion list for the agent diff (Sandbox agents only; see docs/feature/eval/README.md): both arrays are
gitignore-style globs (relative to `workdir`). By default, `.git`, `node_modules`, build artifacts, and package-manager caches are excluded;
`ignore` adds further exclusions on top of the default list; `include` has the highest priority and explicitly adds matching paths back.
The composition rule is fixed as "default ∪ ignore, then punched through by include"; the list is frozen at the ledger's anchor point.

#### `setup`

```ts theme={null}
setup?: (sandbox: Sandbox, ctx: SandboxHookContext) => Promise<void> | void;
```

Eval-level preparation: receives the Sandbox (after the workspace is uploaded and the git baseline is set, before dependencies are installed).
Commands default to the environment's own declared identity (the Docker image `USER`, the Compose service `user:`, and so on); when installing system dependencies, pass `{ user: "root" }`
to `runCommand` (e.g. `runCommand("apt-get", ["install", …], { user: "root" })`) — semantics are consistent across providers.
The second parameter is a narrow context bound to `eval.setup` (`ctx.progress` / `ctx.diagnostic`,
see docs/feature/eval/README.md). `setup` returns no value; to pass an artifact through to `teardown`,
key it off the `sandbox` instance (concurrent attempts share the same module, so a plain module-level variable would get overwritten by other attempts).

#### `teardown`

```ts theme={null}
teardown?: (sandbox: Sandbox, ctx: SandboxHookContext) => Promise<void> | void;
```

Eval-level cleanup: the first link in the attempt's teardown chain (`eval.teardown` → `agent.teardown` →
`sandbox.teardown`), with the Sandbox still alive at this point. Runs if and only if `eval.setup`'s point in time was reached —
an error thrown by `setup` or `test` does not exempt it, and not declaring `setup` does not affect whether it fires. An error thrown,
or exceeding the 30s cleanup budget, only records a `teardown-failed` diagnostic and does not change the verdict. Use it for
temporary Fixtures outside the Sandbox (a temporary repo / bucket); things inside the Sandbox are reclaimed automatically when it's destroyed and don't need this.

#### `test`

```ts theme={null}
test(t: TestContext): Promise<void> | void;
```

The eval body: receives the `TestContext`, drives the conversation / Sandbox operations, and asserts in place.

## Test context: `t`

`t` (`TestContext`) is the high-level context the eval author receives. The runner assembles it according to the agent's actual capabilities — fields like `t.sandbox` are only meaningful on agents constructed with `defineSandboxAgent`; the full rule set is in the [capabilities reference](/docs/reference/capabilities). All members:

#### `send`

```ts theme={null}
send(input: SendInput): Promise<TurnHandle>;
```

Sends a message on the default session (a string or a structured message), returning that turn's `TurnHandle`. The events are also accumulated into the default session's cumulative event stream, for the scoped assertions below to use.

#### `sendFile`

```ts theme={null}
sendFile(path: string, text?: string): Promise<TurnHandle>;
```

Sends a message with an attached file (image, etc. — multimodal input). `path` is relative to the project root; after reading, it is base64-encoded and handed to the adapter via `TurnInput.files`.

#### `requireInputRequest`

```ts theme={null}
requireInputRequest(filter?: InputRequestFilter): InputRequest;
```

Retrieves a pending HITL input request on the default session; without a filter it requires exactly one, and throws if it can't get one.

#### `respond`

```ts theme={null}
respond(...responses: (string | RespondAnswer)[]): Promise<TurnHandle>;
```

Answers the pending input request(s) on the default session, returning the continuation's `TurnHandle`. String form matches requests in order;
when multiple requests are paused together and you need to name which one you're answering, use the `RespondAnswer` object form (see its type comment).

#### `respondAll`

```ts theme={null}
respondAll(optionId: string): Promise<TurnHandle>;
```

Answers every pending input request on the default session in bulk, using the same `optionId`.

#### `reply`

```ts theme={null}
readonly reply: string;
```

The assistant's reply text for the default session's most recent turn.

#### `sessionId`

```ts theme={null}
readonly sessionId: string | undefined;
```

The adapter-side default session id (only present for agents with a session concept).

#### `events`

```ts theme={null}
readonly events: readonly StreamEvent[];
```

The default session's cumulative event stream (across every turn on that session).

#### `newSession`

```ts theme={null}
newSession(): SessionHandle;
```

Opens another, independent session line, returning its `SessionHandle` — not void. This is the key entry point for
multi-session isolation: sending and asserting on the new session are isolated from the default session and from
every other `newSession()` session — commonly used for scenarios like "multiple users in parallel conversations" or
"one main line plus one side branch."

#### `signal`

```ts theme={null}
readonly signal: AbortSignal;
```

The abort signal for this attempt; fires on timeout / EarlyExit / user Ctrl-C, and is passed to the adapter's long-running calls for cancellation.

#### `model`

```ts theme={null}
readonly model?: string;
```

The model name used for this attempt (decided by the experiment/CLI flag); omitted means the agent's native default, not "no model."

#### `reasoningEffort`

```ts theme={null}
readonly reasoningEffort?: string;
```

The reasoning effort for this attempt (e.g. "low"/"medium"/"high"; valid values are decided by the adapter/model).

#### `flags`

```ts theme={null}
readonly flags: Readonly<Record<string, JsonValue>>;
```

The experiment flags in effect for this attempt (a read-only view of `experiment.flags`; an experimental condition, not a command-line switch).

#### `progress`

```ts theme={null}
progress(update: ProgressUpdate): void;
```

Scoped feedback: reports a long-running step the eval itself is performing (uploading a Fixture, running a build, ...). A short-lived
status with scope fixed to `eval.run`; it only reports, it does not assert (see docs/feature/eval/library/context.md, "Reporting long steps back to the run").

#### `diagnostic`

```ts theme={null}
diagnostic(input: DiagnosticInput): void;
```

Scoped feedback: reports an issue that should be kept around after the run ends (a permanent event, recorded in the attempt's diagnostics).
Even when `level` is "error" this does not automatically change the verdict — the test's conclusion is still decided by assertions.

#### `log`

```ts theme={null}
log(msg: string): void;
```

An alias for `progress({ message: msg })` (debug logging); does not appear in the final result.

#### `skip`

```ts theme={null}
skip(reason: string): never;
```

Immediately aborts this eval and forms a `skipped` Verdict in the `niceeval.verdict` channel; `reason` cannot be empty.

#### `check`

```ts theme={null}
check(value: unknown, assertion: ValueAssertion): AssertionHandle;
```

Runs a `ValueAssertion` against any value, returning an `AssertionHandle` that can be chained with `.gate()` / `.atLeast()`.
Assertion evaluation is deferred until finalize at the end of the eval; the call itself is synchronous and never throws — a failure just
records a failed assertion, it does not abort the rest of the code. Use `require` for "abort the eval immediately if this isn't met."

#### `require`

```ts theme={null}
require<T>(value: T, assertion: ValueAssertion): Promise<T>;
```

Runs a `ValueAssertion` against any value, evaluated immediately (when awaited); if it isn't met, throws and aborts the rest of the eval's
steps (the assertion is still recorded in the report, and does not affect other already-recorded assertions). The difference from `check`:
`check` only records and never throws, with assertion evaluation left to the end; `require` evaluates on the spot and aborts on failure — suited to
"if this prerequisite isn't met, nothing written after it matters." On success it returns the original value with its type preserved.

#### `group`

```ts theme={null}
group<T>(title: string, fn: () => Promise<T> | T): Promise<T>;
```

Groups a set of assertions under a titled section (analogous to vitest's `test('title', ...)`). Purely for organization/reporting —
it does not change assertion evaluation: every assertion inside the group is still evaluated independently. Can be nested (titles joined with ›).

#### `succeeded`

```ts theme={null}
succeeded(): AssertionHandle;
```

Asserts the default session's cumulative status is "completed" (across every turn on that session, not just the last one).

#### `parked`

```ts theme={null}
parked(): AssertionHandle;
```

Asserts the default session is currently stuck on an HITL input request.

#### `messageIncludes`

```ts theme={null}
messageIncludes(token: string | RegExp): AssertionHandle;
```

Asserts the default session's cumulative assistant replies contain `token` (across every turn on that session, not just the last one).

#### `calledTool`

```ts theme={null}
calledTool(name: string, match?: ToolMatch): AssertionHandle;
```

Asserts the default session has cumulatively called the named tool; `match` can constrain arguments / count / status.

#### `notCalledTool`

```ts theme={null}
notCalledTool(name: string, match?: ToolMatch): AssertionHandle;
```

Asserts the default session has not cumulatively called the named tool (or not under the `match` condition).

#### `toolOrder`

```ts theme={null}
toolOrder(names: string[]): AssertionHandle;
```

Asserts the default session's cumulative tool calls appear in the given order (other calls may be interleaved).

#### `usedNoTools`

```ts theme={null}
usedNoTools(): AssertionHandle;
```

Asserts the default session has not called any tool so far.

#### `maxToolCalls`

```ts theme={null}
maxToolCalls(max: number): AssertionHandle;
```

Asserts the default session's cumulative tool call count does not exceed `max`.

#### `loadedSkill`

```ts theme={null}
loadedSkill(skill: string): AssertionHandle;
```

Asserts the default session has cumulatively loaded the named skill.

#### `noFailedActions`

```ts theme={null}
noFailedActions(): AssertionHandle;
```

Asserts the default session has no cumulative failed tool calls / commands.

#### `event`

```ts theme={null}
event(type: StreamEvent["type"], opts?: { count?: number }): AssertionHandle;
```

Asserts the default session has cumulatively seen an event of the given type; `opts.count` can constrain the occurrence count.

#### `notEvent`

```ts theme={null}
notEvent(type: StreamEvent["type"]): AssertionHandle;
```

Asserts the default session has not cumulatively seen an event of the given type.

#### `calledSubagent`

```ts theme={null}
calledSubagent(name: string, match?: SubagentMatch): AssertionHandle;
```

Asserts the default session has cumulatively called the named subagent; `match` can constrain count / status / remoteUrl.

#### `eventOrder`

```ts theme={null}
eventOrder(types: StreamEvent["type"][]): AssertionHandle;
```

Asserts the default session's cumulative events appear in the given type order (other events may be interleaved).

#### `eventsSatisfy`

```ts theme={null}
eventsSatisfy(label: string, predicate: (events: readonly StreamEvent[]) => boolean): AssertionHandle;
```

Asserts against the default session's whole cumulative event stream using a custom predicate; `label` is required and becomes the assertion's title.

#### `sandbox`

```ts theme={null}
readonly sandbox: SandboxHandle;
```

A restricted Sandbox view: can run commands / read and write files / view the final diff, but cannot stop the Sandbox itself (see `SandboxHandle`).

#### `usage`

```ts theme={null}
readonly usage: Usage;
```

The default session's cumulative token usage and estimated cost.

#### `maxTokens`

```ts theme={null}
maxTokens(max: number): AssertionHandle;
```

Asserts the default session's cumulative token usage does not exceed `max`.

#### `maxCost`

```ts theme={null}
maxCost(usd: number): AssertionHandle;
```

Asserts the default session's cumulative spend (USD) does not exceed `usd`.

#### `judge`

```ts theme={null}
readonly judge: JudgeNamespace;
```

The available judge namespace (`t.judge.autoevals.*`).

## Judge assertions

```ts theme={null}
t.judge.autoevals.factuality(expected, { on: t.reply }).atLeast(0.8);
t.judge.autoevals.closedQA(question, { on: t.reply }).atLeast(0.7);
t.judge.autoevals.closedQA(rubric, { on: t.reply }).atLeast(0.75);
t.judge.autoevals.summarizes(sourceText, { on: t.reply }).atLeast(0.7);
```

`t.judge.autoevals` currently has three methods: `factuality` (whether the reply is consistent with the `expected` facts), `closedQA` (whether the reply satisfies the given question or rubric), and `summarizes` (whether the reply is a valid summary of the given source text `source`). All three directly use scorers from the [autoevals](https://github.com/braintrustdata/autoevals) (braintrust) library; `{ on: t.reply }` specifies the text being scored.

## `Turn` return type

`t.send(...)` returns a `TurnHandle`: convenience fields derived from the event stream, plus a full set of turn-scoped assertions.

#### `events`

```ts theme={null}
readonly events: StreamEvent[];
```

This turn's raw event stream (tool calls, message deltas, etc.); all the derived fields below are computed from it.

#### `toolCalls`

```ts theme={null}
readonly toolCalls: readonly ToolCall[];
```

List of tools called within this turn, derived from `events`.

#### `status`

```ts theme={null}
readonly status: "completed" | "failed" | "waiting";
```

This turn's end status: "completed" ended normally, "failed" errored, "waiting" stuck on an HITL input request.

#### `message`

```ts theme={null}
readonly message: string;
```

The assistant's final text reply for this turn (the concatenation of the message deltas in `events`).

#### `data`

```ts theme={null}
readonly data?: JsonValue;
```

Structured output attached by the adapter (if any), for `outputEquals` / `outputMatches` to compare against.

#### `usage`

```ts theme={null}
readonly usage?: Usage;
```

This turn's token usage and estimated cost (present only for agents that reported usage).

#### `outputEquals`

```ts theme={null}
outputEquals(value: unknown): AssertionHandle;
```

Asserts `data` deep-equals the given value.

#### `outputMatches`

```ts theme={null}
outputMatches(schema: unknown): AssertionHandle;
```

Asserts `data` satisfies the given schema (e.g. a zod schema).

#### `messageIncludes`

```ts theme={null}
messageIncludes(token: string | RegExp): AssertionHandle;
```

Asserts this turn's assistant reply contains `token` (limited to this turn's event stream, not across turns).

#### `succeeded`

```ts theme={null}
succeeded(): AssertionHandle;
```

Asserts this turn's status is "completed".

#### `parked`

```ts theme={null}
parked(): AssertionHandle;
```

Asserts this turn is stuck on an HITL input request (status is "waiting").

#### `calledTool`

```ts theme={null}
calledTool(name: string, match?: ToolMatch): AssertionHandle;
```

Asserts this turn called the named tool; `match` can further constrain arguments / count / status.

#### `notCalledTool`

```ts theme={null}
notCalledTool(name: string, match?: ToolMatch): AssertionHandle;
```

Asserts this turn did not call the named tool (or not under the `match` condition).

#### `toolOrder`

```ts theme={null}
toolOrder(names: string[]): AssertionHandle;
```

Asserts this turn's tool calls appear in the given order (other calls may be interleaved).

#### `usedNoTools`

```ts theme={null}
usedNoTools(): AssertionHandle;
```

Asserts this turn did not call any tool.

#### `maxToolCalls`

```ts theme={null}
maxToolCalls(max: number): AssertionHandle;
```

Asserts this turn's total tool call count does not exceed `max`.

#### `loadedSkill`

```ts theme={null}
loadedSkill(skill: string): AssertionHandle;
```

Asserts this turn loaded the named skill.

#### `noFailedActions`

```ts theme={null}
noFailedActions(): AssertionHandle;
```

Asserts this turn has no failed tool calls / commands.

#### `event`

```ts theme={null}
event(type: StreamEvent["type"], opts?: { count?: number }): AssertionHandle;
```

Asserts this turn saw an event of the given type; `opts.count` can constrain the occurrence count.

#### `notEvent`

```ts theme={null}
notEvent(type: StreamEvent["type"]): AssertionHandle;
```

Asserts this turn did not see an event of the given type.

#### `calledSubagent`

```ts theme={null}
calledSubagent(name: string, match?: SubagentMatch): AssertionHandle;
```

Asserts this turn called the named subagent; `match` can constrain count / status / remoteUrl.

#### `eventOrder`

```ts theme={null}
eventOrder(types: StreamEvent["type"][]): AssertionHandle;
```

Asserts this turn's events appear in the given type order (other events may be interleaved).

#### `eventsSatisfy`

```ts theme={null}
eventsSatisfy(label: string, predicate: (events: readonly StreamEvent[]) => boolean): AssertionHandle;
```

Asserts against this turn's whole event stream using a custom predicate; `label` is required and becomes the assertion's title (the predicate itself is opaque, so the burden of explanation falls on `label`).

#### `maxTokens`

```ts theme={null}
maxTokens(max: number): AssertionHandle;
```

Asserts this turn's token usage does not exceed `max`.

#### `maxCost`

```ts theme={null}
maxCost(usd: number): AssertionHandle;
```

Asserts this turn's spend (USD) does not exceed `usd`.

#### `judge`

```ts theme={null}
readonly judge: JudgeNamespace;
```

The judge namespace available for this turn (`t.judge.autoevals.*`).

## Dataset export

```ts theme={null}
export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
    },
  }),
);
```

Array exports generate stable IDs: `file/0000`, `file/0001`, and so on.

When you already have a stable business key, you can instead default-export a `Record<string, EvalDef>`. For example, the key `15193` in `swelancer.eval.ts` generates `swelancer/15193`. A key must be a non-empty single path segment: it cannot be `.` or `..`, and cannot contain `/`, `\`, or control characters; discovery order is fixed by lexicographic order over the keys.