Evals in NiceEval: lifecycle, verdicts, and files

An eval is a runnable test case. It usually lives in a *.eval.ts file and is declared with defineEval.

What an eval contains

import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
    t.check(t.reply, includes("sunny"));
  },
});

Core fields:

Field	Meaning
`description`	Human-readable description shown in reports
`environment`	Optional environment requirement profile, mapped by the Sandbox spec’s `environments` table to a concrete preset environment
`test(t)`	Interaction and assertion logic

An eval does not declare which Agent it runs against — it stays agent-neutral by default, so the same eval can run against different Agents under different experiments. Choosing the Agent is a field of the experiment, not something the CLI overrides ad hoc.

Do not hand-write id or name. NiceEval derives the eval ID from the file path.

Path is identity

evals/weather/brooklyn.eval.ts has ID weather/brooklyn. Positional arguments after the experiment name filter by ID prefix:

npx niceeval exp local weather
npx niceeval exp local weather/brooklyn

This keeps IDs stable and readable, and naturally consistent with the directory structure. Positional arguments use a bare string-prefix match: terminal-swe-bench also matches terminal-swe-bench-astropy-1 — the character right after the match is not required to be /.

Lifecycle

Discovery

The runner loads the *.eval.ts files and fixture directories under evals/.

Scheduling

Build the execution plan from concurrency, cache, attempts, and early exit.

agent.send

t.send() calls the selected Adapter and receives a standard Turn.

Evaluate assertions

NiceEval collects value assertions, scoped assertions, judge scores, and test results.

Verdict

All assertion results fold into one final verdict.

Report

The console and reporters emit the results, and facts are committed to the .niceeval/ Record root.

Verdict types

passed

All gate assertions passed (under --strict, soft assertions also met their thresholds), and no execution error occurred.

failed

At least one gate assertion failed, or under --strict a soft assertion came in below its threshold.

errored

An execution error, timeout, or authoring error — this run cannot produce a trustworthy conclusion.

skipped

The eval skipped itself, usually through t.skip(reason).

Gate and soft

gate is a hard threshold. Failure makes the eval fail. soft contributes a score without necessarily failing the eval. See Assert for the full rules.

The `*.eval.ts` convention

Only files ending in .eval.ts are discovered. Use directories for grouping:

evals/
└─ billing/
   └─ refund.eval.ts  # id: billing/refund

Array exports and data-driven tests (dataset fan-out)

A file can also default-export an array of defineEval(...) calls to generate multiple cases from the same logic:

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

This generates IDs such as sql/0000 and sql/0001. See Dataset fan-out.

Experiment — The other half: who to evaluate and how to run it, and why it is kept separate from the eval (late binding).
Assert — The full verdict rules for gate and soft assertions.

Core Model

Interaction & Assertions

Execution Model

Evals in NiceEval: lifecycle, verdicts, and files

What an eval contains

Path is identity

Lifecycle

Verdict types

passed

failed

errored

skipped

Gate and soft

The `*.eval.ts` convention

Array exports and data-driven tests (dataset fan-out)

​What an eval contains

​Path is identity

​Lifecycle

​Verdict types

passed

failed

errored

skipped

​Gate and soft

​The *.eval.ts convention

​Array exports and data-driven tests (dataset fan-out)

​Related reading

What an eval contains

Path is identity

Lifecycle

Verdict types

Gate and soft

The `*.eval.ts` convention

Array exports and data-driven tests (dataset fan-out)

Related reading