Eval cases in NiceEval: lifecycle, verdicts, and files

An eval case says what to send and what counts as correct. It is usually exported from a *.eval.ts file and declared with defineEval. It reads like a unit test, but the output of the system under test is nondeterministic. The same eval case runs several times according to the Experiment’s attempts. Each run is one Attempt, and together they give a pass rate or a score.

What an eval case contains

import { defineEval } from "niceeval";
import { includes, jsonMatch, toolMatch } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool(
      toolMatch("get_weather", { input: jsonMatch({ city: "Brooklyn" }) }).exactly(1),
    );
    t.check(t.reply, includes("sunny"));
  },
});

Core fields:

Field	Description
`description`	A human-readable description shown in reports
`sandbox`	Optional. This eval case’s own Sandbox configuration; omit it when the Experiment already names the prebuilt environment
`test(t)`	The interaction and assertion logic

An eval case does not declare which Agent to use, so the same eval case can run against different Agents under different Experiments. The Agent is an Experiment field, and the CLI does not override it ad hoc.

Do not write id or name by hand. NiceEval derives the ID from the file path.

Path is identity

evals/weather/brooklyn.eval.ts has the ID weather/brooklyn. Positional arguments after the Experiment name filter by ID prefix:

npx niceeval exp local weather
npx niceeval exp local weather/brooklyn

IDs follow the directory structure, so they are stable and readable. Filtering matches by string prefix: terminal-swe-bench also matches terminal-swe-bench-astropy-1, and the next character does not have to be /.

Lifecycle

Discover

The Runner loads *.eval.ts files and Fixture directories under evals/.

Schedule

It combines concurrency, result reuse, attempts, and earlyExit to produce the Attempts for this run.

Call the system under test

t.send() calls the Adapter selected by the Experiment and receives a standard Turn.

Check

NiceEval records every check: value assertions, assertions on the Turn and Session, Judge scores, and each item’s score contribution.

Judge the outcome

A pass/fail eval combines the check results into a final verdict; a scored eval accumulates a score.

Save and report

The terminal and Reporters output results; the Run and published Attempts are written to .niceeval/record.sqlite.

Four verdicts

Each Attempt of a pass/fail eval gets exactly one of four verdicts:

passed

Every required Boolean check is satisfied, every measurement with .gate(minimum) reaches its minimum, and no execution error occurred.

failed

At least one required Boolean check is not satisfied, or a measurement with .gate(minimum) falls below its minimum.

errored

An execution exception, a timeout, or a mistake in the eval code means this run cannot reach a trustworthy conclusion.

skipped

The eval case skipped itself, usually with t.skip(reason).

A scored eval is read mainly by its accumulated score, and higher is better. With no scoring items, the score is 0. Reports show the score by default and do not count it toward the pass rate, but the verdict is still kept. A result with a failed verdict does not enter the success ranking even if its score was fully computed. The points in .score(points) is the weight of that item, not the total score. Reports do not convert scores to percentages automatically. When one Experiment contains both kinds of evals, the report shows the pass rate and the score separately.

Verdicts and measurements

In pass/fail evals, Boolean checks affect the verdict by default. In scored evals, Boolean checks are only recorded by default. Continuous measurements and Judge scores are numbers between 0 and 1. Both kinds of evals use .gate(minimum) to set a minimum, and falling below it gives failed. In a scored eval, the score already earned is kept when a gate is not met. See Assertion and Match for the complete rules.

The `*.eval.ts` convention

Only files ending in .eval.ts are discovered. Directories only form ID prefixes:

evals/
└─ billing/
   └─ refund.eval.ts  # id: billing/refund

Array exports and data-driven tests (dataset fan-out)

A file can also default-export an array of defineEval(...) calls, generating multiple eval cases from the same logic:

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

This generates IDs such as sql/0000 and sql/0001. See Data-driven tests.

Experiment — The other half: what to evaluate and how to run it, and why it is separate from eval cases.
Assert — Assertion and verdict rules.

Core Model

Interaction and Scoring

Execution Model

Eval cases in NiceEval: lifecycle, verdicts, and files

What an eval case contains

Path is identity

Lifecycle

Four verdicts

passed

failed

errored

skipped

Verdicts and measurements

The `*.eval.ts` convention

Array exports and data-driven tests (dataset fan-out)

What an eval case contains

Path is identity

Lifecycle

Four verdicts

passed

failed

errored

skipped

Verdicts and measurements

The *.eval.ts convention

Array exports and data-driven tests (dataset fan-out)

Related reading

The `*.eval.ts` convention