Evals in NiceEval: lifecycle, Verdicts, and files

An Eval is a runnable test case. It is usually exported from a *.eval.ts file and declared with defineEval.

What an Eval contains

import { defineEval } from "niceeval";
import { includes, jsonMatch, toolMatch } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool(
      toolMatch("get_weather", { input: jsonMatch({ city: "Brooklyn" }) }).exactly(1),
    );
    t.check(t.reply, includes("sunny"));
  },
});

Core fields:

Field	Description
`description`	A human-readable description shown in reports
`sandbox`	An optional Sandbox layer owned by the Eval; omit it when the Experiment provides the template-bearing layer
`test(t)`	The interaction and assertion logic

An Eval does not declare which Agent to run. It stays agent-neutral by default, so the same Eval can run against different Agents under different Experiments. The Experiment chooses the Agent; the CLI does not override it ad hoc.

Do not write id or name by hand. NiceEval derives the ID from the file path.

Path is identity

evals/weather/brooklyn.eval.ts has the ID weather/brooklyn. Positional arguments after the Experiment name filter by ID prefix:

npx niceeval exp local weather
npx niceeval exp local weather/brooklyn

This keeps IDs stable and readable, and naturally aligned with the directory structure. Positional arguments use a string prefix: terminal-swe-bench also matches terminal-swe-bench-astropy-1; the following character does not have to be /.

Lifecycle

Discovery

The Runner loads *.eval.ts files and Fixture directories under evals/.

Scheduling

It creates an execution plan from concurrency, cache, attempts, and early exit.

agent.send

t.send() calls the selected Adapter and receives a standard Turn.

Assertions and Verdict

NiceEval collects value assertions, scoped assertions, and Judge measurements, then registers score contributions.

Final state

A pass-style Eval folds assertion results into its final Verdict; a score-style Eval accumulates its official score.

Report

The console and Reporters output results. Run facts and published Attempt facts are written to the one .niceeval/record.sqlite file.

Verdict types

A pass-style Eval Attempt has exactly four terminal states:

passed

Every required Boolean Assertion matched, every measurement with .gate(minimum) met its threshold, and no execution error occurred.

failed

At least one required Boolean condition mismatched, or a measurement with .gate(minimum) fell below its threshold.

errored

An execution exception, timeout, or authoring error means this execution cannot form a trustworthy conclusion.

skipped

The Eval actively skipped itself, usually with t.skip(reason).

A score-style Eval’s primary reading is its cumulative earned score; higher is better. A score of 0 is normal when there are no score items. The default report shows earned score and does not count normal closed states as pass rate, while retaining the Verdict. A failed + complete result cannot enter successful rankings. The points in .score(points) are the weight of one contribution, not a fixed total score. The default report does not automatically convert earned score into a percentage. Historical mixed Records created before homogeneous admission remain readable, with pass-style and score-style primary readings displayed separately.

Verdicts and measurements

Boolean matched enters the Verdict by default in a pass-style Eval and is record-only by default in a score-style Eval. Continuous measurements and Judge output are measurements in [0, 1]. Both Eval kinds use .gate(minimum) on the handle to include a threshold in the Verdict. A failed gate keeps the earned score in a score-style Eval. See Assert for the complete rules.

The `*.eval.ts` convention

Only files ending in .eval.ts are discovered. Directories only form ID prefixes:

evals/
└─ billing/
   └─ refund.eval.ts  # id: billing/refund

Array exports and data-driven tests (Dataset fan-out)

A file can default-export an array of defineEval(...) calls to generate multiple cases from the same logic:

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

This generates IDs such as sql/0000 and sql/0001. See Dataset fan-out.

Experiment — The other half: who to evaluate and how to run it, and why it is separate from the Eval (late binding).
Assert — Assertion and Verdict rules.

Core Model

Interaction & Assertions

Execution Model

Evals in NiceEval: lifecycle, Verdicts, and files

What an Eval contains

Path is identity

Lifecycle

Verdict types

passed

failed

errored

skipped

Verdicts and measurements

The `*.eval.ts` convention

Array exports and data-driven tests (Dataset fan-out)

​What an Eval contains

​Path is identity

​Lifecycle

​Verdict types

passed

failed

errored

skipped

​Verdicts and measurements

​The *.eval.ts convention

​Array exports and data-driven tests (Dataset fan-out)

​Related reading

What an Eval contains

Path is identity

Lifecycle

Verdict types

Verdicts and measurements

The `*.eval.ts` convention

Array exports and data-driven tests (Dataset fan-out)

Related reading