Evals in NiceEval: lifecycle, Verdicts, and files

An Eval is a runnable test case. It is usually exported from a *.eval.ts file and declared with defineEval.

What an Eval contains

import { defineEval } from "niceeval";
import { includes, jsonMatch, toolMatch } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool(
      toolMatch("get_weather", { input: jsonMatch({ city: "Brooklyn" }) }).exactly(1),
    );
    t.check(t.reply, includes("sunny"));
  },
});

Core fields:

Field	Description
`description`	A human-readable description shown in reports
`sandbox`	An optional Sandbox layer owned by the Eval; omit it when the Experiment provides the template-bearing layer
`test(t)`	The interaction and assertion logic

An Eval does not declare which Agent to run. It stays agent-neutral by default, so the same Eval can run against different Agents under different Experiments. The Experiment chooses the Agent; the CLI does not override it ad hoc.

Do not write id or name by hand. NiceEval derives the ID from the file path.

Path is identity

evals/weather/brooklyn.eval.ts has the ID weather/brooklyn. Positional arguments after the Experiment name filter by ID prefix:

npx niceeval exp local weather
npx niceeval exp local weather/brooklyn

This keeps IDs stable and readable, and naturally aligned with the directory structure. Positional arguments use a string prefix: terminal-swe-bench also matches terminal-swe-bench-astropy-1; the following character does not have to be /.

Lifecycle

Discovery

The Runner loads *.eval.ts files and Fixture directories under evals/.

Scheduling

It creates an execution plan from concurrency, cache, attempts, and early exit.

agent.send

t.send() calls the selected Adapter and receives a standard Turn.

Assertions and Verdict

NiceEval collects value assertions, scoped assertions, and Judge measurements, then registers score contributions.

Final state

A pass-style Eval folds assertion results into its final Verdict; a score-style Eval accumulates its official score.

Report

The console and Reporters output results. Run facts and published Attempt facts are written to the one .niceeval/record.sqlite file.

Verdict types

A pass-style Eval Attempt has exactly four terminal states:

passed

Every Boolean Assertion matched, every gated measurement met the threshold defined by its Match, and no execution error occurred.

failed

At least one Boolean condition mismatched, or a gated measurement fell below the threshold defined by its Match.

errored

An execution exception, timeout, or authoring error means this execution cannot form a trustworthy conclusion.

skipped

The Eval actively skipped itself, usually with t.skip(reason).

A score-style Eval’s primary reading is its cumulative earned score; higher is better. A score of 0 is normal when there are no score items. The default report shows earned score and does not count normal closed states as pass rate. The points in .score(points) are the weight of one contribution, not a fixed total score. The default report does not automatically convert earned score into a percentage. Historical mixed Records created before homogeneous admission remain readable, with pass-style and score-style primary readings displayed separately.

Verdicts and measurements

Boolean matched enters the Verdict by default. Continuous measurements and Judge output are measurements in [0, 1]. Call .atLeast(n) on the ScoreMatch before registration to form a local condition, then call the registered handle’s parameterless .gate() to include that condition in the pass-style Verdict. See Assert for the complete rules.

The `*.eval.ts` convention

Only files ending in .eval.ts are discovered. Directories only form ID prefixes:

evals/
└─ billing/
   └─ refund.eval.ts  # id: billing/refund

Array exports and data-driven tests (Dataset fan-out)

A file can default-export an array of defineEval(...) calls to generate multiple cases from the same logic:

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);

This generates IDs such as sql/0000 and sql/0001. See Dataset fan-out.

Experiment — The other half: who to evaluate and how to run it, and why it is separate from the Eval (late binding).
Assert — Assertion and Verdict rules.

Core Model

Interaction & Assertions

Execution Model

Evals in NiceEval: lifecycle, Verdicts, and files

What an Eval contains

Path is identity

Lifecycle

Verdict types

passed

failed

errored

skipped

Verdicts and measurements

The `*.eval.ts` convention

Array exports and data-driven tests (Dataset fan-out)

​What an Eval contains

​Path is identity

​Lifecycle

​Verdict types

passed

failed

errored

skipped

​Verdicts and measurements

​The *.eval.ts convention

​Array exports and data-driven tests (Dataset fan-out)

​Related reading

What an Eval contains

Path is identity

Lifecycle

Verdict types

Verdicts and measurements

The `*.eval.ts` convention

Array exports and data-driven tests (Dataset fan-out)

Related reading