NiceEval Assertions and Matches

NiceEval evaluates by Assertion, and one Assertion is one check. When you call t.check(value, match), t.judge(value, definition), turn.succeeded(), or calledTool(...), NiceEval registers one Assertion immediately. There is no step where you first create a check object and then hand it to another API to run. The handle returned by the call only configures the Assertion just registered, for example to add a label, set a gate, or score it.

import { includes } from "niceeval/expect";

t.check(turn.reply, includes("Completed")).label("Reply says it completed");
turn.succeeded().label("Turn completed");
turn.calledTool("get_weather").label("Looks up weather");
t.judge({ question, answer: turn.reply }, answerQuality)
  .gate(0.8).label("Answer quality");

Each Assertion’s key and label can each be set at most once. Setting either again is an error, even when both writes use the same value.

Matches only compare

Everything niceeval/expect exports is a Match, and a Match is one comparison rule. It is reusable and never changes; the same input always gets the same result, and it has no side effects. A Match itself carries no gate, score, or control flow. Those are all configured on the handle. t.check(value, match) takes only these two arguments. It reads value at the call and registers the Assertion.

import { equals, includes, matches, satisfies, toolMatch } from "niceeval/expect";

t.check(turn.reply, includes("Order confirmed"));
t.check(turn.data, equals({ status: "refund", amount: 42 }));
t.check(turn.data, matches(OrderSchema));
t.check(turn.reply.length, satisfies("Reply is not empty", (length) => length > 0));

Common Match Reference

Choose a Match based on your assertion target:

Target	Recommended Approach	Note
Specific substring	`includes("text")`	Deterministic substring check
RegEx or wording variations	`pattern(/regex/)`	Tolerates minor formatting variations
Deep equality (numbers, enums, objects)	`equals(expected)`	Exact deep equality
Object conforms to Schema	`matches(Schema)`	Type/schema validation
Tool calls and arguments	`calledTool(toolMatch(name, { input }))`	Process assertion
Subjective quality or semantic grounding	`t.judge(material, definition)`	LLM-as-a-judge model
Stepwise progress or scores	`measured(...)` / `.score(n)`	Score contributions

Whether the result is matched or mismatched only says whether the value satisfies this rule. It does not say whether the rule itself is written correctly. When a task allows several correct results, the rule must cover all of them. equals(expected) does a deep equality comparison; for strings, it requires an exact match. If the task accepts several equivalent forms, first parse or normalize the output into the value the task cares about, then compare. Compare strings directly only when the literal form is itself the task requirement.

Check on a Turn, a Session, or the whole eval

Methods such as succeeded and calledTool register one Boolean Assertion when called. The object you call them on decides which evidence they see:

turn.* sees only that one Turn.
session.* sees everything in that Session up to the call.
t.* sees every Session that has started at the time of the call.

const turn = await t.send("Run checks and fix the problem.");
turn.succeeded().label("Turn succeeded");
turn.calledTool("shell").label("Uses shell");
turn.calledTool(toolMatch("read_file").exactly(2)).label("Reads twice");

calledTool does not take a count argument. Put the count on toolMatch, for example .exactly(2). Conditions on input, output, and status also go in toolMatch: use jsonMatch for JSON structure and commandMatch for command arguments. notCalledTool accepts the same matcher, but without a count.

Pass/fail and scored

defineEval is pass/fail. Boolean checks directly affect the Attempt’s verdict: one mismatched makes the final verdict failed, but later checks still run and register. Continuous-value checks and Judges give measurements between 0 and 1. By default, measurements do not affect the verdict. After you call .gate(minimum) on the handle, falling below the minimum gives failed.

t.judge({ task, answer: turn.reply }, actionableQuality)
  .gate(0.8).label("Actionable steps");

defineScoreEval is scored. It has both a verdict and an accumulated score. The score is the sum of each item’s contribution, with no implied total or percentage. By default, an Assertion only records its result and does not score. Adding .score(n) makes it contribute. A satisfied Boolean adds n and an unsatisfied one adds 0; a measurement m contributes m * n. t.score(n) adds points directly, and its returned handle can only set key and label.

turn.calledTool("search");                       // Saves only the evaluation
turn.calledTool("search").score(2);              // +2 when matched
t.check({ task, answer: turn.reply }, completeness).score(5); // +4 when the measurement is 0.8
t.score(5);                                      // Directly +5

In a scored eval, a measurement can score without a gate. When you need a quality gate, call .gate(minimum). It does not affect that item’s score, so the same Assertion can have both a gate and a score. A low score alone is not a failure. Only when a gate is set and not met is the verdict failed, and the score already earned is kept. A check without .score() does not show +0. A scored eval with no scoring items normally gets score: 0.

Stop later steps

.orStop() stops later steps when the check is not satisfied. It returns a Promise and must be awaited:

A Boolean check uses its own condition.
For a measurement with no gate yet, use .orStop(minimum) to set the minimum for stopping.
If .gate(minimum) is already written, use .orStop() with no argument to reuse the same minimum.

const turn = await t.send("Search for information and explain the conclusion.");
await turn.succeeded().orStop();
const search = await t.check(turn.reply, includes("conclusion")).orStop();

Stopping only affects code after the await that has not run yet, and that code produces no results. Ordinary JavaScript side effects that already happened are not undone, and concurrent tasks started earlier are not cancelled. The verdict after a stop depends on the reason:

Stopping because a gate was not met gives failed.
A measurement that uses only .orStop(minimum) without a gate and falls below the minimum only stops later steps. It does not give failed for that reason.
Stopping because of unavailable or errored gives errored.

After the first stop, this Assertion’s configuration can no longer change. Awaiting .orStop() with no argument again returns the same Promise.

Judge

defineJudge defines a Judge. t.judge(material, definition) only accepts a Judge definition and is shorthand for t.check(material, definition); both take the same execution path. Both pass/fail and scored evals can use .gate(minimum), and scored evals can also use .score(n). One Assertion calls the Judge only once. For material, configuration, and failure behavior, see Judge.

Missing evidence and saved results

Missing evidence must not be treated as an ordinary failure to match. When the result is unavailable or errored, the reason and redacted evidence are kept. Each Assertion’s result is also more than “passed” or “did not pass”. It saves three kinds of content:

The value being checked: safe structured content, or a stable reference to it.
Which rule was used: the identity and complete configuration of the Match or evaluation function.
The evaluation result, plus the information needed for gates, scoring, and display.

With t.check(a, b) and t.judge(material, definition), you provide the value being checked and the rule. Methods such as calledTool and succeeded get the value and build the Match for you, and save the same things. Two examples:

t.check(await t.sandbox.runCommand(...), commandSucceeded()) saves the command name, arguments, exit status, and duration of that command, plus redacted stdout and stderr (content or references).
calledTool(...) saves every tool call it saw: the call identity, redacted input, status, references to output or errors, whether the record is complete, and which events matched. When nothing matched, it still saves the scope it looked at and the candidate calls, not just false.

So query, view, and the source view can all build text and interfaces directly from the saved result, without rerunning the Match or calling the Judge again. The expected and received shown in interfaces are generated at read time from this content; they are not just two stored strings. Assertion only defines what must be saved. Record decides how it is written to disk. When the data needed for a decision cannot be saved safely, the result must be unavailable; it cannot keep only matched or mismatched.

Match Reference — Built-in Matches, composition, and custom Matches.
Eval API — The complete field set for t, Session, Turn, and handles.
Pass/fail and scored evals — The complete form of both kinds of eval.
Judge — Judge and model configuration.

Core Model

Interaction and Scoring

Execution Model

NiceEval Assertions and Matches

Matches only compare

Common Match Reference

Check on a Turn, a Session, or the whole eval

Pass/fail and scored

Stop later steps

Judge

Missing evidence and saved results

Matches only compare

Common Match Reference

Check on a Turn, a Session, or the whole eval

Pass/fail and scored

Stop later steps

Judge

Missing evidence and saved results

Related reading