NiceEval Assertions and Matches

An Assertion is NiceEval’s unit of evaluation. check(subject, match) is the registration primitive: it reads the subject and registers one Assertion immediately. Scoped helpers such as calledTool(...) select a managed subject and Match, then use that same path. The returned handle only configures the registered entry.

import { includes, toolMatch } from "niceeval/expect";

t.check(turn.message, includes("Completed")).label("Reply says it completed");
turn.succeeded().label("Turn completed");
turn.calledTool(toolMatch("get_weather").exactly(1)).label("Looks up weather");
t.judge({ question, answer: turn.message }, answerQuality)
  .gate(0.8).label("Answer quality");

Each Assertion’s key and label can be configured at most once. Setting either more than once is an authoring error, even when setting it to the same value.

Matches compare values only

niceeval/expect exports pure Match factories. A Match is a reusable, immutable, deterministic, side-effect-free comparison rule. It has no identity, call site, threshold, score, or control flow. t.check(subject, match) strictly takes two arguments, reads the subject at the call site, and registers an Assertion.

import { equals, includes, matches, satisfies } from "niceeval/expect";

t.check(turn.message, includes("Order confirmed"));
t.check(turn.data, equals({ status: "refund", amount: 42 }));
t.check(turn.data, matches(OrderSchema));
t.check(turn.message.length, satisfies("Reply is not empty", (length) => length > 0));

Scoped Assertions

Scoped methods directly register a Boolean Assertion when called. The receiver determines the evidence scope: turn.* sees only that immutable Turn, session.* sees the prefix of that Session before the call, and root t.* sees the vector cut of all Sessions started at the call.

const turn = await t.send("Run checks and fix the problem.");
turn.succeeded().label("Turn completed");
turn.calledTool("shell").label("Uses shell");
turn.calledTool(toolMatch("read_file").exactly(2)).label("Reads twice");

Put occurrence quantifiers such as .exactly(2) or .atLeast(1) on toolMatch before passing it to calledTool. Put conditions on input, output, and status there too; give JSON structure to jsonMatch and command tokens to commandMatch. notCalledTool accepts an unquantified tool Match and selects the zero-occurrence condition.

Two kinds of Eval

defineEval creates a pass-style Eval. Boolean matched enters the Attempt Verdict; mismatched makes the final Verdict failed, but does not prevent later checks from registering. Continuous evaluators and Judges yield a [0,1] measurement. Call .gate(minimum) on the registered handle to include the measurement threshold in the Verdict.

t.judge({ task, answer: turn.message }, actionableQuality)
  .gate(0.8)
  .label("Actionable steps");

defineScoreEval creates a score-style Eval. It has an Attempt Verdict and a cumulative score, with no implicit total score, percentage, or other numeric unit. An Assertion saves its evaluation without scoring by default; .score(n) makes an existing Assertion contribute score. A Boolean match contributes n and a mismatch contributes 0; a measurement m contributes m * n. t.score(n) registers a contribution directly, and its returned handle can configure only key and label.

turn.calledTool("search");                         // Saves only the evaluation
turn.calledTool("search").score(2);                // +2 when it matches
turn.check(
  { task, answer: turn.message },
  completeness,
).score(5);                                         // +4 when measurement is 0.8
t.score(5);                                        // Directly +5

A score-style Eval measurement can close without a threshold. Call .gate(minimum) on its handle to add a quality gate without changing the contribution; the same entry can have both a gate and a score. A low measurement alone does not fail, but an unmet gate makes the Verdict failed and keeps the earned score. An Assertion without .score() does not display +0. A score-style Eval with no score items normally receives an official, rankable score: 0.

Control flow

.orStop() is an async barrier on the same Assertion handle and must be awaited. A Boolean mismatch, or a thresholded measurement below its threshold, sets an authoring stop latch and rejects a private control signal. It stops only the continuation currently awaiting it. It does not undo ordinary JavaScript side effects or cancel concurrent work that has already started; source that has not executed produces no result.

const turn = await t.send("Search for information and explain the conclusion.");
await turn.succeeded().orStop();
const search = await t.check(turn.message, includes("conclusion")).orStop();

After a normal stop, a pass-style Eval still receives a failed Verdict from the Assertion that triggered it. A score-style Eval remains scored, preserving its official score and stop cause, so it can be ranked.

Judge

defineJudge creates a managed JudgeDefinition. judge(material, definition) is a thin helper that only accepts a Judge definition; check(material, definition) uses the same dispatcher. Both pass-style and score-style Evals can call .gate(minimum), and score-style Evals can also call .score(n). A Judge runs once per entry. See Judge for material, configuration, and failure semantics.

Unavailable results and reading them

Missing evidence must not masquerade as an ordinary mismatch. unavailable and errored preserve their reasons and redacted evidence. AssertionResult does not save only a pass or failure. Every entry follows the same check(a, b) model: each result saves safe structured content or a stable reference for subject a, the evaluator / Match b identity and complete safe configuration, and the structured result needed for evaluation, policy, and display. Explicit t.check(a, b) lets the author supply both. calledTool and other scoped helpers obtain a and construct or select b for the author. Judge factories supply only b; authors pass explicit { input, output } material as a. For example, t.check(await t.sandbox.runCommand(...), commandSucceeded()) retains the evaluated CommandResult command, arguments, exit status, duration, and redacted stdout / stderr content or references. calledTool(...) retains normalized tool-occurrence context in its scope. That context includes operation / event identity, redacted input, status, output / error references, coverage, and matching event references. When nothing matches, it still retains observation scope and candidate occurrence references rather than only false. Fixed query operations, View, and source results can therefore organize copy and interfaces from the same structured AssertionResult without rerunning a Match or calling Judge. expected / received are reading-surface text generated from a, b, and the evaluation; they are not the only two stored strings. Assertion decides what must be retained, while Record decides how it is persisted. If subject data needed to preserve an evaluator decision safely cannot be retained, the result must be unavailable; it cannot retain only matched or mismatched.

Match Reference — Built-in Matches, composition, and custom Matches.
Eval API — The complete field set for t, Session, Turn, and handles.
Pass-style and score-style Evals — The complete form of both kinds of Eval.
Judge — Judge and model configuration.

Core Model

Interaction & Assertions

Execution Model

NiceEval Assertions and Matches

Matches compare values only

Scoped Assertions

Two kinds of Eval

Control flow

Judge

Unavailable results and reading them

​Matches compare values only

​Scoped Assertions

​Two kinds of Eval

​Control flow

​Judge

​Unavailable results and reading them

​Related reading

Matches compare values only

Scoped Assertions

Two kinds of Eval

Control flow

Judge

Unavailable results and reading them

Related reading