NiceEval assertions: values, scoped facts, tests, and cost

Assertion is the process of taking everything an agent did during an eval — every message, tool call, file change, and token spent — and folding it into a single, explainable result. NiceEval gives you four assertion mechanisms that complement each other: some check values immediately, some assess the whole run after it completes, some execute tests inside the Sandbox, and some measure efficiency. All four produce the same Assertion type and feed into the same verdict rules. For the fifth mechanism — asking a language model to judge open-ended quality — see Judge.

The four assertion mechanisms

1. Value assertions

t.check(value, matcher) and t.require(value, matcher) evaluate a specific value immediately against a matcher from niceeval/expect. Use these for facts you can verify inline.

2. Scoped assertions

t.succeeded(), t.calledTool(), t.messageIncludes(), and friends are registered during test(t) but evaluated after the function returns, against the complete turn data. Use these for whole-run facts.

3. Project-test assertions

For Sandbox evals, run project tests, build scripts, or focused probe commands from test(t). Use this for coding tasks where file content and build results are the ground truth.

4. Efficiency assertions

t.maxTokens() and t.maxCost() turn token usage and estimated cost into scoreable dimensions. An agent that answers correctly but burns ten times the expected tokens should not score the same as one that answers efficiently.

Gate vs soft severity

Every assertion carries a severity that determines how it influences the final verdict. There are exactly two severities:

gate
soft

A gate assertion is a hard requirement. If it fails, the entire eval is immediately classified as failed — regardless of how well every other assertion passed. Use gate for facts that must be true: “the agent called the correct tool”, “the response parsed as valid JSON”, “no shell commands errored.”Most matchers in niceeval/expect (includes, equals, matches, satisfies) default to gate. Scoped assertions like t.succeeded() and t.calledTool() also default to gate.

A soft assertion is a quality score with a numeric threshold. If the score falls below the threshold, the eval becomes passed rather than failed — a signal that there is a quality regression, but not a hard breakage. Soft failures only count as failures when you run with --strict.Use soft for continuous judgments where the answer is “how good” rather than “correct or not”: similarity measurement, LLM-as-judge factuality ratings, cost budgets you want to track without blocking CI.Matchers that produce a continuous score (similarity) and all judge calls default to soft.

You can override the default severity with a chain method on any matcher or assertion:

t.check(t.reply, includes("confirmed"));          // gate (default)
t.check(t.reply, similarity(expected).gate());    // promote to gate
t.maxTokens(80_000).atLeast(0.7);                 // demote to soft with a threshold

Verdict rules

Once all assertions are collected, the runner takes the first matching rule in this fixed order and folds them into a single result:

Execution error, timeout, or author mistake                        → errored
Any gate assertion failed, or a soft is below threshold --strict   → failed
t.skip(reason) was called explicitly                                → skipped
Otherwise                                                            → passed

errored outranks everything else, because the execution evidence can no longer be trusted. failed outranks skipped, so that a t.skip() cannot mask a hard failure recorded earlier.

passed

No errors, all gate assertions passed, all soft assertions met their thresholds (or you did not run with --strict).

failed

At least one gate assertion did not pass, or a soft assertion fell below its threshold under --strict. Hard failure.

errored

An execution error, timeout, or author mistake — this run cannot support a trustworthy conclusion, and it is not disguised as an assertion failure.

skipped

t.skip("reason") was called. Excluded from pass-rate calculations entirely.

When you run an eval more than once (attempts > 1), the per-eval summary becomes a pass rate (the fraction of attempts that produced passed) and an average latency, rather than a single verdict.

1. Value assertions — `niceeval/expect` matchers

t.check(value, assertion) evaluates the assertion immediately and records the result. t.require(value, assertion) does the same but throws immediately if the assertion fails, aborting the rest of the test function. Use t.require for preconditions: if a required fact is false, there is no point continuing. The matchers available from niceeval/expect:

import {
  includes,    // substring or regex match          (default: gate)
  equals,      // deep equality                     (default: gate)
  matches,     // Standard Schema (Zod etc.) check  (default: gate)
  similarity,  // normalized Levenshtein 0–1        (default: soft)
  satisfies,   // custom predicate + label          (default: gate)
} from "niceeval/expect";

Usage examples:

// Check that the agent's reply contains a specific string
t.check(t.reply, includes("order confirmed"));

// Deep-equal check on structured output
t.check(turn.data, equals({ status: "refund", amount: 42 }));

// Validate structured output against a Zod schema
t.check(turn.data, matches(z.object({ intent: z.enum(["refund", "ship"]) })));

// Similarity with an explicit threshold
t.check(t.reply, similarity("expected answer").atLeast(0.8));

// Custom predicate
t.check(turn.data, satisfies((d) => d.total > 0, "total is positive"));

Matchers are pure functions — (value) => number — so you can write your own and pass them to t.check without any special registration.

2. Scoped assertions

Scoped assertions are registered during test(t) but evaluated after the function returns, against the complete accumulated turn data. They read from the standard event stream that t.send() produces (see Drive) and its derived facts — so as long as your adapter produces correct events, these assertions work identically for every agent.

Scoped assertions only appear on t if the agent has declared the corresponding capability. Calling t.calledTool() when the agent has not declared toolObservability: true is a compile error.

Run / session dimension

await t.succeeded().stopOnFailure(); // run completed; stop dependent checks if it did not
t.parked();                     // cleanly stopped on a HITL input.requested event
t.messageIncludes("Regards,");  // all message events concatenated contain this string/regex

Tool / action dimension

t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });
t.notCalledTool("shell", { input: { command: /npm i/ } });
t.toolOrder(["read_file", "write_file"]);   // relative order of tool calls
t.usedNoTools();
t.maxToolCalls(5);
t.loadedSkill("memory-v2");                // sugar for calledTool("load_skill", ...)
t.calledSubagent("researcher", { remoteUrl: /api\.example/ });
t.noFailedActions();                       // no tool, subagent, or skill has status "failed"

The input argument to calledTool and notCalledTool supports a small matching language: a plain object performs deep partial matching, a RegExp matches against the serialized input, and a predicate function receives the raw input value.

Event stream dimension (low-level escape hatch)

t.event("input.requested", { count: 1 });
t.notEvent("error");
t.eventOrder(["operation.started", "operation.finished"]);
t.eventsSatisfy("read before write", (events) => /* custom predicate */ true);

All scoped assertions above are syntactic sugar for these low-level event stream queries. When none of the higher-level assertions fit your use case, you can drop down to eventsSatisfy and write an arbitrary predicate over the raw StreamEvent[].

Structured output (on `turn`, not `t`)

const turn = await t.send("Return the result as JSON");
turn.outputEquals({ status: "ok" });                         // deep equality on turn.data
turn.outputMatches(z.object({ status: z.string() }));        // Standard Schema validation

Workspace dimension (Sandbox agents only)

t.sandbox.fileChanged("src/Button.tsx");
t.sandbox.fileDeleted("src/old.ts");
t.sandbox.noChanges();                         // no repository files were modified this turn
t.sandbox.notInDiff(/sk-[A-Za-z0-9]/);         // attributed diff contains no secrets / inline styles
t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());         // npm test exited 0
t.check(await t.sandbox.runCommand("npm", ["run", "build"]), commandSucceeded()); // npm run build exited 0
t.sandbox.noFailedShellCommands();

These declarations are evaluated against the agent-attributed diff after the run: fileChanged("src/Button.tsx") matches when the Agent modified that path, fileDeleted("src/old.ts") matches when the Agent deleted it, and changedPaths([...]) matches the exact unordered set of changed paths. fileChanged(path, { before, after, status }) also matches content: before and after are read from the same change’s endpoints. To read a file’s current content — for example, to hand it to a Judge as explicit { input, output } material — use await t.sandbox.readText(path). Attribution is decided by fileChanged, not by what you read. Scoped assertions follow one rule everywhere: the receiver decides the scope, not the assertion name. t.* aggregates every turn of the whole eval run (including any t.newSession() sessions); session.* (from t.newSession()) scopes to that one session; turn.* (from t.send()’s return value) scopes to that single turn only. Same vocabulary, different receiver — see Drive for what each receiver is.

3. Project-test assertions (Sandbox evals)

For Sandbox coding evals, run validation commands inside test(t) and record their result as assertions.

import { commandSucceeded, includes } from "niceeval/expect";

const testResult = await t.sandbox.runCommand("npm", ["test"]);
t.check(testResult, commandSucceeded());

const src = await t.sandbox.runShell("find . -name '*.ts' -exec cat {} +");
t.check(src.stdout, includes(/z\.object\s*\(/));

You can also assert behavior through the standard event stream: t.calledTool(...), t.sandbox.noFailedShellCommands(), t.eventsSatisfy(...), and diff assertions such as t.sandbox.fileChanged(...).

4. Efficiency / cost assertions

Token usage is a first-class evaluation dimension. An agent that answers correctly but burns far more tokens than expected should not be treated identically to one that answers efficiently.

t.maxTokens(50_000);            // hard token limit for the entire run (gate by default)
t.maxCost(0.5);                 // estimated cost cap in USD (requires a price table in config)
t.maxTokens(80_000).atLeast(0.7);     // soft variant — tracked but only red under --strict
t.check(t.usage.outputTokens, satisfies((n) => n < 10_000, "not verbose"));

t.usage is available anywhere inside test(t) and exposes { inputTokens, outputTokens, cacheReadTokens?, … }. For Sandbox agents, token counts are extracted from the transcript by the adapter; for Direct Agents, they are returned in Turn.usage.

Custom scorers

A value assertion is just a function (value) => number | Promise<number>. You can write custom matchers using makeAssertion:

import { makeAssertion } from "niceeval/expect";
import type { Assertion } from "niceeval/expect";

function jsonValid(): Assertion {
  return makeAssertion({
    name: "jsonValid",
    severity: "gate",
    score: (value) => {
      try { JSON.parse(String(value)); return 1; }
      catch { return 0; }
    },
  });
}

t.check(t.reply, jsonValid());

Custom matchers compose with the same chain methods as built-ins: .gate(), .atLeast(0.7).

Drive — t.send(), t.newSession(), and HITL: how you produce the Turn data these assertions read from.
Judge — the fifth assertion mechanism, for open-ended quality that can’t be expressed as a fixed rule.
Write send — how the standard event stream is produced, and what scoped assertions depend on.
Evals — how assertions fold into the eval lifecycle and verdict types.

Core Model

Interaction & Assertions

Execution Model

NiceEval assertions: values, scoped facts, tests, and cost

The four assertion mechanisms

1. Value assertions

2. Scoped assertions

3. Project-test assertions

4. Efficiency assertions

Gate vs soft severity

Verdict rules

passed

failed

errored

skipped

1. Value assertions — `niceeval/expect` matchers

2. Scoped assertions

Run / session dimension

Tool / action dimension

Event stream dimension (low-level escape hatch)

Structured output (on `turn`, not `t`)

Workspace dimension (Sandbox agents only)

3. Project-test assertions (Sandbox evals)

4. Efficiency / cost assertions

Custom scorers

​The four assertion mechanisms

1. Value assertions

2. Scoped assertions

3. Project-test assertions

4. Efficiency assertions

​Gate vs soft severity

​Verdict rules

passed

failed

errored

skipped

​1. Value assertions — niceeval/expect matchers

​2. Scoped assertions

​Run / session dimension

​Tool / action dimension

​Event stream dimension (low-level escape hatch)

​Structured output (on turn, not t)

​Workspace dimension (Sandbox agents only)

​3. Project-test assertions (Sandbox evals)

​4. Efficiency / cost assertions

​Custom scorers

​Related reading

The four assertion mechanisms

Gate vs soft severity

Verdict rules

1. Value assertions — `niceeval/expect` matchers

2. Scoped assertions

Run / session dimension

Tool / action dimension

Event stream dimension (low-level escape hatch)

Structured output (on `turn`, not `t`)

Workspace dimension (Sandbox agents only)

3. Project-test assertions (Sandbox evals)

4. Efficiency / cost assertions

Custom scorers

Related reading