> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# NiceEval assertions: values, scoped facts, tests, and cost
> NiceEval's assertion vocabulary — value assertions, scoped assertions, project-test assertions, and efficiency checks — plus gate vs soft severity and the rules that fold them into a verdict.
Assertion is the process of taking everything an agent did during an eval — every message, tool call, file change, and token spent — and folding it into a single, explainable result. [NiceEval](https://niceeval.com/) gives you four assertion mechanisms that complement each other: some check values immediately, some assess the whole run after it completes, some execute tests inside the Sandbox, and some measure efficiency. All four produce the same `Assertion` type and feed into the same verdict rules. For the fifth mechanism — asking a language model to judge open-ended quality — see [Judge](/docs/explanation/judge).
## The four assertion mechanisms
`t.check(value, matcher)` and `t.require(value, matcher)` evaluate a specific value immediately against a matcher from `niceeval/expect`. Use these for facts you can verify inline.
`t.succeeded()`, `t.calledTool()`, `t.messageIncludes()`, and friends are registered during `test(t)` but evaluated **after** the function returns, against the complete turn data. Use these for whole-run facts.
For Sandbox evals, run project tests, build scripts, or focused probe commands from `test(t)`. Use this for coding tasks where file content and build results are the ground truth.
`t.maxTokens()` and `t.maxCost()` turn token usage and estimated cost into scoreable dimensions. An agent that answers correctly but burns ten times the expected tokens should not score the same as one that answers efficiently.
## Gate vs soft severity
Every assertion carries a **severity** that determines how it influences the final verdict. There are exactly two severities:
A gate assertion is a hard requirement. If it fails, the entire eval is immediately classified as `failed` — regardless of how well every other assertion passed. Use gate for facts that must be true: "the agent called the correct tool", "the response parsed as valid JSON", "no shell commands errored."
Most matchers in `niceeval/expect` (`includes`, `equals`, `matches`, `satisfies`) default to gate. Scoped assertions like `t.succeeded()` and `t.calledTool()` also default to gate.
A soft assertion is a quality score with a numeric threshold. If the score falls below the threshold, the eval becomes `passed` rather than `failed` — a signal that there is a quality regression, but not a hard breakage. Soft failures only count as failures when you run with `--strict`.
Use soft for continuous judgments where the answer is "how good" rather than "correct or not": similarity measurement, LLM-as-judge factuality ratings, cost budgets you want to track without blocking CI.
Matchers that produce a continuous score (`similarity`) and all judge calls default to soft.
You can override the default severity with a chain method on any matcher or assertion:
```ts theme={null}
t.check(t.reply, includes("confirmed")); // gate (default)
t.check(t.reply, similarity(expected).gate()); // promote to gate
t.maxTokens(80_000).atLeast(0.7); // demote to soft with a threshold
```
## Verdict rules
Once all assertions are collected, the runner takes the first matching rule in this fixed order and folds them into a single result:
```
Execution error, timeout, or author mistake → errored
Any gate assertion failed, or a soft is below threshold --strict → failed
t.skip(reason) was called explicitly → skipped
Otherwise → passed
```
`errored` outranks everything else, because the execution evidence can no longer be trusted. `failed` outranks `skipped`, so that a `t.skip()` cannot mask a hard failure recorded earlier.
No errors, all gate assertions passed, all soft assertions met their thresholds (or you did not run with `--strict`).
At least one gate assertion did not pass, or a soft assertion fell below its threshold under `--strict`. Hard failure.
An execution error, timeout, or author mistake — this run cannot support a trustworthy conclusion, and it is not disguised as an assertion failure.
`t.skip("reason")` was called. Excluded from pass-rate calculations entirely.
When you run an eval more than once (`attempts > 1`), the per-eval summary becomes a **pass rate** (the fraction of attempts that produced `passed`) and an average latency, rather than a single verdict.
## 1. Value assertions — `niceeval/expect` matchers
`t.check(value, assertion)` evaluates the assertion immediately and records the result. `t.require(value, assertion)` does the same but **throws immediately** if the assertion fails, aborting the rest of the test function. Use `t.require` for preconditions: if a required fact is false, there is no point continuing.
The matchers available from `niceeval/expect`:
```ts theme={null}
import {
includes, // substring or regex match (default: gate)
equals, // deep equality (default: gate)
matches, // Standard Schema (Zod etc.) check (default: gate)
similarity, // normalized Levenshtein 0–1 (default: soft)
satisfies, // custom predicate + label (default: gate)
} from "niceeval/expect";
```
Usage examples:
```ts theme={null}
// Check that the agent's reply contains a specific string
t.check(t.reply, includes("order confirmed"));
// Deep-equal check on structured output
t.check(turn.data, equals({ status: "refund", amount: 42 }));
// Validate structured output against a Zod schema
t.check(turn.data, matches(z.object({ intent: z.enum(["refund", "ship"]) })));
// Similarity with an explicit threshold
t.check(t.reply, similarity("expected answer").atLeast(0.8));
// Custom predicate
t.check(turn.data, satisfies((d) => d.total > 0, "total is positive"));
```
Matchers are pure functions — `(value) => number` — so you can write your own and pass them to `t.check` without any special registration.
## 2. Scoped assertions
Scoped assertions are registered during `test(t)` but evaluated **after the function returns**, against the complete accumulated turn data. They read from the standard event stream that `t.send()` produces (see [Drive](/docs/explanation/drive)) and its derived facts — so as long as your adapter produces correct events, these assertions work identically for every agent.
Scoped assertions only appear on `t` if the agent has declared the corresponding capability. Calling `t.calledTool()` when the agent has not declared `toolObservability: true` is a compile error.
### Run / session dimension
```ts theme={null}
await t.succeeded().stopOnFailure(); // run completed; stop dependent checks if it did not
t.parked(); // cleanly stopped on a HITL input.requested event
t.messageIncludes("Regards,"); // all message events concatenated contain this string/regex
```
### Tool / action dimension
```ts theme={null}
t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });
t.notCalledTool("shell", { input: { command: /npm i/ } });
t.toolOrder(["read_file", "write_file"]); // relative order of tool calls
t.usedNoTools();
t.maxToolCalls(5);
t.loadedSkill("memory-v2"); // sugar for calledTool("load_skill", ...)
t.calledSubagent("researcher", { remoteUrl: /api\.example/ });
t.noFailedActions(); // no tool, subagent, or skill has status "failed"
```
The `input` argument to `calledTool` and `notCalledTool` supports a small matching language: a plain object performs deep partial matching, a `RegExp` matches against the serialized input, and a predicate function receives the raw input value.
### Event stream dimension (low-level escape hatch)
```ts theme={null}
t.event("input.requested", { count: 1 });
t.notEvent("error");
t.eventOrder(["operation.started", "operation.finished"]);
t.eventsSatisfy("read before write", (events) => /* custom predicate */ true);
```
All scoped assertions above are syntactic sugar for these low-level event stream queries. When none of the higher-level assertions fit your use case, you can drop down to `eventsSatisfy` and write an arbitrary predicate over the raw `StreamEvent[]`.
### Structured output (on `turn`, not `t`)
```ts theme={null}
const turn = await t.send("Return the result as JSON");
turn.outputEquals({ status: "ok" }); // deep equality on turn.data
turn.outputMatches(z.object({ status: z.string() })); // Standard Schema validation
```
### Workspace dimension (Sandbox agents only)
```ts theme={null}
t.sandbox.fileChanged("src/Button.tsx");
t.sandbox.fileDeleted("src/old.ts");
t.sandbox.noChanges(); // no repository files were modified this turn
t.sandbox.notInDiff(/sk-[A-Za-z0-9]/); // attributed diff contains no secrets / inline styles
t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded()); // npm test exited 0
t.check(await t.sandbox.runCommand("npm", ["run", "build"]), commandSucceeded()); // npm run build exited 0
t.sandbox.noFailedShellCommands();
```
These declarations are evaluated against the agent-attributed diff after the run: `fileChanged("src/Button.tsx")` matches when the Agent modified that path, `fileDeleted("src/old.ts")` matches when the Agent deleted it, and `changedPaths([...])` matches the exact unordered set of changed paths. `fileChanged(path, { before, after, status })` also matches content: `before` and `after` are read from the same change's endpoints.
To read a file's current content — for example, to hand it to a Judge as explicit `{ input, output }` material — use `await t.sandbox.readText(path)`. Attribution is decided by `fileChanged`, not by what you read.
Scoped assertions follow one rule everywhere: **the receiver decides the scope, not the assertion name.** `t.*` aggregates every turn of the whole eval run (including any `t.newSession()` sessions); `session.*` (from `t.newSession()`) scopes to that one session; `turn.*` (from `t.send()`'s return value) scopes to that single turn only. Same vocabulary, different receiver — see [Drive](/docs/explanation/drive) for what each receiver is.
## 3. Project-test assertions (Sandbox evals)
For Sandbox coding evals, run validation commands inside `test(t)` and record their result as assertions.
```ts theme={null}
import { commandSucceeded, includes } from "niceeval/expect";
const testResult = await t.sandbox.runCommand("npm", ["test"]);
t.check(testResult, commandSucceeded());
const src = await t.sandbox.runShell("find . -name '*.ts' -exec cat {} +");
t.check(src.stdout, includes(/z\.object\s*\(/));
```
You can also assert behavior through the standard event stream: `t.calledTool(...)`, `t.sandbox.noFailedShellCommands()`, `t.eventsSatisfy(...)`, and diff assertions such as `t.sandbox.fileChanged(...)`.
## 4. Efficiency / cost assertions
Token usage is a first-class evaluation dimension. An agent that answers correctly but burns far more tokens than expected should not be treated identically to one that answers efficiently.
```ts theme={null}
t.maxTokens(50_000); // hard token limit for the entire run (gate by default)
t.maxCost(0.5); // estimated cost cap in USD (requires a price table in config)
t.maxTokens(80_000).atLeast(0.7); // soft variant — tracked but only red under --strict
t.check(t.usage.outputTokens, satisfies((n) => n < 10_000, "not verbose"));
```
`t.usage` is available anywhere inside `test(t)` and exposes `{ inputTokens, outputTokens, cacheReadTokens?, … }`. For Sandbox agents, token counts are extracted from the transcript by the adapter; for Direct Agents, they are returned in `Turn.usage`.
## Custom scorers
A value assertion is just a function `(value) => number | Promise`. You can write custom matchers using `makeAssertion`:
```ts theme={null}
import { makeAssertion } from "niceeval/expect";
import type { Assertion } from "niceeval/expect";
function jsonValid(): Assertion {
return makeAssertion({
name: "jsonValid",
severity: "gate",
score: (value) => {
try { JSON.parse(String(value)); return 1; }
catch { return 0; }
},
});
}
t.check(t.reply, jsonValid());
```
Custom matchers compose with the same chain methods as built-ins: `.gate()`, `.atLeast(0.7)`.
## Related reading
* [Drive](/docs/explanation/drive) — `t.send()`, `t.newSession()`, and HITL: how you produce the Turn data these assertions read from.
* [Judge](/docs/explanation/judge) — the fifth assertion mechanism, for open-ended quality that can't be expressed as a fixed rule.
* [Write send](/docs/tutorials/write-send) — how the standard event stream is produced, and what scoped assertions depend on.
* [Evals](/docs/explanation/evals) — how assertions fold into the eval lifecycle and verdict types.