niceeval/expect matchers and custom assertion reference

niceeval/expect provides a set of composable matchers, passed to t.check() or t.require(). A matcher returns an Assertion with a default severity: gate or soft.

import { includes, equals, matches, similarity, satisfies } from "niceeval/expect";

How matchers are used

t.check(t.reply, includes("confirmed"));
t.check(turn.data, equals({ intent: "refund" }));
t.require(turn.status, equals("completed"));

Method	On failure	Best for
`t.check`	Records the result, execution continues	Most assertions
`t.require`	Throws immediately, aborting the test	Preconditions where continuing makes no sense

Matchers

The signature and description of each matcher below is generated from the niceeval/expect source, and stays in sync with the current implementation.

`includes`

export function includes(needle: string | RegExp, opts?: MatchOptions): ValueAssertion { ... }

1 if String(value) contains the substring / matches the regex, otherwise 0. A hard gate by default. With opts.stripComments, only real code is examined.

`excludes`

export function excludes(needle: string | RegExp, opts?: MatchOptions): ValueAssertion { ... }

The inverse of includes: 1 if it does not contain the substring / does not match the regex, otherwise 0. A hard gate by default. With opts.stripComments, only real code is examined.

`equals`

export function equals(expected: unknown): ValueAssertion { ... }

1 on deep equality, otherwise 0. A hard gate by default.

`matches`

export function matches(schema: unknown): ValueAssertion { ... }

Validates value against a schema — this is not regex matching, it’s Standard Schema / zod-style structural validation. Prefers Standard Schema (schema['~standard'].validate), otherwise falls back to zod-style .safeParse / .parse. 1 if validation passes, otherwise 0; any exception → 0. A hard gate by default.

`similarity`

export function similarity(expected: string): ValueAssertion { ... }

Pure string edit distance, not semantic similarity — a normalized Levenshtein distance in [0,1] (1 − edit distance / length of the longer string), with no understanding of meaning, so paraphrasing / reordering scores low. A soft score by default, threshold 0.6.

`includesUrl`

export function includesUrl(min = 1): ValueAssertion { ... }

1 if the text contains at least min (default 1) deduplicated http(s) links, otherwise 0. A hard gate by default. A shape assertion for “does the answer cite real sources”: when no Judge key is available, this is the lowest-cost backstop for “cites at least one source link” — echoing the question back cannot pass it; fabricated links are left for negative cases or the Judge to catch.

`hasSections`

export function hasSections(min = 2): ValueAssertion { ... }

1 if the text contains at least min (default 2) Markdown headings (line-leading # through ######), otherwise 0. A hard gate by default. A shape assertion for “is the answer a structured document” — suited to research reports, design docs, and other produce-type answers; a wall of flowing text with no section headings does not pass.

`satisfies`

export function satisfies(predicate: (v: unknown) => boolean, label?: string): ValueAssertion { ... }

1 if the predicate is true, otherwise 0. A hard gate by default; label is added to the name for easier identification in reports.

`isDefined`

export function isDefined(label?: string): ValueAssertion { ... }

1 if value is not null / not undefined, otherwise 0. Saves the boilerplate of x !== undefined + isTrue. A hard gate by default.

`isTrue`

export function isTrue(label?: string): ValueAssertion { ... }

1 if value === true, otherwise 0. A boolean assertion with a label (for checks like fileExists). A hard gate by default.

`commandSucceeded`

export function commandSucceeded(): ValueAssertion { ... }

1 if CommandResult.exitCode === 0, otherwise 0. A hard gate by default.

`isFalse`

export function isFalse(label?: string): ValueAssertion { ... }

1 if value === false, otherwise 0. A boolean assertion with a label. A hard gate by default.

`makeAssertion`

export function makeAssertion(spec: {
  name: string;
  severity?: Severity;
  threshold?: number;
  score: (value: unknown) => number | Promise<number>;
}): ValueAssertion { ... }

Custom assertion factory: give it a name / severity / threshold / score directly, and one call returns a ready-to-use ValueAssertion — unlike gate()/atLeast(), it does not need a second chained call to set the level. severity defaults to gate when omitted.

Common usage examples

t.check(t.reply, includes("Paris"));
t.check(t.reply, includes(/order #\d+/i));
t.check(turn.data, equals({ intent: "refund" }));
t.check(turn.data, matches(OrderSchema)); // Standard Schema / Zod structural validation; use includes for regex matching
t.check(t.reply, similarity("The answer explains the refund window").atLeast(0.8));
t.check(turn.data, satisfies((value) => Array.isArray(value)));

`gate` and `soft`

t.check(t.reply, includes("required").gate());
t.check(t.reply, includes("nice to have").atLeast(0.7));

Every Assertion (i.e. ValueAssertion) returned by a matcher has these members; .gate() turns it into a hard gate, .atLeast() turns it into a soft threshold:

`name`

readonly name: string;

`severity`

readonly severity: Severity;

`threshold`

readonly threshold?: number;

`isOptional`

readonly isOptional?: boolean;

Marker set by chaining .optional(): when this assertion can’t be scored, it is only recorded as unavailable, without dragging the attempt into errored.

`expected`

readonly expected?: string;

A bounded text description of the expected condition (e.g. contains "Brooklyn"), included in AssertionResult.expected on failure.

`score`

score(value: unknown): number | Promise<number>;

`gate`

gate(threshold?: number): ValueAssertion;

Turns it into a hard-gate assertion: when the threshold isn’t met (omitting threshold judges by score > 0), the whole eval is marked failed. Returns a new instance, does not mutate the original.

`atLeast`

atLeast(threshold: number): ValueAssertion;

Turns it into a soft-threshold assertion: when threshold isn’t met, this assertion is recorded as failed, but by default it does not drag down the whole eval’s verdict; under --strict, a soft-threshold failure also makes the whole eval’s verdict count as failed. Returns a new instance, does not mutate the original.

`optional`

optional(): ValueAssertion;

Allows this assertion’s evidence to be absent: when it can’t be scored, only records outcome: "unavailable", without affecting the verdict. Orthogonal to severity (severity determines whether it affects the quality verdict; optional determines whether evidence is allowed to be absent). Returns a new instance, does not mutate the original.

Custom matchers

import { makeAssertion, type Assertion } from "niceeval/expect";

function validEmail(): Assertion {
  return makeAssertion({
    name: "valid email",
    score(value) {
      return typeof value === "string" && value.includes("@") ? 1 : 0;
    },
  });
}

t.check(t.reply, validEmail());

makeAssertion takes a single spec object (name, optional severity/threshold, and score) and returns an Assertion directly; by convention you wrap it in a function of the same name and call that, rather than calling makeAssertion itself as the factory at the call site.

API

Adapters

Results & Reports

CLI

niceeval/expect matchers and custom assertion reference

How matchers are used

Matchers

`includes`

`excludes`

`equals`

`matches`

`similarity`

`includesUrl`

`hasSections`

`satisfies`

`isDefined`

`isTrue`

`commandSucceeded`

`isFalse`

`makeAssertion`

Common usage examples

`gate` and `soft`

`name`

`severity`

`threshold`

`isOptional`

`expected`

`score`

`gate`

`atLeast`

`optional`

Custom matchers

​How matchers are used

​Matchers

​includes

​excludes

​equals

​matches

​similarity

​includesUrl

​hasSections

​satisfies

​isDefined

​isTrue

​commandSucceeded

​isFalse

​makeAssertion

​Common usage examples

​gate and soft

​name

​severity

​threshold

​isOptional

​expected

​score

​gate

​atLeast

​optional

​Custom matchers

How matchers are used

Matchers

`includes`

`excludes`

`equals`

`matches`

`similarity`

`includesUrl`

`hasSections`

`satisfies`

`isDefined`

`isTrue`

`commandSucceeded`

`isFalse`

`makeAssertion`

Common usage examples

`gate` and `soft`

`name`

`severity`

`threshold`

`isOptional`

`expected`

`score`

`gate`

`atLeast`

`optional`

Custom matchers