niceeval/expect Match Reference

niceeval/expect provides only pure Match factories. A Match compares one candidate value; it does not record a result or decide a Verdict or score. t.check(value, match) reads value when called and registers an Assertion directly; the returned handle only configures that same entry.

import { and, excludes, includes, pattern } from "niceeval/expect";

How to use it

t.check(t.reply, includes("confirmed"));
t.check(t.reply, and(pattern(/order #\d+/i), excludes("cancelled")));
const config = await t.check(rawConfig, pattern(/runtime/)).orStop();

Entry point	Purpose
`t.check(value, match)`	Takes exactly two arguments and registers an Assertion when called; a failure does not stop later code
`await handle.orStop()`	An async barrier on the same handle; stops the current continuation on mismatch or below
`handle.score(n)` / `t.score(n)`	Lets this item contribute a score in a scored Eval, or registers a contribution directly

Numeric units

Calculate with ordinary numbers, then wrap the result with ms, seconds, tokens, usd, ratio, count, or bytes. Each constructor returns a frozen Quantity<U> with .value and .unit. Quantities do not support arithmetic or automatic conversion. Constructors require finite numbers. ratio(n) also requires 0 ≤ n ≤ 1; invalid values throw RangeError.

import { atMost, between, measured, ms, quantile, ratio } from "niceeval/expect";

const latency = measured(ms(512), { unavailable: "missing" });
t.check(latency, atMost(ms(600)));
t.check(quantile([ms(100), ms(200)], 0.9), between(ms(100), ms(200)));
t.check(t.usage.cacheRead.rate(), atMost(ratio(0.9)));

measured accepts number | null or Quantity<U> | null. quantile accepts ordinary numbers or quantities sharing one unit, and preserves that unit. Mixing units or mixing numbers with quantities throws TypeError. Empty samples and missing observations remain unavailable. atLeast, atMost, greaterThan, and lessThan require thresholds with the observation’s unit. between(minimum, maximum) uses inclusive bounds with one unit, ordered from lower to upper. Unitless observations require ordinary numeric thresholds. TypeScript rejects incompatible units. When JavaScript bypasses the types, the check becomes unavailable and reports the mismatch. show and view format units as 512 ms, 61.2%, and $0.031.

Matchers

The signature and description for every matcher below are generated from the niceeval/expect source and stay in sync with the current implementation.

`and`

export function and(
  first: BooleanMatch<unknown, unknown, MatchDomain>,
  ...rest: readonly BooleanMatch<unknown, unknown, MatchDomain>[]
): BooleanMatch<unknown, unknown, MatchDomain> { ... }

Requires every Boolean Match to hold for the same candidate. All operands must belong to the same domain.

`or`

export function or(
  first: BooleanMatch<unknown, unknown, MatchDomain>,
  ...rest: readonly BooleanMatch<unknown, unknown, MatchDomain>[]
): BooleanMatch<unknown, unknown, MatchDomain> { ... }

Accepts a candidate when any Boolean Match holds. All operands must belong to the same domain, and unknown evidence is not a definite mismatch.

`not`

export function not<T, R extends T>(match: BooleanMatch<T, R, "value">): BooleanMatch<T, T, "value"> { ... }

Negates a Boolean Match within its domain. Unavailable evidence remains unavailable rather than becoming a successful negation.

`includes`

export function includes(text: string, options?: TextMatchOptions): BooleanMatch<string, string> { ... }

Checks that text contains a substring. Comment stripping is opt-in, and an empty expected string is rejected.

`excludes`

export function excludes(text: string, options?: TextMatchOptions): BooleanMatch<string, string> { ... }

Checks that text does not contain a substring. Comment stripping is opt-in, and an empty expected string is rejected.

`pattern`

export function pattern(expression: RegExp, options?: TextMatchOptions): BooleanMatch<string, string> { ... }

Checks text against a regular expression. Each evaluation starts with a fresh cursor and leaves the supplied expression unchanged.

`similarity`

export function similarity(expected: string): ScoreMatch<string> { ... }

Normalized Levenshtein similarity; it is a continuous score and carries no default threshold or Verdict policy.

`includesUrl`

export function includesUrl(min = 1): BooleanMatch<string, string> { ... }

A plain-text matcher that requires at least min distinct http(s) URLs.

`hasSections`

export function hasSections(min = 2): BooleanMatch<string, string> { ... }

A plain-text matcher that requires at least min Markdown headings.

`isDefined`

export function isDefined<T = unknown>(label?: string): BooleanMatch<T, Exclude<T, null | undefined>> { ... }

`isTrue`

export function isTrue<T = unknown>(label?: string): BooleanMatch<T, T & true> { ... }

A refinement matcher for value === true.

`isFalse`

export function isFalse<T = unknown>(label?: string): BooleanMatch<T, T & false> { ... }

A refinement matcher for value === false.

`commandSucceeded`

export function commandSucceeded<T = unknown>(): BooleanMatch<T, T & { readonly exitCode: 0 }> { ... }

An exitCode === 0 matcher for CommandResult-like candidates; it does not read stdout / stderr.

`equals`

export function equals<const T>(expected: T): BooleanMatch<unknown, T> { ... }

Checks deep value equality against the supplied reference. It compares structure rather than object identity.

`matches`

export function matches<S extends StandardSchema>(
  schema: S,
): BooleanMatch<unknown, StandardSchema.InferInput<S>, "value"> { ... }

Validates a value using Standard Schema v1. Refinement preserves the original candidate and ignores schema transforms.

`satisfies`

export function satisfies<T, R extends T = T>(
  label: string,
  predicate: (value: T) => boolean | Promise<boolean>,
): BooleanMatch<T, R> | BooleanMatch<T, T> { ... }

Wraps a named synchronous or asynchronous Boolean predicate as a value Match. The callback must return a Boolean and remain deterministic and free of side effects.

`defineValueMatch`

export function defineValueMatch<T, R extends T = T>(spec: {
  readonly name: string;
  readonly evaluate: (value: T) => boolean | { readonly state: "unavailable"; readonly reason: string } | Promise<boolean | { readonly state: "unavailable"; readonly reason: string }>;
}): BooleanMatch<T, R> | BooleanMatch<T, T> { ... }

Defines a reusable Boolean value comparison with optional refinement. Report unknown evidence explicitly as unavailable rather than returning false.

`defineScoreMatch`

export function defineScoreMatch<T, E = ScoreMatchLlmFailure>(spec: ManagedScoreMatchOptions<T, E> | {
  readonly name: string;
  readonly score: (value: T) => number | Promise<number>;
}): ScoreMatch<T> { ... }

Defines a reusable finite [0,1] measurement, optionally using managed LLM primitives. Apply thresholds through the assertion handle, not inside Match configuration.

`eventMatch`

export function eventMatch<K extends keyof EventOptionsByType>(
  type: K,
  options?: EventOptionsByType[K],
): EventMatch<Extract<EventOccurrenceView, { readonly type: K }>> { ... }

Matches a supported message or operation event. Use event-specific filters; tool filters belong on operation events.

`mapValue`

export function mapValue<T, U>(
  label: string, project: (value: T) => U, match: Match<NoInfer<U>, "value"> & { readonly kind: "boolean" },
): BooleanMatch<T, T> { ... }

Project a single value without replacing the original subject’s refinement.

`mapEach`

export function mapEach<T, U, Subject extends CollectionValue<T>>(
  project: (value: T) => U,
  aggregate: Match<CollectionValue<NoInfer<U>>, "value"> & { readonly kind: "boolean" },
): BooleanMatch<Subject & CollectionValue<T>, Subject> { ... }

Project each complete item, retaining its material identity and order.

`filterWhere`

export function filterWhere<T, R extends T, Subject extends CollectionValue<T>>(
  selector: BooleanMatch<T, R>, aggregate: BooleanMatch<CollectionValue<R>, CollectionValue<R>>,
): BooleanMatch<Subject, Subject> { ... }

Select only definite matches; an unknown selector makes the aggregate unavailable.

`countWhere`

export function countWhere<T, Subject extends CollectionValue<T>>(
  item: BooleanMatch<T, NoInfer<T>>, count: Match<number, "value"> & { readonly kind: "boolean" },
): BooleanMatch<Subject, Subject> { ... }

Compare an exact match count only after every item in a complete collection is known.

`defineMaterialMatch`

export function defineMaterialMatch(definition: unknown): ContextualMatch { ... }

Defines a synchronous context collection reader and item selector. Preserve collection completeness and unique item IDs instead of silently discarding unknown items.

`defineContextMatch`

export function defineContextMatch(definition: unknown): ContextualMatch { ... }

Defines a synchronous readonly context reader and a Match for one fact. Return an unavailable fact explicitly when evidence cannot be read.

Match tool calls

ToolMatch compares one LogicalToolOccurrence, and expresses the occurrence count with methods such as .exactly(n) and .atLeast(n). calledTool accepts a name, an unquantified ToolMatch, or a positive occurrence Match; notCalledTool uses an unquantified selector to express zero matches.

import {
  commandMatch,
  jsonMatch,
  referencesAnyPath,
  toolMatch,
} from "niceeval/expect";

turn.calledTool(
  toolMatch("get_weather", {
    input: jsonMatch({ city: "Taipei", unit: "celsius" }),
    status: "completed",
  }).exactly(1),
).label("Query weather precisely");

turn.calledTool(
  toolMatch("read_file", {
    input: referencesAnyPath([".env", "secrets"]),
  }),
).label("Read a restricted path");

turn.notCalledTool(commandMatch("rm", { argsStart: ["-rf"] }));

Use jsonMatch for ordinary JSON, referencesAnyPath for paths, and commandMatch for command tokens. When output is absent, material is incomplete, or HITL has not finished, the Match becomes unavailable; it does not pretend to be an ordinary mismatch. referencesAnyPath searches literal path components and directory prefixes, with case-sensitive matching by default. It does not support glob patterns or interpret command semantics. Use secrets, rather than secrets/**, for that directory. For example, rg --files -g '!secrets/**' still mentions secrets and matches even though it excludes the directory. A match does not prove that a file was read or changed. Use commandMatch for command arguments and file-change assertions for changes. For complete signatures, material states, and three-valued counting rules, see Scoped assertions.

Common usage examples

t.check(t.reply, includes("Paris"));
t.check(t.reply, pattern(/order #\d+/i));
t.check(turn.data, equals({ intent: "refund" }));
t.check(turn.data, matches(OrderSchema));
t.check(turn.data, satisfies("returns an array", Array.isArray));
t.check(t.reply, similarity("The answer explains the refund window")).gate(0.8);

Composition and three states

t.check(t.reply, and(includes("required"), excludes("forbidden")));
t.check(t.reply, or(includes("approved"), includes("accepted")));

Matches within the same and / or must belong to the same domain. The internal result of a Boolean Match is matched, mismatched, or unavailable. or returns unavailable only when no child matches and at least one child’s evidence is incomplete; it does not treat unknown as negative. A Match itself holds no Verdict, scoring, or control flow; .gate(minimum), .score(n), and .orStop(minimum) for a continuous measurement all belong to the registered AssertionHandle. The following lists stable fields on Match values themselves. Brands and evaluators are module-private implementation details, not author API.

`Match`

`domain`

readonly domain: D;

`name`

readonly name: string;

`BooleanMatch`

`kind`

readonly kind: "boolean";

`[matchRefinementBrand]`

readonly [matchRefinementBrand]: () => R;

`ScoreMatch`

`kind`

readonly kind: "score";

Ordering conditions in a collection

inOrder combines 2–64 ordinary value Matches into a collection Match, which you pass to t.check or to an application’s collection assertion method. It accepts a read-only array or a MaterialCollection with a completeness state, and looks for a strictly increasing subsequence by original position; the items need not be adjacent.

import { equals, inOrder, not, or } from "niceeval/expect";

t.check([1, 0, 2], inOrder([equals(1), equals(2)])).gate();
t.check([2, 1], not(inOrder([equals(1), equals(2)]))).gate();
t.check([3, 4], or(
  inOrder([equals(1), equals(2)]),
  inOrder([equals(3), equals(4)]),
)).gate();

The same item cannot satisfy two steps, and each step can use and to constrain the same fact. The application is responsible for providing the formal commit order; a request being sent does not mean the operation was adopted or finished executing. When you only need both parties to have spoken, check existence separately. An incomplete collection returns undecidable. A complete collection passes when there is known ordering evidence, fails when no ordering is possible, and is undecidable when it can only hold by depending on unknown items. Diagnostics keep the original positions, material IDs, and step judgments. Ordering combinations of ToolMatch and EventMatch keep their own Agent scope and cannot be mixed with ordinary value steps.

Custom matchers

import { defineValueMatch } from "niceeval/expect";

const validEmail = () =>
  defineValueMatch<string>({
    name: "valid email",
    evaluate(value) {
      return value.includes("@");
    },
  });

t.check(t.reply, validEmail());

Use defineValueMatch for a custom Boolean Match and defineScoreMatch for a continuous score. Both implement only a single-candidate comparison; reading files, scanning events, deciding coverage, and changing a Verdict remain the responsibility of Assertion registration entry points and the Runner.