Evaluation kinds, assertions, and cost limits

A good eval should decompose “did this succeed?” into signals you can explain. NiceEval lets you mix precise assertions, semantic judges, event-stream checks, and real tests.

Choose the assertion type

Goal	Recommended mechanism
Reply contains a fixed substring or matches a regex	`includes`
Structured JSON matches exactly	`equals`
A tool call happened	`t.calledTool` / `t.notCalledTool`
Output is semantically correct	`t.judge.*`
Code actually works	`t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded())`
Cost stays under control	`t.maxTokens` / `t.maxCost`

Value assertions

import { includes, equals } from "niceeval/expect";

t.check(t.reply, includes("refund"));
t.check(turn.data, equals({ intent: "refund" }));
t.check(t.reply, includes(/order #\d+/));

includes accepts both substrings and regexes. matches only accepts a Standard Schema / Zod schema, for structural validation — it does not do regex matching. Value assertions fit results that are precise, stable, and low-ambiguity.

Scoped assertions

t.succeeded();
t.calledTool("search");
t.usedNoTools();
t.sandbox.fileChanged("src/app.ts");
t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());

These check facts about the whole run, usually from the standard event stream or sandbox artifacts.

LLM-as-judge

t.judge.autoevals.factuality("Refunds are available within 30 days.", { on: t.reply }).atLeast(0.8);
t.judge.autoevals.closedQA("The answer is specific, polite, and does not invent policy.", { on: t.reply }).atLeast(0.75);

Judge checks fit semantic quality, but they don’t replace every deterministic assertion. Prefer exact checks wherever you can make them.

Gate vs soft

t.check(t.reply, includes("refund").gate());
t.check(t.reply, includes("friendly tone").atLeast(0.7));

gate: a failure is a failure.
soft: keep the score, used to compare quality.

Use tests to score code tasks

Run project tests or short probe scripts inside .eval.ts:

import { commandSucceeded, includes } from "niceeval/expect";

const testResult = await t.sandbox.runCommand("npm", ["test"]);
t.check(testResult, commandSucceeded());

const src = await t.sandbox.runShell("find . -name '*.ts' -exec cat {} +");
t.check(src.stdout, includes(/z\.object\s*\(/));

Cost and efficiency

t.maxTokens(25_000);
t.maxCost(0.1);

This is especially useful for coding agents and long-chain agents, where hiding quality problems behind heavy retries or excessive tool calls becomes possible.

Practical advice

Write one or two gates first to establish the task’s baseline.
Then add soft scores to compare quality.
Use Judge for complex semantics, but give the Judge a clear rubric.
For coding-agent results, verify with real tests wherever possible.
Write failure messages clearly, so you can locate problems directly from the report.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Evaluation kinds, assertions, and cost limits

Choose the assertion type

Value assertions

Scoped assertions

LLM-as-judge

Gate vs soft

Use tests to score code tasks

Cost and efficiency

Practical advice

​Choose the assertion type

​Value assertions

​Scoped assertions

​LLM-as-judge

​Gate vs soft

​Use tests to score code tasks

​Cost and efficiency

​Practical advice

Choose the assertion type

Value assertions

Scoped assertions

LLM-as-judge

Gate vs soft

Use tests to score code tasks

Cost and efficiency

Practical advice