> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation kinds, assertions, and cost limits

> Pass and points evals, value and scoped assertions, LLM-as-judge, and efficiency checks.

A good eval should decompose "did this succeed?" into signals you can explain. [NiceEval](https://niceeval.com/) lets you mix precise assertions, semantic judges, event-stream checks, and real tests.

## Choose the assertion type

| Goal                                                | Recommended mechanism                                                      |
| --------------------------------------------------- | -------------------------------------------------------------------------- |
| Reply contains a fixed substring or matches a regex | `includes`                                                                 |
| Structured JSON matches exactly                     | `equals`                                                                   |
| A tool call happened                                | `t.calledTool` / `t.notCalledTool`                                         |
| Output is semantically correct                      | `t.judge.*`                                                                |
| Code actually works                                 | `t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded())` |
| Cost stays under control                            | `t.maxTokens` / `t.maxCost`                                                |

## Value assertions

```ts theme={null}
import { includes, equals } from "niceeval/expect";

t.check(t.reply, includes("refund"));
t.check(turn.data, equals({ intent: "refund" }));
t.check(t.reply, includes(/order #\d+/));
```

`includes` accepts both substrings and regexes. `matches` only accepts a Standard Schema / Zod schema, for structural validation — it does not do regex matching. Value assertions fit results that are precise, stable, and low-ambiguity.

## Scoped assertions

```ts theme={null}
t.succeeded();
t.calledTool("search");
t.usedNoTools();
t.sandbox.fileChanged("src/app.ts");
t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());
```

These check facts about the whole run, usually from the standard event stream or sandbox artifacts.

## LLM-as-judge

```ts theme={null}
t.judge.autoevals.factuality("Refunds are available within 30 days.", { on: t.reply }).atLeast(0.8);
t.judge.autoevals.closedQA("The answer is specific, polite, and does not invent policy.", { on: t.reply }).atLeast(0.75);
```

Judge checks fit semantic quality, but they don't replace every deterministic assertion. Prefer exact checks wherever you can make them.

## Gate vs soft

```ts theme={null}
t.check(t.reply, includes("refund").gate());
t.check(t.reply, includes("friendly tone").atLeast(0.7));
```

* `gate`: a failure is a failure.
* `soft`: keep the score, used to compare quality.

## Use tests to score code tasks

Run project tests or short probe scripts inside `.eval.ts`:

```ts theme={null}
import { commandSucceeded, includes } from "niceeval/expect";

const testResult = await t.sandbox.runCommand("npm", ["test"]);
t.check(testResult, commandSucceeded());

const src = await t.sandbox.runShell("find . -name '*.ts' -exec cat {} +");
t.check(src.stdout, includes(/z\.object\s*\(/));
```

## Cost and efficiency

```ts theme={null}
t.maxTokens(25_000);
t.maxCost(0.1);
```

This is especially useful for coding agents and long-chain agents, where hiding quality problems behind heavy retries or excessive tool calls becomes possible.

## Practical advice

* Write one or two gates first to establish the task's baseline.
* Then add soft scores to compare quality.
* Use Judge for complex semantics, but give the Judge a clear rubric.
* For coding-agent results, verify with real tests wherever possible.
* Write failure messages clearly, so you can locate problems directly from the report.