> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals in NiceEval: lifecycle, verdicts, and files

> An eval is a test case: a description and a test function, agent-neutral. Learn how evals are discovered, scheduled, scored, and reported.

An eval is a runnable test case. It usually lives in a `*.eval.ts` file and is declared with `defineEval`.

## What an eval contains

```ts theme={null}
import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    await t.send("What's the weather like in Brooklyn today?");
    t.succeeded();
    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
    t.check(t.reply, includes("sunny"));
  },
});
```

Core fields:

| Field         | Meaning                                                                                                                      |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `description` | Human-readable description shown in reports                                                                                  |
| `environment` | Optional environment requirement profile, mapped by the Sandbox spec's `environments` table to a concrete preset environment |
| `test(t)`     | Interaction and assertion logic                                                                                              |

An eval does not declare which Agent it runs against — it stays agent-neutral by default, so the same eval can run against different Agents under different experiments. Choosing the Agent is a field of the experiment, not something the CLI overrides ad hoc.

<Note>
  Do not hand-write `id` or `name`. [NiceEval](https://niceeval.com/) derives the eval ID from the file path.
</Note>

## Path is identity

`evals/weather/brooklyn.eval.ts` has ID `weather/brooklyn`. Positional arguments after the experiment name filter by ID prefix:

```bash theme={null}
npx niceeval exp local weather
npx niceeval exp local weather/brooklyn
```

This keeps IDs stable and readable, and naturally consistent with the directory structure. Positional arguments use a bare string-prefix match: `terminal-swe-bench` also matches `terminal-swe-bench-astropy-1` — the character right after the match is not required to be `/`.

## Lifecycle

<Steps>
  <Step title="Discovery">
    The runner loads the `*.eval.ts` files and fixture directories under `evals/`.
  </Step>

  <Step title="Scheduling">
    Build the execution plan from concurrency, cache, attempts, and early exit.
  </Step>

  <Step title="agent.send">
    `t.send()` calls the selected Adapter and receives a standard `Turn`.
  </Step>

  <Step title="Evaluate assertions">
    [NiceEval](https://niceeval.com/) collects value assertions, scoped assertions, judge scores, and test results.
  </Step>

  <Step title="Verdict">
    All assertion results fold into one final verdict.
  </Step>

  <Step title="Report">
    The console and reporters emit the results, and facts are committed to the `.niceeval/` Record root.
  </Step>
</Steps>

## Verdict types

<CardGroup cols={2}>
  <Card title="passed" icon="circle-check" color="#22c55e">
    All gate assertions passed (under `--strict`, soft assertions also met their thresholds), and no execution error occurred.
  </Card>

  <Card title="failed" icon="circle-xmark" color="#ef4444">
    At least one gate assertion failed, or under `--strict` a soft assertion came in below its threshold.
  </Card>

  <Card title="errored" icon="triangle-exclamation" color="#f59e0b">
    An execution error, timeout, or authoring error — this run cannot produce a trustworthy conclusion.
  </Card>

  <Card title="skipped" icon="forward" color="#6b7280">
    The eval skipped itself, usually through `t.skip(reason)`.
  </Card>
</CardGroup>

## Gate and soft

`gate` is a hard threshold. Failure makes the eval fail. `soft` contributes a score without necessarily failing the eval. See [Assert](/docs/explanation/assert) for the full rules.

## The `*.eval.ts` convention

Only files ending in `.eval.ts` are discovered. Use directories for grouping:

```text theme={null}
evals/
└─ billing/
   └─ refund.eval.ts  # id: billing/refund
```

## Array exports and data-driven tests (dataset fan-out)

A file can also default-export an array of `defineEval(...)` calls to generate multiple cases from the same logic:

```ts theme={null}
export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.expected));
    },
  }),
);
```

This generates IDs such as `sql/0000` and `sql/0001`. See [Dataset fan-out](/docs/tutorials/dataset-fanout).

## Related reading

* [Experiment](/docs/explanation/experiment) — The other half: who to evaluate and how to run it, and why it is kept separate from the eval (late binding).
* [Assert](/docs/explanation/assert) — The full verdict rules for gate and soft assertions.