> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Data-driven testing (dataset fan-out): run one eval suite across many rows of data

> Export an array or a keyed record from a .eval.ts file to fan one eval out into many cases. Use loadYaml or loadJson to read external datasets, with stable IDs.

Data-driven testing (dataset fan-out) is a good fit when many test cases share the same structure and only the inputs change. Typical examples are SQL generation, intent classification, retrieval QA, and tool selection.

## How fan-out works

When there is no external business ID, a `.eval.ts` file exports an array by default:

```ts theme={null}
import { defineEval } from "niceeval";
import { equals } from "niceeval/expect";

const rows = [
  { task: "Count users", prompt: "Count all users", sql: "SELECT COUNT(*) FROM users;" },
  { task: "Recent orders", prompt: "Find recent orders", sql: "SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;" },
];

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.sql));
    },
  }),
);
```

## Generated IDs

If the file is `evals/sql.eval.ts`, the generated IDs are:

```text theme={null}
sql/0000
sql/0001
```

The numeric suffix is zero-padded so IDs stay stable and easy to filter.

When the data source already carries a stable case, issue, or benchmark ID, export a keyed record by default instead:

```ts theme={null}
export default Object.fromEntries(
  rows.map((row) => [
    row.issueId,
    defineEval({
      description: row.title,
      async test(t) {
        await t.send(row.prompt);
        t.succeeded();
      },
    }),
  ]),
);
```

If the file is `evals/swelancer.eval.ts` and the key is `15193`, the ID is `swelancer/15193`. A key must be a non-empty path segment: it cannot be `.` or `..`, and it cannot contain `/`, `\`, or control characters. NiceEval discovers cases in key lexicographic order, so a change in the data source's return order never changes run order.

## Loading from YAML and JSON

```ts theme={null}
import { loadYaml } from "niceeval/loaders";
import { z } from "zod";

const SqlCases = z.object({
  cases: z.array(z.object({ task: z.string(), prompt: z.string(), sql: z.string() })),
});
const doc = await loadYaml("evals/data/sql-cases.yaml", (value) => SqlCases.parse(value));
const rows = doc.cases;
```

```yaml theme={null}
cases:
  - task: Count users
    prompt: Count all rows in the users table
    sql: SELECT COUNT(*) FROM users;
```

`loadYaml` and `loadJson` require a decoder. The unvalidated dynamic value exists only at the decoder input; after validation, the return value is strongly typed data that the Eval can use directly.

## Filtering dataset evals

```bash theme={null}
# Run the whole dataset
npx niceeval exp local sql

# Run only the first array case
npx niceeval exp local sql/0000

# Run a keyed case
npx niceeval exp local swelancer/15193
```

## Datasets vs separate files

<Tabs>
  <Tab title="Use a dataset">
    The cases have exactly the same structure. Only the input and expected output change.
  </Tab>

  <Tab title="Use separate files">
    Each case has a different flow, different assertions, or needs a different agent configuration.
  </Tab>
</Tabs>

<Tip>
  Datasets are good for broad horizontal coverage. Separate eval files are better when the behavior itself is complex.
</Tip>