Data-driven testing (dataset fan-out): run one eval suite across many rows of data

Data-driven testing (dataset fan-out) is a good fit when many test cases share the same structure and only the inputs change. Typical examples are SQL generation, intent classification, retrieval QA, and tool selection.

How fan-out works

When there is no external business ID, a .eval.ts file exports an array by default:

import { defineEval } from "niceeval";
import { equals } from "niceeval/expect";

const rows = [
  { task: "Count users", prompt: "Count all users", sql: "SELECT COUNT(*) FROM users;" },
  { task: "Recent orders", prompt: "Find recent orders", sql: "SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;" },
];

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.sql));
    },
  }),
);

Generated IDs

If the file is evals/sql.eval.ts, the generated IDs are:

sql/0000
sql/0001

The numeric suffix is zero-padded so IDs stay stable and easy to filter. When the data source already carries a stable case, issue, or benchmark ID, export a keyed record by default instead:

export default Object.fromEntries(
  rows.map((row) => [
    row.issueId,
    defineEval({
      description: row.title,
      async test(t) {
        await t.send(row.prompt);
        t.succeeded();
      },
    }),
  ]),
);

If the file is evals/swelancer.eval.ts and the key is 15193, the ID is swelancer/15193. A key must be a non-empty path segment: it cannot be . or .., and it cannot contain /, \, or control characters. NiceEval discovers cases in key lexicographic order, so a change in the data source’s return order never changes run order.

Loading from YAML and JSON

import { loadYaml } from "niceeval/loaders";
import { z } from "zod";

const SqlCases = z.object({
  cases: z.array(z.object({ task: z.string(), prompt: z.string(), sql: z.string() })),
});
const doc = await loadYaml("evals/data/sql-cases.yaml", (value) => SqlCases.parse(value));
const rows = doc.cases;

cases:
  - task: Count users
    prompt: Count all rows in the users table
    sql: SELECT COUNT(*) FROM users;

loadYaml and loadJson require a decoder. The unvalidated dynamic value exists only at the decoder input; after validation, the return value is strongly typed data that the Eval can use directly.

Filtering dataset evals

# Run the whole dataset
npx niceeval exp local sql

# Run only the first array case
npx niceeval exp local sql/0000

# Run a keyed case
npx niceeval exp local swelancer/15193

Datasets vs separate files

Use a dataset
Use separate files

The cases have exactly the same structure. Only the input and expected output change.

Datasets are good for broad horizontal coverage. Separate eval files are better when the behavior itself is complex.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Data-driven testing (dataset fan-out): run one eval suite across many rows of data

How fan-out works

Generated IDs

Loading from YAML and JSON

Filtering dataset evals

Datasets vs separate files

​How fan-out works

​Generated IDs

​Loading from YAML and JSON

​Filtering dataset evals

​Datasets vs separate files

How fan-out works

Generated IDs

Loading from YAML and JSON

Filtering dataset evals

Datasets vs separate files