Data-Driven Tests: Generate Many Evals from One Dataset

You are evaluating a SQL generation feature and have 80 “question + expected result” samples. The check logic is the same for each: send the question, run the SQL, compare the result. If you write one file per sample, changing the check logic once means changing it in 80 places. Data-driven tests let you write the logic once and generate many evals from a dataset. Large sets of same-shaped tests such as intent classification, retrieval QA, and tool selection are a good fit.

Export a set of evals from one file

Put the samples in an array, use map to generate a defineEval for each row, and default-export the whole array. The check logic is written once, and changing it applies to every sample:

import { defineEval } from "niceeval";
import { equals } from "niceeval/expect";

const rows = [
  { task: "Count users", prompt: "Count all users", sql: "SELECT COUNT(*) FROM users;" },
  { task: "Recent orders", prompt: "Find recent orders", sql: "SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;" },
];

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, equals(row.sql));
    },
  }),
);

Confirm each eval’s ID

When you export an array, NiceEval numbers each eval by position. If the file is evals/sql.eval.ts, the generated IDs are:

sql/0000
sql/0001

The numbers are zero-padded to a fixed width, which makes them easy to reference and filter by prefix. Note that when the array order changes, the same sample’s ID changes with it. When the data source already has stable case, issue, or benchmark IDs, export an object instead and use those IDs as keys:

export default Object.fromEntries(
  rows.map((row) => [
    row.issueId,
    defineEval({
      description: row.title,
      async test(t) {
        await t.send(row.prompt);
        t.succeeded();
      },
    }),
  ]),
);

When the file is evals/swelancer.eval.ts and the key is 15193, the ID is swelancer/15193. A key must be one non-empty path segment: it cannot be . or .., and cannot contain /, \\, or control characters. NiceEval sorts keys lexicographically, so when the data source returns them in a different order, the run order and IDs stay the same. Before running, use pnpm exec niceeval exp local sql --dry to confirm which IDs were generated and how many there are.

Read a dataset from YAML or JSON

As samples grow, a separate data file is easier to maintain. Read it with loadYaml or loadJson, passing a validation function:

import { loadYaml } from "niceeval/loaders";
import { z } from "zod";

const SqlCases = z.object({
  cases: z.array(z.object({ task: z.string(), prompt: z.string(), sql: z.string() })),
});
const doc = await loadYaml("evals/data/sql-cases.yaml", (value) => SqlCases.parse(value));
const rows = doc.cases;

cases:
  - task: Count users
    prompt: Count all rows in the users table
    sql: SELECT COUNT(*) FROM users;

loadYaml and loadJson require a validation function (decoder). The file content only appears as an unknown type inside the validation function; once validation passes, the return value is typed data you can use directly. When the data file has the wrong format, the validation function throws, and this eval file fails to load.

Run only part of the dataset

Write an eval ID prefix after the Experiment to run only some of them:

# Run the whole dataset
pnpm exec niceeval exp local sql

# Run only the first entry in the array
pnpm exec niceeval exp local sql/0000

# Run one entry from an object export
pnpm exec niceeval exp local swelancer/15193

After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view. Each sample is an independent eval, and the failing ones are listed separately.

Choose a dataset or separate files

Use a dataset
Use separate files

Every sample follows exactly the same flow; only the input and expected output differ.

A dataset suits covering one feature with many similar samples; separate eval files suit scenarios with complex flows.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Data-Driven Tests: Generate Many Evals from One Dataset

Export a set of evals from one file

Confirm each eval’s ID

Read a dataset from YAML or JSON

Run only part of the dataset

Choose a dataset or separate files