> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiment matrix: compare agents and models with run matrices

> Use NiceEval experiments to run the same evals across multiple agents, models, and feature flags, then compare pass rate, cost, and latency.

Experiments compare multiple run configurations. Typical questions include: which coding agent has the higher pass rate on the same task set, whether a prompt change lowers cost, and how latency trades off against quality across models.

## Basic shape

**One experiment file = one configuration** (`one agent x one model`). `model` is a single string, not an array. To compare across models or agents, put multiple files in the same **experiment group directory** and keep everything else fixed:

```text theme={null}
experiments/
  compare-models/          # experiment group
    gpt-5.4.ts             #   one cell: pin model = gpt-5.4
    deepseek-v4-pro.ts     #   one cell: pin model = deepseek-v4-pro
```

```ts theme={null}
// experiments/compare-models/gpt-5.4.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../../adapter/adapter.ts";

export default defineExperiment({
  description: "gpt-5.4: comparison model",
  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
  model: "gpt-5.4",   // single string; copy the file and change this line for another model
  attempts: 2,
  earlyExit: true,
});
```

```bash theme={null}
npx niceeval exp compare-models   # renders a side-by-side report for every model in the group
```

Each configuration stays in its own file, which makes naming, diffing, and review straightforward. The directory structure itself explains which files belong to the same comparison.

By default, `niceeval show` and `view` compare Experiments in the current result scope; no extra grouping field is needed.

When one cell's result looks off and you need to reproduce it in isolation, run just that cell by its full id (`group/filename`) instead of the whole group:

```bash theme={null}
npx niceeval exp compare-models/gpt-5.4
```

If the group also has a shared-prefix variant like `gpt-5.4-mini.ts`, you don't need to spell out both ids — running the shared prefix picks them up as a family (the exact id `compare-models/gpt-5.4` still selects only `gpt-5.4.ts` on its own):

```bash theme={null}
npx niceeval exp compare-models/gpt
```

See [Write Experiments](/docs/tutorials/write-experiment) for the full `defineExperiment` field list and how `flags` flow into adapters.

## What this is good for

* Comparing different Adapters
* Comparing different models. Tier 1 is enough as long as the application exposes model choice and the value is forwarded through `ctx.model`
* Comparing prompts or feature flags. This requires Tier 3, because the application has to expose the variant as an experiment-selectable config and forward it through `flags` -> `ctx.flags`
* Comparing different Sandbox providers
* Comparing different runtime environment conditions (for example, whether a memory tool's binary is installed, or whether some state is pre-seeded). Write the environment difference into the `.setup()` / `.teardown()` hooks of the `sandbox` spec, one experiment file per variant. See [Sandbox providers · Lifecycle](/docs/tutorials/sandbox-providers)
* Measuring pass\@N for the same task

See [Tier](/docs/explanation/tier) for what Tier 1, Tier 2, and Tier 3 mean.

## Reading the results

Experiment output is typically shown per `(agent, model, eval)` cell:

```text theme={null}
api-validation   claude-code+zod-skill   pass@3 = 3/3 (100%)   mean 34s
api-validation   claude-code             pass@3 = 1/3 (33%)    mean 41s
```

Beyond pass rate, you should also compare mean time, tokens, cost, and failure types.

```bash theme={null}
npx niceeval show --experiment models
npx niceeval view --experiment models
```

Each Experiment uses its own `evals` selection to choose evals. The default current-project selection keeps every published slot whose identity still matches the current project. Use `--experiment <complete-id>` to narrow the current-project target before you compare it.

## Design advice

* Keep the eval set stable so the comparison does not mix in extra variables.
* Run multiple attempts per cell, especially for non-deterministic coding agents.
* Make the budget and concurrency explicit.
* Group failure modes instead of looking only at the total score.

## Relation to ordinary runs

`npx niceeval exp <experiment>` checks whether a batch of evals passes under a given configuration; an experiment group compares multiple configurations. Both use the same evals, adapters, assertions, and artifacts.