Skip to main content
Experiments compare multiple run configurations. Typical questions include: which coding agent has the higher pass rate on the same task set, whether a prompt change lowers cost, and how latency trades off against quality across models.

​
Basic shape

One experiment file = one configuration (one agent x one model). model is a single string, not an array. To compare across models or agents, put multiple files in the same experiment group directory and keep everything else fixed:
Each configuration stays in its own file, which makes naming, diffing, and review straightforward. The directory structure itself explains which files belong to the same comparison. By default, niceeval show and view compare Experiments in the current result scope; no extra grouping field is needed. When one cell’s result looks off and you need to reproduce it in isolation, run just that cell by its full id (group/filename) instead of the whole group:
If the group also has a shared-prefix variant like gpt-5.4-mini.ts, you don’t need to spell out both ids — running the shared prefix picks them up as a family (the exact id compare-models/gpt-5.4 still selects only gpt-5.4.ts on its own):
See Write Experiments for the full defineExperiment field list and how flags flow into adapters.

​
What this is good for

  • Comparing different Adapters
  • Comparing different models. Tier 1 is enough as long as the application exposes model choice and the value is forwarded through ctx.model
  • Comparing prompts or feature flags. This requires Tier 3, because the application has to expose the variant as an experiment-selectable config and forward it through flags -> ctx.flags
  • Comparing different Sandbox providers
  • Comparing different runtime environment conditions (for example, whether a memory tool’s binary is installed, or whether some state is pre-seeded). Write the environment difference into the .setup() / .teardown() hooks of the sandbox spec, one experiment file per variant. See Sandbox providers · Lifecycle
  • Measuring pass@N for the same task
See Tier for what Tier 1, Tier 2, and Tier 3 mean.

​
Reading the results

Experiment output is typically shown per (agent, model, eval) cell:
Beyond pass rate, you should also compare mean time, tokens, cost, and failure types.
Each Experiment uses its own evals selection to choose evals. The default current-project selection keeps every published slot whose identity still matches the current project. Use --experiment <complete-id> to narrow the current-project target before you compare it.

​
Design advice

  • Keep the eval set stable so the comparison does not mix in extra variables.
  • Run multiple attempts per cell, especially for non-deterministic coding agents.
  • Make the budget and concurrency explicit.
  • Group failure modes instead of looking only at the total score.

​
Relation to ordinary runs

npx niceeval exp <experiment> checks whether a batch of evals passes under a given configuration; an experiment group compares multiple configurations. Both use the same evals, adapters, assertions, and artifacts.