Skip to main content
You only wanted to reproduce one failing eval, but moved too fast and ran all 200 evals in the whole experiment group, spending tens of dollars. When the repository has several groups of Experiments and many evals, first confirm what this run will actually execute, then start it for real. Do not guess IDs from file names, and do not launch a batch that may be the wrong selection.

List runnable Experiments

First see which Experiments the project has, and confirm the ID you want:
Each line shows the Experiment ID, description, Agent, model, number of Attempts, number of selected evals, and labels. exp list only reads configuration; it does not start Sandboxes, call models, or write results. When a script needs the complete list, add --json:

Run only part of it

The first positional argument selects the Experiment and can be an ID, a directory, or a file name prefix. Later arguments narrow further by eval ID prefix:
An exact Experiment ID wins over a prefix match. The directory part must match exactly. Eval prefixes can only narrow within the evals this Experiment already selects; they cannot add evals outside its scope. To run only evals with a certain tag, use --tag:
Positional arguments and --tag only affect this run. To change which evals an Experiment runs long-term, change its evals.

See the plan with --dry

Before really running, add --dry to the same command:
The plan lists each eval, run configuration, and Attempt number, and marks whether each Attempt executes again or reuses the last result. --dry does not create a Run, start Sandboxes, call models, or write JUnit. Check line by line: whether the number of evals is right, whether the model is the one you want, how many times each eval runs, and which ones reuse old results. For the rules on reuse and rerun, see Rerun Only the Evals Affected by a Change. When a script needs to read the plan, add --json:
After confirming, drop --dry and run the same command. After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view.

What to do when nothing is selected

  • When no Experiment is selected, the CLI lists the available directories and suggests running exp <path> --dry first.
  • When an Experiment is selected but not a single eval is, the CLI reports No evals selected and does not fall back to running the whole Experiment.
  • To see the preparation steps and their order in the Sandbox, rather than which evals run, see Check How the Sandbox Will Be Prepared Before Running.