> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Confirm What Runs and How Many Times Before Running

> List which Experiments exist, narrow the scope with ID prefixes and tags, then use --dry to see which evals this run executes and how many times, and confirm before spending money.

You only wanted to reproduce one failing eval, but moved too fast and ran all 200 evals in the whole experiment group, spending tens of dollars.

When the repository has several groups of Experiments and many evals, first confirm what this run will actually execute, then start it for real. Do not guess IDs from file names, and do not launch a batch that may be the wrong selection.

## List runnable Experiments

First see which Experiments the project has, and confirm the ID you want:

```sh theme={null}
pnpm exec niceeval exp list
pnpm exec niceeval exp list compare/codex
```

Each line shows the Experiment ID, description, Agent, model, number of Attempts, number of selected evals, and labels. `exp list` only reads configuration; it does not start Sandboxes, call models, or write results.

When a script needs the complete list, add `--json`:

```sh theme={null}
pnpm exec niceeval exp list --json > experiments.json
```

## Run only part of it

The first positional argument selects the Experiment and can be an ID, a directory, or a file name prefix. Later arguments narrow further by eval ID prefix:

```sh theme={null}
pnpm exec niceeval exp compare/codex memory/commit0
```

An exact Experiment ID wins over a prefix match. The directory part must match exactly. Eval prefixes can only narrow within the evals this Experiment already selects; they cannot add evals outside its scope.

To run only evals with a certain tag, use `--tag`:

```sh theme={null}
pnpm exec niceeval exp compare --tag smoke
```

Positional arguments and `--tag` only affect this run. To change which evals an Experiment runs long-term, change its `evals`.

## See the plan with `--dry`

Before really running, add `--dry` to the same command:

```sh theme={null}
pnpm exec niceeval exp compare/codex memory/commit0 --dry
```

The plan lists each eval, run configuration, and Attempt number, and marks whether each Attempt executes again or reuses the last result. `--dry` does not create a Run, start Sandboxes, call models, or write JUnit.

Check line by line: whether the number of evals is right, whether the model is the one you want, how many times each eval runs, and which ones reuse old results. For the rules on reuse and rerun, see [Rerun Only the Evals Affected by a Change](/docs/tutorials/rerun-and-cache).

When a script needs to read the plan, add `--json`:

```sh theme={null}
pnpm exec niceeval exp compare/codex memory/commit0 --dry --json > plan.json
```

After confirming, drop `--dry` and run the same command. After the run, view results in the terminal with `pnpm exec niceeval show`, or in the browser with `pnpm exec niceeval view`.

## What to do when nothing is selected

* When no Experiment is selected, the CLI lists the available directories and suggests running `exp <path> --dry` first.
* When an Experiment is selected but not a single eval is, the CLI reports `No evals selected` and does not fall back to running the whole Experiment.
* To see the preparation steps and their order in the Sandbox, rather than which evals run, see [Check How the Sandbox Will Be Prepared Before Running](/docs/tutorials/debug-lifecycle-plan).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.