> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# NiceEval CLI: Commands, Flags, and Exit Codes
> NiceEval CLI reference for the exp, debug, show, view, init, list, and clean commands, including result drill-down, experiments, eval filters, concurrency, budgets, and CI output.
The [NiceEval](https://niceeval.com/) CLI discovers, runs, and reviews evals. Running evals follows an experiment-first model: **`exp` selects a checked-in run configuration**. Positional arguments after the experiment narrow by eval ID prefix. Put agents, models, and flags in `experiments/`, not in temporary CLI arguments.
## Commands
Runs the experiment configuration selected by its ID path. Each configuration's `evals` determines which evals it covers.
Shows the paired Sandbox and Plugin lifecycle command plan without running an eval.
Creates an empty `evals/` directory and a minimal `niceeval.config.ts`. It also writes or refreshes the NiceEval guidance block in the project's agent instructions.
Discovers and prints every eval without running one.
Deletes the temporary Record directory left by one named writer. It never deletes a completed Run or Attempt.
Runs a terminal Report. Without a locator or `--run`, it reads current-project results; an exact Attempt locator or Run ID audits history.
Opens a web view for one fixed ReportExecution. End and rerun it to read an edited Record.
## Output language
CLI and runtime copy is English. That affects terminal and runtime text only. It does not change machine fields in results JSON or translate LLM Judge prompts. The browser `niceeval view` has its own English and Chinese switch.
## `npx niceeval exp [path|config] [id-prefix...]`
```bash theme={null}
# Run every experiment under experiments/
npx niceeval exp
# Run a group of experiments in a nested directory
npx niceeval exp models
# Run one configuration
npx niceeval exp models/openai/gpt-5.4
# Run every configuration in a nested directory
npx niceeval exp models/openai
# Within that group, run only evals whose IDs begin with weather
npx niceeval exp models weather
```
An eval filter appears only after an experiment selection. `npx niceeval weather` does not run an eval. Use `npx niceeval exp local weather` or `npx niceeval exp compare weather`.
The first positional argument is an experiment ID, directory path, or filename prefix. Remaining positional arguments narrow by eval ID prefix.
### `npx niceeval exp list [experiment-prefix]`
This command only discovers runnable Experiment configurations. It does not create an Invocation, acquire a lease, or start an Agent or Sandbox.
```bash theme={null}
npx niceeval exp list
npx niceeval exp list compare/codex
npx niceeval exp list --json
```
The default output includes each configuration's `experimentId`, description, agent, model, attempts, selected eval count, and labels. `--json` emits one `niceeval.experiments` document with each item's complete `selectedEvalIds`.
### Inspect the full paired Shell and lifecycle plan
```bash theme={null}
npx niceeval debug compare/codex memory/commit0
npx niceeval debug compare/codex memory/commit0 --json
```
Both selectors must resolve uniquely. Exact IDs take precedence; otherwise, you can use a unique prefix. The Eval can match only within the Experiment's selected scope. Zero or multiple matches list the exact candidates.
The plan includes Plugin lifecycle, author prepare, Agent ensure/setup/teardown, test, cleanup, and the Provider finalizer. Statically declared `shell()` and `command()` calls expand to their actual commands. Ordinary callbacks are marked `opaque`.
When an Eval is in a Group, selecting only that member does not add its other Group members back to the plan, but the output still keeps one Group lifecycle wrapper. Human-readable output shows only `COMMAND PLAN`; `--json` emits one `niceeval.debug-plan/v1` document.
In a TTY, the overview, Experiment, lane, slot, and each lifecycle step use separate rounded panels, just like `show`; the full tree is not enclosed in one outer panel. Each Shell, Plugin occurrence, and phase without a command or with work determined only at runtime gets its own panel:
```text theme={null}
╭─ sandbox.materialize ─────────────────────────────────────── OPAQUE ─╮
│ position: lane eval-group:group · physical lifecycle template enter │
│ owner: provider:docker │
│ template: docker:image │
│ template owner: experiment:suite/one │
│ configured locator: exact · image="node@sha256:cd849..." │
│ reason: provider materialization is a runtime operation │
╰──────────────────────────────────────────────────────────────────────╯
╭─ sandbox.prepare ──────────────────────────────────────────── EXACT ─╮
│ position: lane eval-group:group · slot group/first #0 │
│ owner: eval:group/first │
│ command: shell "printf fixture-ready" │
╰──────────────────────────────────────────────────────────────────────╯
```
A multiline Shell renders as a guttered code block in its own panel instead of a one-line string with `\n`. It preserves indentation, blank lines, terminal newlines, and long-line wrapping. Terminal control characters render as escaped text:
```text theme={null}
╭─ sandbox.prepare ──────────────────────────────────────────────────── EXACT ─╮
│ position: lane eval:group/first · slot group/first #0 │
│ owner: eval:group/first │
│ command: shell · 5 lines │
│ │ set -eu │
│ │ pnpm install │
│ │ │
│ │ pnpm test │
│ │ │
╰──────────────────────────────────────────────────────────────────────────────╯
```
With a pipe, `NO_COLOR`, or a terminal that is too narrow, every panel falls back to an unframed heading and body. The content and order stay the same. `--json` contains no borders.
Within a lane, human output always orders Group before-slot work, physical enter, slots, physical exit, and Group after-slot work. Physical teardown and the Provider finalizer therefore appear after every slot that uses the instance.
A Sandbox materialize node shows the template owner, Provider, kind, and a safe configured locator. `Exact` means only that NiceEval displays the author's configuration verbatim; it does not mean that an image tag, remote content, or final instance has been frozen. Credentials, query strings, and fragments are redacted from remote URLs. A Docker image is shown as `Exact` only when it uses a conservative credential-safe reference, optionally with a standard `sha256` digest; URL, non-digest userinfo, or other unsafe syntax makes the entire locator `Opaque`, without emitting the original string.
Author-provided local paths, `file:` URLs, Dockerfile paths, and local directories make the entire locator `Opaque`, without emitting the path. Only the fixed `author-base-dir` label used when `localSandbox.dir` is omitted can be `Exact`. Environment values, credentials, stdin, build-argument values, and custom identities are not emitted.
E2B template names and Vercel snapshot IDs are arbitrary Provider-managed strings, so their complete locators are also `Opaque`. Human command plans render C0, C1, ESC, tab, and carriage-return characters as visible escape text at one terminal boundary, including template metadata and Shell lines. JSON keeps the structured values and relies on JSON string escaping.
`debug` does not run runtime setup, test, teardown, ensure, materialize, or finalizer work. It does not create an Invocation, Run, Record, lease, Sandbox, or build. It loads `.env` and trusted definitions, and evaluates the Experiment selection predicate. A Provider planner may still read files, invoke read-only CLIs, or query the Docker control plane and remote APIs.
The plan lists every Attempt configured by the Experiment as a candidate dispatch. A normal run can still skip a slot because of carry, early exit, or budget. `debug` accepts only its optional `--json`; any other run flag is a usage error.
`--json` has command-specific output: `exp` writes one ordered NDJSON event stream to stdout; `exp --dry` and `debug` each write one JSON plan document; `show` emits the host data and status for the same ReportExecution without opening a second data path.
## Common flags
The CLI accepts only the flags below. Any other flag is an error and exits nonzero. There is no CLI flag or project-wide default for choosing a Sandbox provider. Declare a template-bearing `SandboxLayer` on an Eval or Experiment with one of these factories from `niceeval/sandbox`:
* `dockerImageSandbox({ image })`
* `dockerComposeSandbox({ file, workspaceService })`
* `dockerfileSandbox({ context })`
* `vercelSandbox({ snapshotId })`
* `e2bSandbox({ template })`
* `localSandbox()`
| Flag | Type | Meaning |
| ---------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--agent` | string | Not supported for experiment runs. Create or copy an `experiments/` file to change the agent. |
| `--model` | string | Not supported for experiment runs. Create or copy an experiment file to change the model. |
| `--attempts` | number | Number of attempts for each eval, commonly for pass\@N. |
| `--max-concurrency` | number | Maximum number of evals to run at once. |
| `--max-build-concurrency` | number | Maximum simultaneous Sandbox image lookups or builds. This is independent of eval concurrency and defaults to 2. |
| `--timeout` | number | Per-attempt timeout in milliseconds. Precedence is `--timeout` → experiment → Eval (`defineEval({ timeoutMs })`) → `niceeval.config.ts`; absent values leave the attempt without a deadline. |
| `--budget` | number | Budget limit for the complete run in US dollars. |
| `--keep-sandbox` | boolean | `exp` only. Keeps the Sandbox for failed or errored attempts; `--keep-sandbox=all` also keeps passed attempts. Review or destroy it later with `niceeval sandbox` commands. |
| `--all` | boolean | `sandbox stop` only. Destroys all retained Sandboxes. |
| `--window` | string | `sandbox diff` only. Shows one send window, such as `--window turn2`; omit it for all windows. |
| `--path` | string | `sandbox diff` only. Shows one file patch; omit it for all files in the selected window. |
| `--leave-running` | boolean | `sandbox enter` only. Leaves the retained Sandbox running after the shell exits. |
| `--orphans` | boolean | `sandbox list` only. Reads unowned Sandboxes left by a hard stop; it does not clean them. |
| `--tag` | string | Runs only evals with the tag declared by `defineEval`. |
| `--junit` | string | Writes an additional JUnit XML file for CI. |
| `--json` | boolean | `exp` writes one ordered NDJSON stream to stdout. `exp --dry` and `debug` each write one JSON plan document. `show` emits the `niceeval.report-show/v1` document for the already computed ReportExecution without opening a second data path. |
| `--smoke` | boolean | `docker profile doctor` only. Starts a constrained DinD container and runs an inner container. |
| `--out` | string | `view` only. Exports a static report site to this directory. |
| `--port` | number | `view` only. Chooses the local server port. |
| `--host` | string | `view` only. Chooses the listening address. Omit it for `127.0.0.1`; bare `--host` listens on `0.0.0.0`. Non-loopback listeners have no authentication or TLS. |
| `--record` | string | For `show`, `view`, `accept`, and `sandbox enter`, `list`, or `stop`: selects the actual Record root. Use it with `show` or `view` only to read another Record root. The CLI does not append `.niceeval/record` or another suffix. |
| `--run` | string | Repeat for `show` or `view` to add one explicit complete Run ID. Duplicate identities are removed. |
| `--report` | string | `show` and `view` only. Replaces their shared default Report with the file's default `defineReport(...)` export. |
| `--page` | string | `show` and `view` only. Selects a planned page. An unknown page ID is a usage error and lists available page IDs. |
| `--teardown` | boolean | `exp` only. Completes interrupted experiment-level teardown for each selected Experiment without running setup or attempts. |
| `--dry` | boolean | Prints the eval × run-configuration plan without running it. With `--json`, it emits one JSON plan document. |
| `--force` | boolean | `sandbox prune` only. Also destroys unverified instances. `exp` rejects it; use `--rerun` or `--rerun all` to run failed or all slots again. |
| `--rerun` | boolean | `exp` only. Reruns failed slots when bare or `failed`, or all slots with `all`, without changing the long-lived fingerprint. |
| `--strict` | boolean | Counts a failed soft threshold as a failed eval. Recommended for CI. |
| `--early-exit` / `--no-early-exit` | boolean | Stops remaining attempts for an eval after its first pass. It is off by default. |
| `--open` / `--no-open` | boolean | `view` only. Opens a browser automatically after startup by default. |
| `--help` | boolean | Prints help and exits. |
| `--version` | boolean | Prints the installed NiceEval version and exits. |
| `--experiment` | string | Repeat for `show` and `view` to narrow current-project results by complete Experiment ID. It cannot combine with explicit `--run`. |
## `list`
```bash theme={null}
npx niceeval list
```
Use this command to verify eval discovery, IDs, and configuration loading.
## `exp`
```bash theme={null}
npx niceeval exp compare-models
npx niceeval exp compare-models weather-tool
```
Run a named experiment to compare agents, models, or flags. The second positional argument and later arguments filter eval IDs by prefix.
`show` and `view` are top-level commands. If `exp show` or `exp view` does not match an experiment, the CLI suggests the top-level command. A repository can still intentionally run an experiment with either valid ID.
## Two outputs: human-readable text and `--json`
`exp` has two output forms. They change presentation, not selection, scheduling, Verdicts, Record commits, or exit codes:
```bash theme={null}
# Human-readable output: a dynamic dashboard in a TTY, or an append-only text stream in CI
npx niceeval exp compare
# Machine output: one ordered NDJSON stream on stdout
npx niceeval exp compare --json
# CI gate: exit status plus JUnit. For a JSON summary, run show after the Invocation with an explicit selection.
npx niceeval exp compare --strict --junit ./artifacts/niceeval-junit.xml
```
See [Runner · Reporter](/docs/explanation/runner#reporter) for choosing an output. See [Coding Agent Feedback Loop](/docs/tutorials/agent-feedback-loop) for the full AI workflow, and [CI Integration](/docs/tutorials/ci-integration) for GitHub Actions, exit codes, and JUnit.
## `view`
```bash theme={null}
npx niceeval view
npx niceeval view --run 01H...
npx niceeval view @01H...
npx niceeval view --host 192.168.0.199
```
This opens the local Record viewer. It shares `show`'s default Report and selection rules. Without a locator or `--run`, it reads every result whose identity still matches the current project from the default Record. An exact `@` shows that Attempt's default overview. `--run` reads named historical Runs. The internal host then projects the needed facts and executes one Report.
Without `--host`, `view` listens only on `127.0.0.1`. Pass a concrete hostname or IP to opt into that listener, or use bare `--host` for all IPv4 interfaces. A wildcard listener prints a loopback URL followed by the usable interface URLs; `--page` is applied to every printed URL and only the first is opened automatically.
Non-loopback mode is an unauthenticated, unencrypted development server. Every network-reachable client can read the report, execution JSON, and downloads, so use it only on a network you trust. Startup prints the same warning. The server accepts only advertised Host authorities and read-only `GET` / `HEAD` requests; future write endpoints require a separate authentication and CSRF design.
A previous local execution can leave `reference` Members for slots that adopted an existing Attempt. These are adoption facts, not a rerun state. The same `AnalysisSample` does not reselect while a page is loading. `--experiment` narrows current-project results by complete ID. Use `--record` only when you intentionally read another Record root.
## `show`
```bash theme={null}
npx niceeval show
npx niceeval show --run 01H...
npx niceeval show --run 01H... --run 01J...
npx niceeval show @01H...
npx niceeval show @01H... --source
npx niceeval show @01H... --execution
npx niceeval show @01H... --timing
npx niceeval show @01H... --timing=full
npx niceeval show --run 01H... --page attempt-0123456789ABCDEFGHJKMNPQRS
npx niceeval show --run 01H... --report ./reports/summary.ts --page overview
```
`show` is the terminal Report entry point for people and coding agents. It accepts no fuzzy positional selection. An exact `@` shows that Attempt's Evaluation identity, evaluation kind, four-state Verdict, assertions, and applicable Score by default.
Add `--source`, `--execution`, or `--timing` to inspect its dedicated evidence surface. `--timing` shows a compact phase tree by default. `--timing=full` adds interval identity, parent, and offset. These three surfaces are mutually exclusive and each requires one exact Attempt locator.
Without a locator or `--run`, `show` plans current-project identity and scans every published Run in the default Record. It includes only slots whose Experiment, Eval, attempt ordinal, evaluation kind, input identity, and config identity still match the current project. It does not retain only one Run by time. When no result matches, it shows an empty Sample. Stale results remain available through a complete `--run` ID.
Repeat `--run` to read only explicit historical Runs. Repeat `--experiment ` to narrow current-project results by complete Experiment ID; it cannot combine with `--run`. `exp`, `show`, and `view` use the same `.niceeval/record` by default when `--record` is absent.
`project-current` changes only Report selection. It does not migrate the Record or change its version.
After selection, `show` reads the Projection declared by the Report and forms one `ReportExecution`. Requested `unavailable`, `migration-required`, `unsupported`, `partial`, or `invalid` Attachment states remain visible. An unknown or damaged Attachment that the current page did not request does not affect it. `show --json` emits the same execution as `niceeval.report-show/v1`. See [Viewing Results](/docs/tutorials/viewing-results) for the full contract.
`view --out ` (or an explicit `--run`) prerenders the same ReportExecution as a self-contained static site. The target directory must not exist. NiceEval validates every page, download, and static asset path before publishing the complete directory in one rename. It does not give a user Report a Record or blob path.
The command completes the needed Projection and Report execution while it holds the sample handle. The returned ReportExecution no longer accesses the Record. A running `view` builds a new fixed execution on change. It replaces the current revision on success and retains the last-good execution on failure. Static export consumes only a completed execution.
## `--early-exit` and `--strict`
`--early-exit` is off by default. With `--attempts` greater than one, NiceEval runs every attempt and reports the real pass rate. Add `--early-exit` only when you need to know whether an eval can pass and do not need its full distribution. If an experiment sets `earlyExit: true`, `--no-early-exit` turns it off.
`--strict` changes the Verdict for soft-threshold assertions. A failed soft threshold such as `.atLeast(n)` is recorded but does not ordinarily fail the eval. With `--strict`, it counts as failed. Use it in CI so an under-target assertion score cannot pass unnoticed.
## Environment variables hold credentials only
Attempt count, timeout, concurrency, budget, and Judge model and endpoint are configuration. NiceEval reads them only from CLI flags, files under `experiments/`, and `niceeval.config.ts`, in that precedence order. They do not have an environment-variable route.
Use environment variables for credentials, such as API keys and provider tokens, and terminal behavior such as `NO_COLOR`. Each Agent, Sandbox, or Judge reads only its named variable. NiceEval loads `.env` from the current working directory without overriding an already present environment variable. See [Configuration and Environment Variables](/docs/tutorials/configuration) for the full list.
## Exit codes
| Exit code | Meaning |
| --------- | -------------------------------------------------------------------------------------- |
| `0` | Every eval passed, or a command such as `list`, `init`, or `clean` completed normally. |
| `1` | At least one eval failed or errored, or command usage was invalid. |
| `2` | The runtime crashed with an uncaught exception caught by the CLI boundary. |
| `130` | The process received an interrupt signal such as Ctrl-C or SIGTERM. |
Use `--strict` in CI when soft-threshold failures must also fail the gate.