exp selects a checked-in run configuration. Positional arguments after the experiment narrow by eval ID prefix. Put agents, models, and flags in experiments/, not in temporary CLI arguments.
Commands
npx niceeval exp [path] [id-prefix]
Runs the experiment configuration selected by its ID path. Each configuration’s
evals determines which evals it covers.npx niceeval debug <experiment> <eval>
Shows the paired Sandbox and Plugin lifecycle command plan without running an eval.
npx niceeval init
Creates an empty
evals/ directory and a minimal niceeval.config.ts. It also writes or refreshes the NiceEval guidance block in the project’s agent instructions.npx niceeval list
Discovers and prints every eval without running one.
npx niceeval clean
Deletes the temporary Record directory left by one named writer. It never deletes a completed Run or Attempt.
npx niceeval show [--run]
Runs a terminal Report. Without a locator or
--run, it reads current-project results; an exact Attempt locator or Run ID audits history.npx niceeval view
Opens a web view for one fixed ReportExecution. End and rerun it to read an edited Record.
Output language
CLI and runtime copy is English. That affects terminal and runtime text only. It does not change machine fields in results JSON or translate LLM Judge prompts. The browserniceeval view has its own English and Chinese switch.
npx niceeval exp [path|config] [id-prefix...]
An eval filter appears only after an experiment selection.
npx niceeval weather does not run an eval. Use npx niceeval exp local weather or npx niceeval exp compare weather.npx niceeval exp list [experiment-prefix]
This command only discovers runnable Experiment configurations. It does not create an Invocation, acquire a lease, or start an Agent or Sandbox.
experimentId, description, agent, model, attempts, selected eval count, and labels. --json emits one niceeval.experiments document with each item’s complete selectedEvalIds.
Inspect the full paired Shell and lifecycle plan
shell() and command() calls expand to their actual commands. Ordinary callbacks are marked opaque.
When an Eval is in a Group, selecting only that member does not add its other Group members back to the plan, but the output still keeps one Group lifecycle wrapper. Human-readable output shows only COMMAND PLAN; --json emits one niceeval.debug-plan/v1 document.
In a TTY, the overview, Experiment, lane, slot, and each lifecycle step use separate rounded panels, just like show; the full tree is not enclosed in one outer panel. Each Shell, Plugin occurrence, and phase without a command or with work determined only at runtime gets its own panel:
\n. It preserves indentation, blank lines, terminal newlines, and long-line wrapping. Terminal control characters render as escaped text:
NO_COLOR, or a terminal that is too narrow, every panel falls back to an unframed heading and body. The content and order stay the same. --json contains no borders.
Within a lane, human output always orders Group before-slot work, physical enter, slots, physical exit, and Group after-slot work. Physical teardown and the Provider finalizer therefore appear after every slot that uses the instance.
A Sandbox materialize node shows the template owner, Provider, kind, and a safe configured locator. Exact means only that NiceEval displays the author’s configuration verbatim; it does not mean that an image tag, remote content, or final instance has been frozen. Credentials, query strings, and fragments are redacted from remote URLs. A Docker image is shown as Exact only when it uses a conservative credential-safe reference, optionally with a standard sha256 digest; URL, non-digest userinfo, or other unsafe syntax makes the entire locator Opaque, without emitting the original string.
Author-provided local paths, file: URLs, Dockerfile paths, and local directories make the entire locator Opaque, without emitting the path. Only the fixed author-base-dir label used when localSandbox.dir is omitted can be Exact. Environment values, credentials, stdin, build-argument values, and custom identities are not emitted.
E2B template names and Vercel snapshot IDs are arbitrary Provider-managed strings, so their complete locators are also Opaque. Human command plans render C0, C1, ESC, tab, and carriage-return characters as visible escape text at one terminal boundary, including template metadata and Shell lines. JSON keeps the structured values and relies on JSON string escaping.
debug does not run runtime setup, test, teardown, ensure, materialize, or finalizer work. It does not create an Invocation, Run, Record, lease, Sandbox, or build. It loads .env and trusted definitions, and evaluates the Experiment selection predicate. A Provider planner may still read files, invoke read-only CLIs, or query the Docker control plane and remote APIs.
The plan lists every Attempt configured by the Experiment as a candidate dispatch. A normal run can still skip a slot because of carry, early exit, or budget. debug accepts only its optional --json; any other run flag is a usage error.
--json has command-specific output: exp writes one ordered NDJSON event stream to stdout; exp --dry and debug each write one JSON plan document; show emits the host data and status for the same ReportExecution without opening a second data path.
Common flags
The CLI accepts only the flags below. Any other flag is an error and exits nonzero. There is no CLI flag or project-wide default for choosing a Sandbox provider. Declare a template-bearingSandboxLayer on an Eval or Experiment with one of these factories from niceeval/sandbox:
dockerImageSandbox({ image })dockerComposeSandbox({ file, workspaceService })dockerfileSandbox({ context })vercelSandbox({ snapshotId })e2bSandbox({ template })localSandbox()
list
exp
show and view are top-level commands. If exp show or exp view does not match an experiment, the CLI suggests the top-level command. A repository can still intentionally run an experiment with either valid ID.
Two outputs: human-readable text and --json
exp has two output forms. They change presentation, not selection, scheduling, Verdicts, Record commits, or exit codes:
view
show’s default Report and selection rules. Without a locator or --run, it reads every result whose identity still matches the current project from the default Record. An exact @<AttemptLocator> shows that Attempt’s default overview. --run reads named historical Runs. The internal host then projects the needed facts and executes one Report.
Without --host, view listens only on 127.0.0.1. Pass a concrete hostname or IP to opt into that listener, or use bare --host for all IPv4 interfaces. A wildcard listener prints a loopback URL followed by the usable interface URLs; --page is applied to every printed URL and only the first is opened automatically.
Non-loopback mode is an unauthenticated, unencrypted development server. Every network-reachable client can read the report, execution JSON, and downloads, so use it only on a network you trust. Startup prints the same warning. The server accepts only advertised Host authorities and read-only GET / HEAD requests; future write endpoints require a separate authentication and CSRF design.
A previous local execution can leave reference Members for slots that adopted an existing Attempt. These are adoption facts, not a rerun state. The same AnalysisSample does not reselect while a page is loading. --experiment narrows current-project results by complete ID. Use --record only when you intentionally read another Record root.
show
show is the terminal Report entry point for people and coding agents. It accepts no fuzzy positional selection. An exact @<AttemptLocator> shows that Attempt’s Evaluation identity, evaluation kind, four-state Verdict, assertions, and applicable Score by default.
Add --source, --execution, or --timing to inspect its dedicated evidence surface. --timing shows a compact phase tree by default. --timing=full adds interval identity, parent, and offset. These three surfaces are mutually exclusive and each requires one exact Attempt locator.
Without a locator or --run, show plans current-project identity and scans every published Run in the default Record. It includes only slots whose Experiment, Eval, attempt ordinal, evaluation kind, input identity, and config identity still match the current project. It does not retain only one Run by time. When no result matches, it shows an empty Sample. Stale results remain available through a complete --run ID.
Repeat --run to read only explicit historical Runs. Repeat --experiment <id> to narrow current-project results by complete Experiment ID; it cannot combine with --run. exp, show, and view use the same .niceeval/record by default when --record is absent.
project-current changes only Report selection. It does not migrate the Record or change its version.
After selection, show reads the Projection declared by the Report and forms one ReportExecution. Requested unavailable, migration-required, unsupported, partial, or invalid Attachment states remain visible. An unknown or damaged Attachment that the current page did not request does not affect it. show --json emits the same execution as niceeval.report-show/v1. See Viewing Results for the full contract.
view --out <directory> (or an explicit --run) prerenders the same ReportExecution as a self-contained static site. The target directory must not exist. NiceEval validates every page, download, and static asset path before publishing the complete directory in one rename. It does not give a user Report a Record or blob path.
The command completes the needed Projection and Report execution while it holds the sample handle. The returned ReportExecution no longer accesses the Record. A running view builds a new fixed execution on change. It replaces the current revision on success and retains the last-good execution on failure. Static export consumes only a completed execution.
--early-exit and --strict
--early-exit is off by default. With --attempts greater than one, NiceEval runs every attempt and reports the real pass rate. Add --early-exit only when you need to know whether an eval can pass and do not need its full distribution. If an experiment sets earlyExit: true, --no-early-exit turns it off.
--strict changes the Verdict for soft-threshold assertions. A failed soft threshold such as .atLeast(n) is recorded but does not ordinarily fail the eval. With --strict, it counts as failed. Use it in CI so an under-target assertion score cannot pass unnoticed.
Environment variables hold credentials only
Attempt count, timeout, concurrency, budget, and Judge model and endpoint are configuration. NiceEval reads them only from CLI flags, files underexperiments/, and niceeval.config.ts, in that precedence order. They do not have an environment-variable route.
Use environment variables for credentials, such as API keys and provider tokens, and terminal behavior such as NO_COLOR. Each Agent, Sandbox, or Judge reads only its named variable. NiceEval loads .env from the current working directory without overriding an already present environment variable. See Configuration and Environment Variables for the full list.
Exit codes
Use
--strict in CI when soft-threshold failures must also fail the gate.