niceeval exp. There are three ways to read results: scripts and CI use niceeval query to get JSON, people and coding agents use niceeval show to drill down in a terminal, and niceeval view is for review in a browser.
All three read the same run record, .niceeval/record.sqlite. At the start of every read, NiceEval fixes a visibility boundary: results already written at that moment are visible, and anything written afterward does not leak into this read. Attempts already written are readable even while their Run is still running.
When you pipe the output of niceeval exp, stdout still shows progress. Each line carries the Experiment, the Eval, and the Attempt number. The same Attempt prints at most once per second, and nothing is repeated when the content has not changed. Run directly in a terminal, it shows a live panel. Add --json when a script needs structured results.
Pass --experiment repeatedly to run several Experiments at once, for example niceeval exp --experiment baseline --experiment memory --max-concurrency 4. These Experiments share the concurrency limit and budget within the same run. An ID given twice runs only once. If any ID does not exist, the whole command fails before execution starts. Once you use --experiment, positional arguments filter Evals.
--dry reuse states
niceeval exp <experiment> --dry reads the plan without executing evals. Each row is an Experiment and Eval pair; the numbers in reused 0 and gap 0:... are zero-based Attempt ordinals.
The
verdict after source is the historical Verdict: passed means passed, failed means an assertion failed, errored means execution errored, and skipped means skipped.
For errored and skipped, use the gap reason to decide whether to rerun; inspect details with niceeval show @<locator>.
not-recorded, migration-required, unsupported, and invalid mean the Verdict attachment was not recorded, needs migration, is unsupported, or has invalid contents; none means passed.
First collect evidence with niceeval show @<locator> --assertions. Run niceeval migrate only when migration is explicitly required; preserve and report unsupported or invalid records.
--dry gap codes
The table lists every reason in gap <ordinal>:<code>. Here, “rerun” means checking cost and selection first, then running niceeval exp <experiment> <eval-prefix>; this preserves historical results.
When locked also appears, wait for the lock holder to finish before assessing the gap.
--dry difference summaries
changed, added, and removed mean changed, added, and removed inputs respectively; review the input before deciding to execute. unknown means comparison is unavailable; inspect the diagnostics in --dry --json.
config: identifies configuration, plan:physical the execution plan, plugins: lifecycle Plugins, and source: and data: source code and managed data respectively.
Complete structured summaries show specific values. When a summary is incomplete, summary sha256:<short-hash> distinguishes the saved summaries on the two sides; it is not the digest of the full execution identity and cannot recover omitted fields.
If two incomplete summaries happen to be identical, comparison unavailable appears once with the reason comparison is unavailable; it does not mean the inputs are identical.
For more evidence, run niceeval exp <experiment> --dry --json. --dry does not modify historical manifests, and summary readability does not change reuse decisions.
niceeval query
query uses the versioned JSON request protocol niceeval.query/v3. Each fixed read question is called an operation. A successful result and a failure defined by the protocol each write exactly one JSON document to stdout. Invalid arguments return a failure document with exit code 2. When the process fails before it can produce a document, the error goes only to stderr.
discoverlists the operation IDs the current version supports. It does not read the run record, so it accepts no--record, and it does not return the request field structure.explainvalidates a complete request and explains which operation it will run and which requests you can send next.runexecutes the request and returns that operation’s complete result.
discover to confirm which operations the current version has. Then write the request from the minimal requests on this page or in Viewing results, and validate it with explain. Do not guess fields, and do not build another query language from SQL, JSON paths, or formulas.
Paged results carry a continuation token. The token remembers the operation, the request content, the record file being read, and the visibility boundary of this read. If any of them changes, query does not continue reading. It returns a restart correction, which means the caller must start again from discover or by resending the request.
Compare Assertions across Experiments
—.
A * marks rows whose scores or decisions differ between columns.
The operation is experiments.assertions, with experimentIds and an optional evalIdPrefix path prefix.
Requests accept 1–32 IDs, deduplicate and sort them, and fail if any Experiment is missing.
matrix.evals[] contains columns, rows, and per-column unlabeled counts.
Duplicate identities within an Attempt retain all cell entries[] without overwriting each other. Unreadable assertion facts retain not-recorded, unsupported, or failed in the cell; these states are not shown as missing entries.
Limits are 100 Evals, 100 columns per Eval, 1000 total rows, 10000 cells, and 450000 UTF-8 bytes for the Eval JSON array.
Each duplicate entry consumes one cell. Limited results use matrix.state: "partial" and report omitted Eval, row, and column counts in matrix.omitted.
--entry, --compact, and --full select a single Attempt’s details and cannot be combined with a matrix.
Request and result fields for common operations
The table lists fields commonly used to read run results. The request’s top level must containprotocol: "niceeval.query/v3" and operation. Put the parameters below inside operation, alongside kind. A successful response has top-level outcome: "success".
run.get reads the selected Run’s plan and publication results; experiment.get reads the results currently adopted for each Eval. After a partial rerun, an Experiment’s current results can include Attempts from different Runs. A publication’s action is executed, carried, or accepted, meaning execution in this run, reuse of an existing result, or explicit adoption respectively.
run.plannedAttempts[] and experiment.cells[].members[] use the same member fields. verdict, score, costUSD, durationMs, and tokens are siblings of publication; before publication, the Verdict is null and metrics are unavailable.
publication describes only the publication binding. First check that publication.state is published, then read the Attempt locator from publication.attemptLocator.
Use --request - to pass a complete request through standard input:
runs[] lists Run IDs. To read one Run, replace the request’s operation with { "kind": "run.get", "runId": "<run-id>" }. Pass the request using exactly one of --request <file|-> or --request-json <document>.
When the selected Run, Experiment, or Attempt does not exist, the response has top-level outcome: "failure", failure.code: "inspection-selection-missing", and exit code 2. This means the selection is missing, not that an Eval’s Verdict is failed. First check IDs with run.list and confirm that you are reading the corresponding project’s run record.
niceeval show
For identity gaps, niceeval show explains the changes directly: Changed lists selectors for configuration, source, or data, with bounded summaries of both sides.
Source and data hashes show only the first 8 characters; additions and removals are marked added or removed. Each gap lists at most three items, with the rest shown as +N more.
When every gap in an Experiment has the same explanation, it appears once with all N gaps.
An identity change that cannot be explained shows Unexplained and a diagnostic summary. Run the niceeval exp <experiment> --dry command suggested under Next at the end of the run to see all changes.
show displays results in a fixed terminal format. With no selection arguments, it shows the default overview. --run <run-id> shows the given Run and --experiment <experiment-id> shows the given Experiment; both can be repeated.
Pass rates and scores are displayed by Eval type:
- Pass/fail Evals show
VerdictsandPass rate. - Score-only Evals show
VerdictsandScore, with no pass rate. - When both are mixed, only the pass/fail rows show a pass rate.
failed shows as failed even with a complete score, and keeps the score it earned. Only Evals with a passed Verdict and a complete score enter the success ranking.
With a passed Verdict and a complete score, terminal details show scored and the browser shows “Scored”. A zero score can also be complete: it says scoring finished, not that the goal was met, and NiceEval has no default passing line. A gate failure still shows as failed. An incomplete (partial) score keeps only a lower bound of the known score and does not count as a valid score.
@<locator> selects an Attempt. A command takes at most one locator, and it cannot be used with --run or --experiment; --run and --experiment cannot be used together either.
After selecting an Attempt, add any one of the options below to look at one aspect. The six options are mutually exclusive, and each requires an Attempt locator:
--assertions: Assertion roles, Verdict, Score, and reasons; use--entry <entry-id>for one entry.--source: the recorded source information and where each assertion sits in the source.--execution: an outline of the execution, with a length limit.--timing: how long each phase took.--usage: usage totals and every reported usage entry.--diff: file changes during the run.
--execution output shows three things separately: the collection state, whether the reporter declared the data complete, and the paging state. Even when the current page is empty or it is the last page, a partial mark from the reporter and the reason for the limit are kept.
- Filter by participant and event type with
--actor <id>and--type <event-type>. - When paging with
--continuation <token>, pass the same filters. - Expand one item directly with
--expand <stable-id>. The ID must come from anitemId,toolOccurrenceId, orcommandIdin the outline, not a positional number.--expandworks only with--execution, and cannot appear together with filter or paging arguments.
Assertion roles and reference results
--assertions, entry titles selected with --entry, and end-of-run Assertion feedback share these markers:
GATE pass/GATE fail/GATE n/a: whether the gate is satisfied.SCORE earned/possible: the scoring contribution; an entry that also acts as a gate shows both markers.NOTE met/NOTE unmet: a recorded condition, including a stop-only condition.
Verdict failed · Score 1/4, followed by Gates 1 pass, 1 fail · Scored 3 · Notes 1 met, 1 unmet. An entry’s author-written name does not determine its role, even if it contains “gate”. Machine JSON and query retain their structured fields.
When the current plan has gaps with previous results, the overview shows Reference results (from previous recorded attempts). Previous attempts N of K gaps counts previous results attached to current gaps. It includes Verdicts, the pass-kind Pass rate, and score-kind Score earned/possible. Each Previous result @… includes its Verdict and, for score-kind results, its score. These results belong to the prior configuration; rerun to update the current target. niceeval show --reference expands the same summary and entries.
niceeval view
view opens NiceEval’s built-in browser interface on your machine. Without --run, it opens the default overview; repeat --run to preselect several Runs. It accepts no positional arguments and does not load the project’s own pages, themes, components, or renderers.
--no-open only starts the server without opening a browser. --port sets the port; the server listens only on 127.0.0.1. Once page assets, local access authorization, and the read endpoint are ready, the command writes an openable local address to stdout once.
With --json, stdout switches to NDJSON in the niceeval.view/v1 format: a ready event when ready and a closed event on close. Startup or runtime failures are reported through the exit code and stderr.
view reads only the current project’s .niceeval/record.sqlite. When new results are written while it is open, the page prompts you to refresh and switches to the new data only after you confirm. It has no --record, SQL, deployment, or sharing options, and cannot specify custom reports, pages, themes, routes, or operations.
Choose a read path
All three paths read the same results, but each owns its own layout and presentation.Find and recover Runs
niceeval run list lists Runs by creation time, newest first, sorting ties by Run ID. Repeated --state filters match any selected state; different filters must all match:
--experiment suite matches suite and suite/..., but not suite-other. --limit must be a positive integer; omitting it leaves the list untruncated. In JSON, omitted counts the filtered results left out.
Rows in active or recovering show an owner observation of alive, gone, or unknown, along with the observation time. alive means the recorded process still existed when observed; gone means its termination was proved. For unknown, the reason distinguishes records from another machine, missing process identity, external Records, and probe failures. Observations do not change Run state; reading an external Record does not probe local processes.
--orphaned freezes the set of active and recovering candidates at one read boundary, lists their Run IDs before confirmation, then rechecks each one’s state, write generation, and process identity. It skips owners observed as alive or unknown. A failure for one Run does not stop the others, but any failure makes the command exit nonzero. JSON items list each Run’s recovered, skipped, or failed result. No candidates is a successful empty result. JSON mode requires --yes; human-readable mode asks for confirmation once, only in an interactive terminal.
recovering means recovery has acquired write ownership, but the Run has not yet been marked as ended. If the recovery process is interrupted, retry the same command; termination of the previous recovery process must still be proved. Recovery only records absence for unpublished slots and ends this Run, preserving published Attempts. It does not release shared State, run Experiment after, or clean up Sandboxes; those operations require their own explicit commands.