Skip to main content
A batch of evals finishes and the terminal shows FAILED: 3 of 12 did not pass. You want to know which ones, what the Agent actually said in those runs, which tools it called, and why it did not meet the assertions. NiceEval gives different readers three entry points, all reading the same results: Results are readable as soon as each Attempt finishes; you do not have to wait for the whole batch.

Start from the overview

The default overview answers “which results is the current project still missing”: the denominator is your current Experiments and evals, not everything ever run. Attempts without problems are collapsed by default, so failures and the next command to run stay on one screen. The output looks like this:
A few numbers worth looking at first:
  • Covered is the number of positions that currently have a usable result, out of the planned number. When Gaps is not 0, some positions have no result yet, or the old result no longer applies, for example because you changed this eval. Under each gap is the reason and an entry to the last result; old scores are not counted in the current pass rate, and the pass rate is followed by (partial) to remind you it is based only on available results.
  • When errored is not 0, handle those first. errored means the run itself did not complete, for example the application was not running or the interface returned an error, so those Attempts reached no conclusion.
  • Scored evals show Score instead of a pass rate; see Pass/Fail and Scored Evals for how to read it.
To fill the gaps, run the suggested niceeval exp <id> --dry on the last line to see what would run, then drop --dry to run for real. When you see Attempts hidden, narrow the scope with the suggested command; to show every Attempt, add --all.

Narrow to one Experiment or one run

--experiment shows the result of each eval under one Experiment. --run shows one run; the Run ID is in the NEXT panel when exp finishes. Both can be passed several times. You can also look while a run is still going: finished Attempts appear, and unfinished ones show as pending.

Look at one failed run

In the overview and Experiment views, each Attempt is preceded by an Attempt locator starting with @, which uniquely points to that run within the project. Copy it:
The main part of the output looks like this:
Next lists the aspects you can keep looking at. Choose by the question you want to answer: --execution gives an outline of the conversation and tool calls. To expand one item, copy the ID shown in the outline:
When an item shows not-recorded, partial, or unavailable, that information was not collected or is incomplete; it does not mean zero or failure. For example, a directly connected application has no file changes, so diff is not-recorded.

Decide whether to change the application or the assertion

failed only means at least one assertion was not met; it does not necessarily mean the Agent got it wrong. After reading --execution, there are three cases:
  • The Agent’s output really does not meet the task requirements: change the application or the prompt.
  • The Agent’s output meets the task requirements, but the assertion rejected it, for example a regex that is too rigid: change the eval.
  • The task itself is vague, so you cannot say whether the Agent got it right: first make the task and criteria clear.
Do not loosen assertions arbitrarily just to turn results green.

View in the browser

Without arguments it opens the overview, and --run opens one or more specific runs. In the browser you see full conversations, the inputs and outputs of tool calls, and duration waterfalls. When new results are published, the page prompts you to refresh and only switches after you confirm, so what you are looking at does not change suddenly. --no-open does not open the browser automatically, and --port sets the port.

See why an assertion got its score

show tells you which assertion failed. To see the reasons the Judge gave and exactly how many points a scoring item got, read that assertion’s detail with query. First get this run’s list of assertions:
Pick an entryId from the returned assertions.entries, then read its detail:
In the detail, contribution is how many points this item earned, evaluation is the actual measured result, and the Judge’s reasons are in explanation. When a field is missing or marked unavailable, interpret it by the returned reason; do not infer it from other fields.

Let scripts and CI read results

query outputs only JSON and does no terminal formatting. Do not parse the text output of show, and do not read files under .niceeval/ directly; the format may change between versions.
discover lists every request type the current version supports, such as listing Runs, reading a Run summary, reading Attempt details, traces, diffs, and comparing two Runs. After writing a request following the CLI reference, first check it with query explain --request <file>, then run it with query run --request <file>. In pnpm projects, add --silent; otherwise pnpm’s own output gets mixed into the JSON. To compare two runs, see Compare Two Runs.