Compare Two Runs: Is the Baseline or the Candidate Better

You ran a baseline on Monday, changed the prompt on Wednesday and ran again, and now want to know which evals this change made better and which it made worse. Before comparing, pin which run each side is. “The latest results” change with the next run; if you compare against them, today’s conclusion and tomorrow’s may differ. Every time niceeval exp finishes, the terminal’s NEXT panel lists the Run IDs created this time. Write down the baseline and candidate IDs.

View them side by side in the browser

pnpm exec niceeval view --run <baseline-run-id> --run <candidate-run-id>

View opens both Runs at once. You can compare pass/fail, scores, duration, and cost eval by eval, then click into a specific run to see the conversation and tool calls.

Let scripts and CI read the comparison

When an automated flow needs a structured comparison, use a runs.compare request:

pnpm --silent exec niceeval query run --request - <<'JSON'
{
  "protocol": "niceeval.query/v2",
  "operation": {
    "kind": "runs.compare",
    "mode": "paired",
    "leftRunIds": ["<baseline-run-id>"],
    "rightRunIds": ["<candidate-run-id>"]
  }
}
JSON

mode decides how the two sides line up:

`mode`	When to use it
`side-by-side`	Each side gives its own results without computing differences. Use it when the two runs have different eval sets and you only want to view them side by side.
`exact`	Requires both sides to have exactly the same eval set, and returns a named issue when they differ. Use it where comparisons must be on the same basis, such as release gates.
`paired`	Pairs by the same eval and compares eval by eval. Most often used for “which evals did this change affect”.

The comparison honestly reports each side’s denominator, plus unpaired evals, excluded evals, and missing results. Missing results are not treated as zero or failure. When the conditions are not met, you get a named error explaining why, not a number that looks normal. First check that the request is valid with query explain, then run it for real:

pnpm --silent exec niceeval query explain --request compare.json

When a comparison can be trusted

Both sides ran the same set of evals, each run a similar number of times.
The two sides differ only in the one variable you want to compare; see Organize Experiments for how to write that.
Look at cost and duration alongside the pass rate. A slightly higher pass rate at double the cost is not necessarily worth switching to.

For how to read a single run’s results, see View Run Results.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Compare Two Runs: Is the Baseline or the Candidate Better

View them side by side in the browser

Let scripts and CI read the comparison

When a comparison can be trusted