Skip to main content
You fixed a tool call bug in your Agent and want to confirm it is really fixed. The full eval suite takes an hour and costs tens of dollars, but this change only affects a few of the evals. NiceEval remembers each eval’s last result. When you run the same Experiment again, evals that were not affected by this change and already have a conclusion from last time reuse their old results directly, and the rest run again. You do not need to pick evals by hand.

Look at the plan before running

Add --dry, and NiceEval only lists the plan, without calling the model or creating a Sandbox:
Each line is one Attempt. After reuse is the locator of the old result to reuse; after execute is the reason for rerunning. The input-identity-changed on the second line above means this eval’s input changed, for example you changed the eval, the Fixture, or the model. When the result differs from what you expected, look at the reason first. For example, if you only changed a group name in the report but see the whole batch as execute, some input that affects results changed along with it.

Which results are reused

An old Attempt is reused automatically only when all of the following hold:
  • It already has a clear conclusion, passed or failed. errored means it did not complete last time, so it always reruns.
  • The eval, Fixture, model, Agent, scoring criteria, and run conditions are all the same as this time.
  • Its duration did not exceed this run’s timeout.
  • This run’s --rerun did not ask to rerun it.
  • This run did not request real execution with --keep-sandbox.
Changing only the Experiment’s description or labels does not invalidate old results. Eval files are different: when the file content changes, that eval reruns.

Control the rerun scope

A bare --rerun is the same as --rerun failed, and evals that already passed do not rerun. To redo the passed ones too, you must write --rerun all. --rerun only affects this run and does not change existing results.

Confirm old results still hold under a new configuration

Sometimes a change makes NiceEval decide that old results cannot be reused, but you know they are actually unaffected. For example, you added a logging switch to the Agent, and this eval’s behavior did not change at all. NiceEval does not guess for you; you need to accept them explicitly:
accept creates a new Run that references this old Attempt, and the original Attempt stays unchanged. To accept several, put all the locators in one command; NiceEval checks every input first and never accepts only half of them. For how to get candidate locators and what to check before accepting, see Review and Accept Historical Attempts.

See the results after the run

When the run ends, the terminal’s NEXT panel lists the Runs created this time, with two commands to view each in the terminal and in the browser:
Both reused and rerun evals appear in this Run, and the pass rate is computed over the complete batch. Do not use show --experiment to view results before exp has finished; at that point there may be no published results yet. For the full ways to view results, see View Run Results. For how reuse and rerun decisions work, see Runner.