Let a Coding Agent Iterate on Results

You ask a Coding Agent to fix a regression. It changes the code, you run the evals, you copy the failing output to it, it changes the code again, and you run again. After a few rounds, most of the time goes into moving terminal output around. A better approach is to let the Agent run the loop itself:

Run the evals.
Get this run’s Run ID from the receipt printed when the run ends.
Use the Run ID to view the results and find the failing runs.
Decide whether to change the application, the Adapter, or the eval.
After the change, run the same scope again and confirm the fix.

This page explains which command to use at each step and a few boundaries the Agent must respect.

Have the Agent read the docs for the installed version

Commands and APIs the Agent writes from training memory may not match the NiceEval version you installed. Have it read the docs that ship with the npm package first. NiceEval ships its Chinese docs in the npm package’s docs-site/zh/ directory and provides an INDEX.md at the package root for Coding Agents. The Agent reads node_modules/niceeval/INDEX.md first, then follows the index to the pages the current task needs. pnpm exec niceeval init initializes the configuration and writes a managed guidance section into the project’s AGENTS.md. If the project only has CLAUDE.md, it writes into CLAUDE.md; if neither exists, it creates AGENTS.md. After upgrading NiceEval, run init again to refresh that section. The guidance separates two things:

When writing code, the Agent can of course read project source such as evals/ and agents/.
When it needs to prove what happened in a run, it uses niceeval show in the terminal, niceeval query for stable JSON, and niceeval view for browser review. It does not scan raw files under .niceeval/, and it does not reconstruct a historical run from current source.

When these commands cannot show the needed information, have the Agent report “the CLI cannot show this” instead of working around it by reading files. You can write the Agent’s starting task like this:

Read node_modules/niceeval/INDEX.md first.
Then read the pages in the index that are relevant to the current task.
Run the Experiment with the command, and pick the Run to inspect from the final receipt.

Run and save the receipt

Have the Agent run with --json and save the output:

pnpm exec niceeval exp checkout --json | tee .niceeval-invocation.ndjson

Each line of --json describes the progress and diagnostics of this run. Exactly one final line is the receipt:

{
  "type": "receipt",
  "receipt": {
    "invocationId": "01J8ZK3M6P4T7V9X2C5N8QW0RY",
    "createdRunIds": ["01J9ZK3M6P4T7V9X2C5N8QW0RY"],
    "publicationCutoff": "{...}",
    "startedAt": "2026-08-09T10:00:00.000Z",
    "completedAt": "2026-08-09T10:01:00.000Z",
    "completion": "completed"
  },
  "summary": { "...": "..." }
}

The Agent trusts only this last line. createdRunIds lists every Run created this time; use it to view results afterwards. Do not parse publicationCutoff; pass it to later commands as is. completion being completed only means the run ended. It does not mean the task is done, and the evals may not have passed. The receipt is also not a long-term result format; read results through createdRunIds.

View results by Run ID

After the run ends, the Agent first opens the Run from the receipt in the terminal:

pnpm exec niceeval show --run 01J9ZK3M6P4T7V9X2C5N8QW0RY

The output lists the Run’s denominator, each eval’s verdict, scores, result coverage, and the locator of each Attempt. To look at an Attempt in detail, copy its locator:

pnpm exec niceeval show @<attempt-locator>
pnpm exec niceeval show @<attempt-locator> --source
pnpm exec niceeval show @<attempt-locator> --execution

show @<attempt-locator> shows the Attempt’s verdict, assertion summary, whether each part is complete, and known limitations. --source shows the eval source used in this run and the result of each assertion. --execution shows a condensed execution trace; to expand one item, copy the ID next to it and run --execution --expand <stable-id>. When a script, CI, or Agent needs to process results reliably, first find the available queries, then send a complete request:

pnpm --silent exec niceeval query discover
pnpm --silent exec niceeval query explain --request request.json
pnpm --silent exec niceeval query run --request request.json

query returns JSON in a fixed format. When a person wants to look in the browser, use pnpm exec niceeval view --run <run-id>; view does not accept locators.

Decide the next step from the status

Status	Next step
Attempt `failed`	Compare the outcomes the task allows, the actual artifacts, the evidence, and the assertion conditions, then decide whether to change the system under test, the Adapter, or the eval.
`errored`	Check the error text, Adapter, Sandbox, or credentials.
`not-recorded`	An Attempt planned for this run has no result. Check why it did not run.
`core-invalid`	Read the issue stated in the output. Do not treat it as missing collection.
`partial`	The result is incomplete. Read the denominator and the issue notes to see which Attempts were not counted.
`record-schema-unsupported` or old project state blocks the run	Keep the original project and its history, and follow the old project recovery guide to rerun in a copy of the project.
Query result is `unsupported` or `failed`	This only affects views that depend on that input. Read the stated issue.

When handling failed, do not change an artifact that already meets the task just to pass an overly narrow Match. If the artifact meets the task but the assertion rejects an outcome the task allows, the eval misjudged it and the eval should change. For the full attribution process, see the debugging guide.

Run again after a change

Have the Agent verify one hypothesis at a time: change the program or the eval, run the same scope, then read the new receipt and Run. By default, Attempts unaffected by the change that already have a conclusion from last time are reused. Add --dry before running to see which ones will be reused, which will rerun, and why. When every Attempt must really execute, add --rerun all:

pnpm exec niceeval exp checkout --rerun all --json

For the rules on reusing and accepting old results, see Rerun Only the Evals Affected by a Change. A finished run only means this run handed off its results. It does not mean the user’s task is done.

When the task asks for a fix and you have authorization, fix what made the run error first, then reverify only the affected scope.
When the task asks only for analysis, only do attribution. Do not change code or rerun.
When a business failure is what the task expects, keep the result and report it honestly. Do not change the application or the eval just to turn it green.

Saved results cannot be changed

Saved results are never modified. If the Agent wants a different result, it changes the program or the eval, runs again, and views the result through the new receipt’s createdRunIds. When you need to hand results to someone else, copy the project’s .niceeval/record.sqlite. Do not use terminal output or NDJSON as a sharing format.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting