Skip to main content
Someone changed a prompt, tried it locally twice, decided it was fine, and merged it. The next day you discover that another class of questions now gets every answer wrong. Put the evals in CI so every PR runs them automatically, and regressions are caught before merge. CI runs the same niceeval exp you use locally, with no extra configuration:
  • When evals fail, a run errors, or the run is interrupted, the process exits with a non-zero status and CI turns red.
  • With --json, the last line is this run’s receipt, which contains the new Run IDs that later steps can use to read results.
  • With --junit <path>, it writes a JUnit file so the CI platform can display results per test case.

Decide what turns CI red

Whether CI turns red depends only on the process exit status. Any of the following exits with a non-zero status:
  • A pass/fail eval has an Attempt that did not pass.
  • A scored eval has an explicit .gate() that was not met, for example a measured value below the minimum of .gate(minimum).
  • An Attempt errored (errored), or not every planned Attempt got a result.
  • The run was interrupted, or the results could not be saved.
A scored eval without a failed gate counts as success even when its score is low. To have a low score block the merge, add .gate() to the matching assertion in the eval. There is no global strict switch; CI and local runs read the same eval source and reach the same verdicts.

Run the evals

This command gives CI three things:
  • Exit status: CI uses it to decide whether this step succeeded or failed.
  • NDJSON output from --json: the earlier lines describe run progress and diagnostics, and the last line is the receipt.
  • JUnit file: for the CI platform to read and display.
JUnit is only a display format for the CI platform. Verdicts, usage, assertions, and diagnostics are still saved in the project’s .niceeval/record.sqlite, and later inspection reads from there.

Configure GitHub Actions

The step below saves the JSON output while passing the exit status of niceeval exp back to CI correctly:
Without set -o pipefail and exit "${PIPESTATUS[0]}", the exit status comes from tee, and failing evals do not turn CI red. Inject model credentials as CI secrets into the environment variables the command needs, such as ANTHROPIC_API_KEY above. Do not write credentials into the Experiment, receipt, JUnit, or logs. For which variable each Agent reads, see Configuration and Environment Variables.

Read this run’s results in later steps

Once the run starts, the last NDJSON line is the receipt. Its receipt.createdRunIds lists the Runs created this time, and later steps use it to open the results:
To have a script process the results, first find the available queries, then send a complete request:
If the command exits before the run starts because of an argument, configuration, or selection error, there is no receipt, and the terminal prints error:. Fix the command, Experiment, or selection directly. Do not guess a Run ID.

Keep CI logs readable

CI logs are for watching run progress and diagnostics. To dig into results, a person runs niceeval show --run <run-id> or niceeval view --run <run-id> locally with the Run ID from the receipt; scripts and AI run query discover first. Do not treat progress and diagnostic lines as a long-term result format.