Run NiceEval in CI

When CI runs an Experiment, the process exit status decides whether the gate passes. --json provides feedback from the current process; its final receipt contains the Invocation’s runIds. A JUnit file gives the CI platform test results to display.

Basic command

npx niceeval exp ci --json --junit ./artifacts/niceeval-junit.xml

This command has three outputs:

Exit status: CI uses it to pass or fail the job.
NDJSON: Progress and diagnostics describe the current process; the final receipt describes the Invocation.
JUnit: A temporary file for the CI platform.

JUnit and external-platform files do not own the Record. The quiescent Record’s named channels keep the recorded Verdict, Usage, assertions, and diagnostics.

GitHub Actions example

This step retains JSON output and correctly preserves the exit status from niceeval exp:

- name: Run evals
  shell: bash
  run: |
    set -o pipefail
    npx niceeval exp ci --json --junit ./artifacts/niceeval-junit.xml \
      | tee ./artifacts/niceeval.ndjson
    exit "${PIPESTATUS[0]}"

- name: Upload JUnit
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: niceeval-junit
    path: ./artifacts/niceeval-junit.xml

Inject model credentials through CI secrets. Do not put credentials in an Experiment, receipt, JUnit file, or log.

Use the receipt in a later step

After an Invocation is created successfully, the final NDJSON record contains its receipt. A later script can use its runIds to select an explicit Run for viewing or export:

npx niceeval view --run 01J9ZK3M6P4T7V9X2C5N8QW0RY \
  --out ./report-site \
  --no-open

If an input error ends the command before an Invocation exists, there is no receipt. Correct the command, Experiment, or selection condition instead of guessing a Run ID.

Keep feedback readable

CI logs are for the current process’s progress and diagnostics. To inspect business results, use the receipt’s runIds with show or view. To share results over time, use view --out to produce a static report site. See Publish a Static Report for the full static-site workflow.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Run NiceEval in CI

Basic command

GitHub Actions example

Use the receipt in a later step

Keep feedback readable

​Basic command

​GitHub Actions example

​Use the receipt in a later step

​Keep feedback readable

Basic command

GitHub Actions example

Use the receipt in a later step

Keep feedback readable