Skip to main content
When niceeval exp is interrupted directly by kill -9, a CI time limit, or a power loss, the process has no chance to clean up. Attempts that already finished are still saved in the project’s .niceeval/record.sqlite and can be read as usual; unfinished ones never leave half-written results. When you press Ctrl+C or send SIGTERM normally, NiceEval cleans up by itself: it saves the finished Attempts and marks the remaining planned Attempts as interrupted. This page only covers cases where the process had no chance to clean up.

Look at the saved results first

List the Runs this run created. If you got the invocationId from --json output or the CI log, use it to narrow the list:
You can also look in the browser: pnpm exec niceeval view --run <run-id>. A Run that was force-killed is still active. It shows the Attempts that already finished, and unfinished ones appear as pending. Do not decide it can be taken over based on how long it has run, an expired PID, or a stopped heartbeat. If you do not know the Invocation ID, use pnpm exec niceeval run list --state active --state recovering --experiment <experiment-prefix> --limit 20 to find Runs that need attention. --experiment matches by path prefix; the list is ordered newest first and reports how many items were omitted. An owner observation of gone means the original process was proved to have terminated; alive means it was still running when observed; unknown means the evidence is insufficient, with a reason provided. Owners in external Records always show as unknown. These are observations at a point in time; recovery checks again.

End the Run that lost its process

Run the recovery only after confirming the original process has ended:
run recover first verifies that the original process really has ended. Once verified, it marks the Run interrupted and records why each unfinished Attempt is missing. It does not delete the Run or the finished Attempts, and the original process cannot write again even if it lingers. When the evidence is insufficient, the command refuses to recover. In that case, first confirm that the original process, the CI job, and any remote workers have all stopped, then retry. Do not bypass this check because “it has waited long enough”. Recovery of a set skips owners observed as alive or unknown and reports the reason for each. A failure for one Run does not stop the others, but the command exits nonzero. JSON mode requires --yes; in an interactive terminal, you can omit it, review the candidates, and confirm once. If recovery itself is interrupted, the Run may remain recovering. After confirming that the previous recovery process has ended, retry the same command to continue. It preserves published Attempts and does not release shared State, run Experiment after, or clean up Sandboxes.

Rerun the unfinished part

When you need complete results, run the original Experiment again. The new run creates a new Run; Attempts that finished last time and meet the reuse conditions are reused, and the rest run again. Before running, you can add --dry to see which will be reused and which will rerun. For the rules, see Rerun only the evals affected by a change.

Reclaim Sandboxes left by a forced kill

Reclaiming residual Sandbox instances after an interruption is fully delegated to native mature toolchains. Use provider CLIs directly to inspect and remove leftover containers or instances:
  • Docker and Compose: Both niceeval.host and niceeval.keep-candidate=true can be present on containers of active Runs. Labels do not prove a resource is an orphan; do not batch-delete solely based on labels.
  • Cloud Sandboxes (Instabox / E2B): Creation requests enforce a server-side lifetime bound. Expiration actions and billing rules depend on the Provider; after a client kill, verify resource status via native tools.
  • Retained Sandboxes: Confirm further investigation is no longer needed before deleting specific instances; see Keep Sandbox state to debug problems.

Recover a shared-state lease

For an Experiment that declares sharedState.key, only one run can hold the same key at a time. When the original run did not release it normally, new runs keep waiting until you recover it explicitly. First run the check command without a token to see the current holder’s information:
After confirming that the original run has ended, and that the remote services and checkpoints it operated on have stopped changing, copy the token the command shows and confirm explicitly:
Recovery reruns after only for this Experiment. When the token is wrong, the token has expired, or the after callback fails, the lease is not released, and later holders’ leases are not affected. Fix the after callback, then try again with the token currently shown.