Skip to main content
When niceeval exp is interrupted directly by kill -9, a CI time limit, or power loss, the process cannot run its cleanup. Published Attempts remain readable in the project’s single .niceeval/record.sqlite; unpublished work never becomes partial public fact. Normal Ctrl+C and SIGTERM follow a controlled interruption path: NiceEval publishes completed Attempts and closes the remaining slots with an interruption reason. This page covers only cases where the process never had a chance to clean up.

​
Inspect published results first

List the Runs created by the Invocation. If you have the Invocation ID, use it to narrow the list:
An active Run still shows published Attempts, while unpublished expected slots appear as pending. Do not decide that it can be taken over from elapsed time, an expired PID, or a stopped heartbeat.

​
Close a Run that lost its owner

Recover only after confirming that the original process has ended:
run recover verifies the former owner’s exact process identity. After verification succeeds, it closes the Run as interrupted and records an absence reason for each unpublished slot. It does not delete the Run or published Attempts, and it does not let the old generation write again. The command refuses recovery when evidence is insufficient. Confirm that the original process, CI job, and any remote workers have stopped, then check again; do not substitute a timeout or “it has waited too long” for owner evidence. To obtain results again, rerun the original Experiment. The new Invocation creates a new Run; whether it adopts existing Attempts depends on the current Experiment’s eligibility and --rerun policy.

​
Reclaim Sandboxes left by a forced kill

Sandboxes running at the time of a forced kill do not enter the retention registry. First inspect orphaned instances without modifying them:
After confirmation, reclaim them:
  • orphan means the owner on the same host is proven to have ended and the Sandbox can be destroyed safely.
  • unverified means the instance came from another host or cannot be confirmed; it is not deleted by default. Use niceeval sandbox prune --force only after confirmation.
  • Compose companion containers and networks are listed and destroyed as a group.
  • Vercel Sandbox has no queryable orphan channel; it is reclaimed under the Provider’s retention policy.
  • Environments retained with --keep-sandbox are unaffected by prune; continue managing them with niceeval sandbox stop.

​
Recover a shared-state lease

An Experiment that declares sharedState.key does not start a new Invocation with the same key until the original owner releases the lease normally or an operator completes explicit recovery. First run the inspection command without a token to read current owner evidence:
After confirming that the original owner has ended and that the remote service and checkpoint it controlled are quiescent, copy the displayed token and confirm explicitly:
Recovery reruns teardown only for that Experiment. A wrong or stale token, or failed cleanup, never releases a later owner’s lease. After fixing teardown, retry with the currently displayed token.