> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Recover After a Forced Kill

> After kill -9, a CI timeout, or a power loss interrupts a process, preserve published results, close a Run that lost its owner, and reclaim leftover Sandboxes.

When `niceeval exp` is interrupted directly by `kill -9`, a CI time limit, or power loss, the process cannot run its cleanup. Published Attempts remain readable in the project's single `.niceeval/record.sqlite`; unpublished work never becomes partial public fact.

Normal Ctrl+C and SIGTERM follow a controlled interruption path: NiceEval publishes completed Attempts and closes the remaining slots with an interruption reason. This page covers only cases where the process never had a chance to clean up.

## Inspect published results first

List the Runs created by the Invocation. If you have the Invocation ID, use it to narrow the list:

```bash theme={null}
niceeval run list --invocation <invocation-id>
niceeval run show <run-id>
niceeval view --run <run-id>
```

An `active` Run still shows published Attempts, while unpublished expected slots appear as `pending`. Do not decide that it can be taken over from elapsed time, an expired PID, or a stopped heartbeat.

## Close a Run that lost its owner

Recover only after confirming that the original process has ended:

```bash theme={null}
niceeval run recover <run-id> --yes
```

`run recover` verifies the former owner's exact process identity. After verification succeeds, it closes the Run as `interrupted` and records an absence reason for each unpublished slot. It does not delete the Run or published Attempts, and it does not let the old generation write again.

The command refuses recovery when evidence is insufficient. Confirm that the original process, CI job, and any remote workers have stopped, then check again; do not substitute a timeout or "it has waited too long" for owner evidence.

To obtain results again, rerun the original Experiment. The new Invocation creates a new Run; whether it adopts existing Attempts depends on the current Experiment's eligibility and `--rerun` policy.

## Reclaim Sandboxes left by a forced kill

Sandboxes running at the time of a forced kill do not enter the retention registry. First inspect orphaned instances without modifying them:

```bash theme={null}
niceeval sandbox list --orphans
```

After confirmation, reclaim them:

```bash theme={null}
niceeval sandbox prune
```

* `orphan` means the owner on the same host is proven to have ended and the Sandbox can be destroyed safely.
* `unverified` means the instance came from another host or cannot be confirmed; it is not deleted by default. Use `niceeval sandbox prune --force` only after confirmation.
* Compose companion containers and networks are listed and destroyed as a group.
* Vercel Sandbox has no queryable orphan channel; it is reclaimed under the Provider's retention policy.
* Environments retained with `--keep-sandbox` are unaffected by `prune`; continue managing them with `niceeval sandbox stop`.

## Recover a shared-state lease

An Experiment that declares `sharedState.key` does not start a new Invocation with the same key until the original owner releases the lease normally or an operator completes explicit recovery. First run the inspection command without a token to read current owner evidence:

```bash theme={null}
niceeval exp <experiment> --teardown \
  --recover-shared-state <key>
```

After confirming that the original owner has ended and that the remote service and checkpoint it controlled are quiescent, copy the displayed token and confirm explicitly:

```bash theme={null}
niceeval exp <experiment> --teardown \
  --recover-shared-state <key> \
  --owner-token <displayed-token> \
  --confirm-owner-terminated \
  --confirm-remote-quiesced
```

Recovery reruns `teardown` only for that Experiment. A wrong or stale token, or failed cleanup, never releases a later owner's lease. After fixing teardown, retry with the currently displayed token.
