Skip to main content
By default, every Sandbox is destroyed after an Attempt finishes. First open the relevant Run or Attempt in niceeval view, or use a fixed query operation to inspect Verdicts, assertions, diffs, commands, and event streams. See the Debugging guide for the complete workflow. Some problems, though, are visible only in a live environment:
  • The environment never starts: a dependency install fails during setup, or the Agent CLI will not start. The Agent has not started running, so the event stream is empty. Entering the Sandbox and rerunning the install command manually is the quickest path.
  • The change is outside git diff: which packages were installed globally, which configuration was written under $HOME, or what PATH actually resolves to. Published File Changes do not contain any of that.
  • Rerunning is too slow: a cold start and installation can take minutes. When testing one hypothesis at a time, keeping the environment is much faster than starting over for every rerun.

​
Keep the state while running

--keep-sandbox is a run parameter for niceeval exp with two settings. failed (the default) keeps Attempts whose Verdict is failed or errored, including ones interrupted by a timeout. all also keeps passing Attempts. Use it when iterating on a setup Hook or checking the actual state of a passing environment; do not deliberately fail an Eval just to inspect it. Without this parameter, every Sandbox is destroyed. When the run finishes, the summary lists the Sandboxes kept and how to enter them:
Each line gives three things: the Attempt identity, the Sandbox instance ID, and the command that enters the environment. Use niceeval view --run <run-id> and select the relevant Attempt to inspect persisted evidence. A kept Sandbox does not run continuously and consume resources: a Docker container stops on disk, an E2B microVM pauses billing, and Vercel preserves the filesystem. niceeval sandbox enter wakes the environment before opening a shell in workdir. When you exit the shell, the environment automatically goes back to sleep. Add --leave-running to keep it running. Inside, you have the environment as it was when the Attempt finished, so you can run commands, inspect files, and reproduce the failure manually.

​
Inspect and clean up

The project’s retention registry records kept Sandboxes. Manage them with niceeval sandbox:
These management commands do not read niceeval.config.ts. You can still inspect or destroy a retained environment when that configuration cannot run. The commands read .env to prepare Provider credentials; repair it before retrying if it cannot be read. sandbox has no --json output. niceeval sandbox list --json reports Unknown option '--json' directly and does not continue by printing human-readable text. Machine-output flags belong only to commands that explicitly declare them. stop is idempotent. If the Sandbox has already gone away because it was deleted manually or expired in the cloud, that is not an error: the command removes the record and explains what happened. If a Provider fails to destroy it, the command keeps the record and returns an error so you can retry later; it does not hide a resource that is still alive. There is also a cleanup reminder: at the start of the next run, NiceEval prints one line when Sandboxes from the previous run are still kept.

​
Provider differences

  • Docker: keeping means the container is stopped on disk. It uses no memory, survives a Docker restart, and starts automatically when you enter it. Containers never disappear on their own, so Docker is the only Provider that needs active cleanup. In addition to niceeval sandbox stop, you can check directly with docker ps -a -f label=niceeval.keep-candidate=true.
  • E2B: keeping means pausing the microVM. Files and memory are retained together, billing stops while it is paused, it is retained indefinitely, and it resumes automatically when you enter it.
  • Vercel Sandbox: keeping means stopping the microVM. The filesystem is kept and can be resumed later, but memory state is not retained, so processes must restart after waking. Once the Provider retention period passes, niceeval sandbox list marks it expired.
  • Custom Providers: a Provider created by defineSandbox does not support retention, because the later sandbox stop call does not load user configuration and cannot safely recover a custom teardown function in a new process.

​
Boundaries

A kept Sandbox is only for investigation. It cannot resume or be rescored. Verdicts, assertions, and diffs remain published facts. See Viewing results for how to inspect results.