Skip to main content
By default, a Sandbox is destroyed after each Attempt ends. Most problems can be located first with niceeval show @<attempt-locator> or niceeval view by looking at the verdict, assertions, file changes, commands, and the call duration waterfall timeline. For the full workflow, see the Debugging guide. Some problems, though, can only be seen in a live environment:
  • The environment does not start: installing dependencies fails during setup, or the agent CLI cannot start. The agent has not started running yet, so the conversation record is empty. The fastest fix is to connect to the live Sandbox and rerun the install command by hand.
  • The change is outside git diff: which packages were installed globally, what configuration was written under $HOME, and what PATH actually is. Workspace file diffs do not include these.
  • Testing hypotheses: a cold start plus installation takes minutes. When you want to test guesses one by one, keeping the environment is much faster than rerunning every time.

Pause or keep state while running

Add --pause-on-failure or --keep-sandbox to niceeval exp:
--pause-on-failure blocks automatic teardown upon any assertion failure or error, printing native connect commands and waiting for user input. When Sandboxes are kept, the summary clearly displays native connect and cleanup commands:

Debug directly via native provider CLIs

NiceEval does not build intermediate terminal proxies or custom command wrappers. Developers and AI Coding Agents use battle-tested native CLIs directly:

1. Interactive debugging for developers

Run the native command directly in a new terminal tab:
  • Paused live Attempt (--pause-on-failure): The Sandbox remains running live in the background while waiting for confirmation:
  • Kept completed Attempt (--keep-sandbox): Docker stops the container upon completion to save memory while preserving disk state. Start it before entering:
Inside the container, you can rerun build commands to observe raw errors, verify $HOME configs, and check processes.

2. Automated non-interactive execution for AI Coding Agents

Coding Agents avoid entering interactive shells, and instead execute one-shot native commands to obtain structured output:

Inspect and clean up residual resources

Sandbox resource management is delegated to mature native tools:
niceeval.keep-candidate=true is attached when a container is created; it does not indicate the container has stopped or been kept. Label filter results may include active Runs and should not be batch-deleted without inspection. Native CLIs do not load niceeval.config.ts, but still require valid Provider connectivity and credentials.

Server-side safety net against client interruption

When creating cloud Sandboxes, NiceEval sends a server-side lifetime bound. When not explicitly specified, it derives from the Attempt timeout plus a 2-minute buffer, defaulting to 15 minutes when no Attempt timeout is declared. When a local process is terminated by network loss, power outage, or kill -9, this server-side bound limits the remote instance’s lifetime. Expiration actions and billing rules depend on the Provider and should be verified via native tools. Lifetime limits and idle auto-pause are distinct mechanisms; one should not assume instances immediately pause or cease billing upon network disconnection.