Sandbox interface. Switching Providers changes one line in the Experiment, and the evals stay unchanged.
Choose a Provider
Start by choosing based on where you run evals:
Choose one in the Experiment’s
sandbox field:
If it is missing, creating the Sandbox reports an error with the install command, for example
Docker sandbox requires 'dockerode'. Install it with: pnpm add dockerode @types/dockerode. Install it as shown and run again.
Use an Incus VM
Declare Docker, Compose, and minimum data capacity withsandboxRequirements() in the eval, and select the environment with incusSandbox() in the Experiment. Each Attempt gets its own VM, which is destroyed when it finishes. Incus does not support --keep-sandbox or sandboxReuse.
Run pnpm exec niceeval sandbox provider doctor incus before starting. Development environments use a separate --development check and require acceptDevelopmentDomain: true in the Experiment. For configuration and cache maintenance, see Run Docker Inside a Sandbox.
Connect to OpenSandbox
Use the matchingrelease-1.1.1 service and execd, with SDK 1.1.1. The older execd:v1.1.0 does not support the native argv this Provider requires.
Set OPEN_SANDBOX_DOMAIN and OPEN_SANDBOX_API_KEY on the machine running NiceEval. The endpoint is a complete HTTPS URL; local development can use HTTP. Credentials stay on the host and are not written into the Sandbox.
lifetimeMs must cover the remaining Attempt deadline plus 30 seconds for cleanup. Normal completion destroys the instance immediately. Use user to select the execution identity and pathPrepend to add executable directories.
The OpenSandbox service chooses the runtime, such as Docker or Kubernetes; NiceEval does not need that setting. Restrict the endpoint to trusted hosts, or authenticate every HTTP and WebSocket route at a gateway. OpenSandbox’s Docker proxy routes are not protected by the control API key and must not be exposed directly to the public internet.
The image must start execd as root and provide Bash and Git. Use user to run the Agent as a non-root identity. Interactive Coding Agents also require Node.js 24.
Cancelling or timing out a command destroys the entire instance. This Provider does not support retention or reuse across Attempts. A running Sandbox does not automatically have a Docker daemon.
When the service enables rootfs snapshots, images pinned to a digest can use before preparation caching. The cache saves filesystem contents, and each Attempt creates an independent instance from the snapshot. It does not save running processes, volumes, or tmpfs. Start services in a command step that cannot be cached. Added mounts or known sensitive content let the Attempt continue without publishing a cache entry.
Use sandboxCache: { before: "bypass" } or --sandbox-before-cache bypass when troubleshooting to execute preparation again and skip cache reads and publication. If a snapshot request fails after it has been sent, the current Attempt ends. Completed preparation steps are not automatically replayed within that Attempt.
When connecting a service, verify that a successful destruction response means the instance and its processes have stopped. A service that only acknowledges asynchronous deletion cannot meet this requirement; naming the runtime does not replace verification.
Instabox: start from a snapshot
SetINSTABOX_API_KEY in the host’s .env, then select a specific snapshot ID that is ready to use:
/data/home/workspace. Prepare stable dependencies when building the snapshot; start services and authenticate the Agent at run time to avoid saving login credentials in the snapshot.
spec supports the official standard sizes instabox-small, instabox-medium, instabox-large, instabox-xlarge, and instabox-2xlarge, along with the numeric aliases 2/4, 4/8, 8/16, 16/32, and 32/64. Explicit resources: { vcpus, memMiB } takes priority over spec; when both are omitted, the default is instabox-medium.
Each Attempt creates its own Machine and deletes it on completion; the base snapshot is kept. lifetimeMs is a whole-second duration in milliseconds, from 60 seconds to 24 hours, and must cover the remaining Attempt deadline plus 30 seconds. It controls automatic deletion after a lost connection; normal cleanup deletes the Machine sooner. Cancelling or timing out a command also deletes the entire Machine.
This Provider does not support retention, reuse across Attempts, or before snapshot caching. The host must be able to reach the Instabox gateway over SSH. Neither the HTTP API key nor the SSH token is written into the Machine. Creation disables automatic Vault and Org Secrets injection.
For snapshot build steps, see Prebuilt Sandbox Environments.
Docker Compose: workspaceService names the main Sandbox
When one eval needs more than one container (for example an application container plus a database), declare the whole Compose environment with dockerComposeSandbox:
workspaceService names which service in the Compose file is the main Sandbox. The Agent, the commands and file operations of t.sandbox, workdir, and file change records all apply only to this one container.
Other services (databases, mock servers, and so on) are only supporting resources: you cannot runCommand on them, upload to them, or read and write their files, and their file changes are not visible. To use them, go over the network, for example reaching db:5432 from the client container through Compose’s built-in DNS.
Let the Agent run Docker itself
When the Agent needs to rundocker build, docker run, or docker compose, use an Incus VM with a dedicated kernel and a Docker daemon inside the VM. See Run Docker Inside a Sandbox. An OpenSandbox service running on Docker does not mean the Agent can run Docker inside its container.
Work with the Sandbox in Adapters and evals
Whichever Provider you use, Adapters and evals get the sameSandbox interface. Common operations:
The error messages thrown by the two
OrThrow methods include the last few lines of stderr, already cleaned, redacted, and truncated. When stderr is empty, stdout is used instead. When debugging, read this summary first, and read SandboxCommandExitError.result when you need the full output.
Prepare the Sandbox in order
Before the Agent starts working, the Sandbox usually needs tools installed, configuration written, and materials in place. Write these preparation steps with.before(), and cleanup that runs whether the run succeeds or fails with .after(). Every Provider function returns a SandboxLayer that you can keep appending steps to, and appending never modifies the original value:
.before() steps run in the order they are written. A callback also counts as a step, and .after() does not break the chain. A later step waits for the previous one even when its changeFrequency is lower.
changeFrequency decides how different chains interleave: once dependencies are satisfied, steps with lower values run first. Put stable preparation steps early in the chain so they can reuse the cache when only later steps change. The frequency is an ordinary non-negative number; the built-in constants make it easier to read:
dependsOn or capabilities to declare dependencies across chains, then frequency chooses among the ready steps. Each Plugin’s preparation array forms its own chain, with no automatic dependency edges to author layers.
When a script is longer than a few lines, put it in a separate file and use script instead of command:
script accepts a module-relative file: URL. Choose either script or command. The file must be inside the project root; the URL cannot contain a query or fragment, and a symlink cannot point outside the project. If the file is missing, unreadable, or not UTF-8 text, the run fails before creating a Sandbox and identifies the path.
NiceEval reads the script once during planning and freezes its original contents; execution does not read it again. The script contents and env values enter the fingerprint; the file path, file name, and label do not. Changing one byte reruns this step and every step after it.
Run progress shows only label, or id when the label is omitted. A label must be non-empty, contain no control characters, and be at most 256 Unicode code points. Built-in actions and custom families follow the same rule. Use niceeval debug <experiment> <eval> to see the full script, relative path, and digest; use niceeval show @<locator> --execution to see the commands actually executed.
Very short progress updates may be merged in the display. To observe progress step by step, split a long script into several named actions. NiceEval does not track steps inside shell scripts. Dynamic callbacks report their own progress with context.progress().
Declare preparation cache coverage
On supported Providers, unchanged preparation steps can be reused on the next run without executing again. For this, each.before(action) must truthfully state which parts of the Sandbox it changes:
- Omitting
cache.stateequalssandboxState.all: all of this step’s changes must be saveable and restorable. It is not a switch that lets the Provider pick some directories to cache. - Incus uses
sandboxState.allto save the complete preparation state, including the VM root disk and Docker data disk. There is no need to declare the Docker data directory separately.
cache.state can be written at the step definition or at a single call site, but only in one place, and a call site cannot override the definition.
The step’s ID, inputs, expanded command, declared scope, and content digest are compared automatically. Changes to execution content rerun the step. When execution order stays the same, changing the owning Experiment or declaration location alone does not invalidate identical cached content. cache.fingerprint only supplements version information NiceEval cannot see; it does not replace the automatic comparison.
Providers can save different scopes:
Reuse can only continue consecutively from the first step. At the first step that cannot be saved, such as a default
all the Provider does not support, an opaque callback, or a scope the Provider does not support, this step and every step after it really execute. Later steps cannot be skipped because the earlier state they depend on was not saved.
On local single-container Docker, when only the final .env step changes, the fixed Git checkout, directory upload, and tool installation before it do not rerun. Each Attempt starts from its own independent copy of the cached result and does not inherit changes left by the previous Agent.
To confirm which steps will be reused, run pnpm exec niceeval debug <experiment> <eval> to see the plan; see Check the Sandbox Plan Before Running.
Define reusable preparation steps
The built-inshell(), writeText(), uploadDirectory(), and gitCheckout() are all SandboxActions. When the same kind of preparation step is used in several places, you can define your own step with the same mechanism:
Sandbox. The rules above about cache scope and automatic comparison apply as well.
Writing, uploading, and Git checkout
These built-in functions can be passed directly to.before():
uploadFile() and uploadDirectory() take addresses such as new URL("fixture/", import.meta.url), not relative paths that change with the current directory. File contents, permissions, and directory structure are compared automatically. gitCheckout() only accepts HTTPS addresses without credentials and full commit IDs.
Do not put hidden content that should only be provided after the Agent returns into .before(). First get a handle with sandboxContent.file() or sandboxContent.directory(), then call t.sandbox.upload(handle, targetPath) in test(t). The handle carries only a content digest and does not expose host paths.
Dynamic callbacks and cleanup
For branches that can only be decided at run time, write.before(async (sandbox, context) => ...) directly. The callback really executes every time, and the steps after it no longer reuse the cache.
After successfully acquiring a resource, register its release right away with context.onCleanup(). The closure can hold the handle obtained this time directly, so you do not need another callback that guesses the resource’s state:
sandbox.sandboxId in the callback is the current Sandbox’s native ID on the Provider side. Normally treat it only as a value for correlation, and do not parse it. Only when this layer is explicitly bound to a particular Provider should your own trusted code hand it to the matching SDK to create helper resources for this Attempt.
These helper resources are not managed by NiceEval and are not cached; after creating them, you must immediately register a cleanup that can safely run more than once. Before the first callback starts, NiceEval may swap the underlying Sandbox instance; once a callback has started, this Attempt’s cleanup and .after() all see the same ID.
Cleanup that must run whether the run succeeds or fails, and can safely run more than once, goes in .after(). All registered cleanups run in reverse registration order, and a failing earlier cleanup does not stop later ones:
context.progress() in a callback only updates the current Attempt’s temporary status, and context.diagnostic() saves a problem to review afterwards. When the environment cannot continue, throw directly; NiceEval records this Attempt as an environment error, not as the Agent getting the task wrong.
A defineAgent without a Sandbox has no preparation steps. A Sandbox Agent can add command steps in the form of sandboxLayer() in its own sandbox field, but cannot choose or replace the Provider and template.
Prebuilt environments
When the same content is downloaded or installed for every Attempt, and it is stable, expensive, or large, bake it into a prebuilt environment and let Attempts start directly from the installed environment. The three Providers are written the same way:Retries for transient errors
OpenSandbox creation requests are not retried automatically: a connection failure may occur after the service has allocated an instance. The creation retries below apply to Docker, Vercel, and E2B. When the built-in Providers create a Sandbox and hit transient failures such as rate limits,fetch failed, connection resets, 5xx, or temporary network unreachability, they retry automatically with exponential backoff. Configuration errors such as a missing template or missing credentials fail on the first try. When retries run out, the Attempt is recorded as errored. The create of a custom defineSandbox Provider is your own function, and NiceEval does not retry it for you.
readText, readBytes, downloadFile, uploadFile, file writes, and directory uploads retry a limited number of times on transient transport errors such as 429, 5xx, fetch failed, and connection resets. Missing files, permission errors, cancellation, and a terminated Sandbox are not retried.
runCommand and runShell do not retry automatically, because a command may already have had side effects. Retry yourself in a hook or eval only when you have confirmed a command is safe to repeat.
Custom Providers
To connect another service, usedefineSandbox. You are responsible for isolating the filesystem, processes, network, and credentials yourself; NiceEval does not add these security boundaries automatically to an arbitrary Sandbox implementation.
The feedback that create receives reports status during creation, such as allocating an instance, pulling an image, or restoring a snapshot:
Sandbox interface. recommendedConcurrency is the default concurrency when nothing else is set.
Do not write a Provider SDK’s raw logs directly to the host process’s stdout / stderr: use feedback.progress for temporary status, feedback.diagnostic for problems that should be kept, and throw when the environment cannot be created. That way the terminal’s progress panel is not scrambled by logs, and CI has only one ordered output.
Permissions and execution identity
Providers differ in their limits on the executing user, network, filesystem, and process lifecycle. When writing a Fixture, avoid depending on the host environment as much as possible, and put dependencies inpackage.json or the Fixture’s preparation steps.
Performance tips
- Bake stable heavy dependencies into an image, template, or snapshot, or write them as low-frequency
.before()steps. - Reduce the size of the Fixture’s dependencies.
- Use small, explicit caches or pre-checks for dynamic content.
- Control
maxConcurrency(the Experiment field or--max-concurrency) to avoid exhausting local Docker resources. - Split slow checks: keep the ones that must pass as gates, and use
.optional()for the nice-to-haves.
Reuse Provider startup across Attempts
On Providers that support reuse, declaresandboxReuse: true in the Experiment when startup time dominates. Incus and OpenSandbox do not support this setting. Several Attempts share one Sandbox in turn: each Sandbox is created only once, each Attempt still runs its own .before() / .after(), and the working directory is reset between evals. On Docker, unchanged preparation steps can also be reused directly.