Run Docker Inside a Sandbox

Use incusSandbox() when the Agent needs docker build, docker run, or docker compose. Each Attempt gets its own VM. The Docker daemon runs inside the VM, and Docker data lives on a separate disk. NiceEval destroys the VM when the Attempt finishes. The eval declares its required capabilities, and the Experiment selects an environment that meets them. A host Docker socket, privileged container, or rootless DinD cannot provide the dedicated kernel required here.

Prepare the host

Ask an administrator to provision Incus, a trusted base image, and /etc/niceeval/incus-provider.json. The descriptor uses niceeval.incus-provider/v2 and records the environment, images, and capacity. The user running NiceEval must be able to read the descriptor and invoke the Incus CLI. NiceEval does not install host services, mount storage, or import base images. The base image must already exist and include the Docker CLI, Compose, and Docker daemon. Obtain the full image digest, project, and storage pool name from the administrator. In the eval project, check the reference environment:

pnpm exec niceeval sandbox provider doctor incus

doctor checks the environment, trusted image, object correspondence, and capacity without making changes. It does not create eval VMs or clean up resources. Run the experiment after all checks pass. If the administrator provided a development environment, use:

pnpm exec niceeval sandbox provider doctor incus --development

A passing development check does not imply the reference environment passes. Development results lack the reference environment’s capacity guarantees and cannot be compared directly with reference results.

Declare Docker capabilities in the eval

Put the capability requirements in the eval’s sandbox field. This example requires Compose v2, a dedicated kernel, and at least 4 GiB of Docker data space:

import { defineEval } from "niceeval";
import { sandboxRequirements } from "niceeval/sandbox";

export default defineEval({
  sandbox: sandboxRequirements({
    docker: {
      api: "docker/v1",
      compose: "v2",
      isolation: "dedicated-kernel/v1",
      minimumDataBytes: 4 * 1024 ** 3,
    },
  }),
  async test(t) {
    await t.send("Start the project with Docker Compose and verify that the service is available.");
  },
});

Set compose to "not-required" when Compose is unnecessary. The eval does not choose an image, project, or storage pool.

Select Incus in the Experiment

Replace the example digest below with the real digest supplied by the administrator:

import { incusSandbox } from "niceeval/sandbox";

const sandbox = incusSandbox({
  image: "niceeval/docker-execution-v1@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  project: "niceeval-eval",
  storagePool: "niceeval-evals",
  resources: {
    cpus: 4,
    memoryBytes: 6 * 1024 ** 3,
    dockerDataBytes: 4 * 1024 ** 3,
  },
});

Pass sandbox to defineExperiment(). Commands run as node, with /home/sandbox/workspace as the working directory. The Agent uses the Docker socket inside the VM and cannot access the host Docker daemon. For development, explicitly choose the corresponding project, pool, and permission:

const sandbox = incusSandbox({
  image: "niceeval/docker-execution-v1@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  project: "niceeval-eval-dev",
  storagePool: "niceeval-sandbox-dev",
  acceptDevelopmentDomain: true,
});

Omitting acceptDevelopmentDomain accepts only the reference environment. Setting it to true cannot be combined with the reference project’s name and pool. Inspect the plan before running:

pnpm exec niceeval exp my-experiment --dry

The plan shows the required capabilities, the Provider’s capabilities, and the selected environment. If the requirements are not met, NiceEval fails before creating a VM and does not automatically switch to another Provider.

Reuse preparation results

Write stable installations, file uploads, and Docker image preparation as declarative .before() steps with the default sandboxState.all. Incus saves and verifies the complete preparation result, including the VM root disk and Docker data disk. Each Attempt starts from its own copy. When execution order stays the same, moving a step’s declaration alone does not invalidate identical execution content. Changes to a command, input, or file content rebuild from the changed step onward. Dynamic callbacks always execute, as do the steps after them. See Prepare the Sandbox in order. Caching preparation results does not reuse a running VM. Incus does not support sandboxReuse: true or --keep-sandbox; both settings fail before resource creation.

Inspect and reclaim preparation caches

Preparation caches occupy disk space. doctor checks both object count and available bytes; spare object slots do not prove enough disk space is available. Inspect objects and storage first:

pnpm exec niceeval sandbox provider inventory incus --json

Add --development to every maintenance command for the development environment. These commands do not load project configuration, evals, or Experiments. Garbage collection has separate preview and apply steps. This command selects preparation caches unused for seven days:

pnpm exec niceeval sandbox provider gc incus --unused-for 7d --json

Review the candidates and estimated reclaimable bytes, then use the returned plan ID:

pnpm exec niceeval sandbox provider gc incus --apply <plan-id> --json

A plan expires after 15 minutes. Apply checks eligibility again; candidates that are now in use or whose object metadata has changed are not deleted. After a valid cache is reclaimed, a later run prepares it again. Without --unused-for, preview includes only eligible invalid or superseded records and interrupted retirements that can continue. NiceEval does not choose an age threshold for valid caches. Unregistered objects and records with mismatched object metadata are displayed for investigation. Do not directly delete Incus instances managed by NiceEval. If both an instance and its data volume have already been deleted manually, inspect the record with inventory, then preview and apply gc to reclaim it. After interrupted reclamation, create a fresh preview; the new plan checks resources again and continues eligible work.

Troubleshoot failures

Result	Next step
Missing Provider descriptor or trusted image	Ask the administrator to check deployment, permissions, and the full image digest
`sandbox-capability-unsatisfied`	Check eval requirements, the Experiment’s Provider, and development permission
`artifact-capacity` fails	Inspect preparation cache counts and records in use, then preview reclaimable entries
`artifact-storage` fails or is `unknown`	Check actual pool free space and read permissions; raising the object limit alone does not solve it
`dangling`	Objects are missing; preview and reclaim the record with `gc`
`orphan` or `metadata-drift`	Investigate ownership and configuration differences; GC does not delete these objects

For NixOS, also check the host deployment boundary. To connect an existing OpenSandbox service, see Choose and Configure a Sandbox Provider. A service running on Docker does not imply that it meets this page’s Docker capability requirements.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting