Reuse a Sandbox Across a Batch

• Results from a reused run still enter the cache. When a pair has a stable carry identity and a terminal-result fingerprint matches, NiceEval adopts it directly and creates no Sandbox. A Plugin’s attachment owner, name, instance key, behavior revision, declared identity, host lifecycle shape, and Sandbox action declarations all belong to carry identity; changing any of them makes the slot fresh. Callback function bodies remain opaque. When behavior changes, update declared identity too or use --rerun all. Only unadopted Attempts execute in the shared Sandbox for this Invocation.
• Interruption does not roll back external state. Resuming is the same Experiment trajectory only when cross-Attempt state can return to the boundary of the last terminal commit. Otherwise, rebuild a clean cohort from the beginning.
• State outside the workdir remains. $HOME, /tmp, global installations, and background processes survive the between-task reset. Eval setup code must account for this, as described in Place preparation by what changes.
• It is incompatible with --keep-sandbox.

​

Enable it: three Experiment settings

import { defineExperiment } from "niceeval";
import { e2bSandbox } from "niceeval/sandbox";
import { codexAgent } from "niceeval/adapter";

export default defineExperiment({
  evals: ["react-datepicker/"],
  agent: codexAgent(),
  sandbox: e2bSandbox({
    template: "acme-evals",
    lifetimeMs: 60 * 60_000,   // ① Sandbox lifetime: required for reuse; omitting it fails before dispatch
  }),
  sandboxReuse: true,          // ② Declare reuse
  maxConcurrency: 3,           // ③ Several Sandbox lanes run concurrently; each lane accepts Attempts serially
  timeoutMs: 20 * 60_000,
});

​

Sandboxes are not shared between terminals

export default defineExperiment({
  sandbox: e2bSandbox({ template: "mempal" })
    .before(async (sandbox, context) => {
      await restoreMempal(sandbox);
      context.onCleanup(async (sandbox) => {
        await saveMempal(sandbox);
      });
    }),
  sandboxReuse: true,
  maxConcurrency: 1,
  sharedState: { key: "mempal/codex/cohort-a" },
});

​

Lifecycles: how often each runs

​

Place preparation by what changes

• Heavy dependencies every Experiment needs—an Agent CLI or language runtime—go into a Provider image, template, or snapshot, or into the lowest-frequency .before() actions.
• Preparation the whole batch shares—installing a toolchain, cloning a common repository, warming a build cache—belongs in low-frequency .before() actions. Docker can reuse the matching preparation prefix across Invocations. The current occurrence remains attempt; author code cannot request physical-instance promotion.
• Material only this task needs—its own repository, data, or dependencies—belongs in that Eval’s setup or test(t). It is replayed after every task reset, so replaying it must stay correct. When every task clones, use one named temporary clone directory for the whole Experiment, such as .niceeval-task-clone, and add it to diff.ignore. Before cloning, run rm -rf .niceeval-task-clone to clear only the previous task’s leftovers. Never remove workdir’s root .git: NiceEval needs it for the between-task git reset, and it survives that reset.
• A background process or occupied port must be cleaned up by the Eval’s own teardown; the between-task reset does not kill processes.

​

Idempotence is mandatory: one bad pattern and one good one

# Bad: detect something and skip the whole operation. When rustup was installed only halfway
# (cargo exists but no default toolchain is configured), detection passes, installation is skipped,
# and the next cargo build errors, making the whole Attempt errored.
if ! command -v cargo; then curl https://sh.rustup.rs | sh; fi

# Good: declare the target state directly. Replaying it any number of times converges on the same result.
rustup default 1.79.0
npm install          # Reconciles to the state declared by package.json/lockfile; it is itself idempotent.

​

Native Agent Plugins: installation converges in the Adapter

• A postSetup script must be idempotent. It reruns on a $HOME with leftovers for every Attempt. A script that registers a hook in global configuration must converge on the same configuration when it runs again. Before enabling reuse, run two or three tasks down one Sandbox lane; the second task onward is the real test.
• Where Plugin data lives. The Plugin installation directory is reinstalled and overwritten for every Attempt. Runtime data that must last until the next task can only live outside that directory. Whether retained data contaminates the next task is a question your Experiment design must answer.

​

A batch with different Sandbox layers

​

A typical rhythm

# 1. After changing common configuration, inspect the plan before spending money.
npx niceeval exp smoke memory/commit0 --dry

# 2. Quickly run an Experiment that declares sandboxReuse.
npx niceeval exp smoke memory/commit0

# 3. One task fails. Run it alone with an Experiment that does not declare reuse to keep the environment.
npx niceeval exp baseline memory/commit0/tool-first --keep-sandbox

# 4. After fixing it, verify with the Experiment that does not declare reuse and produce trustworthy results.
npx niceeval exp baseline memory/commit0

​

Cannot do it? Use concurrency, not reuse

export default defineExperiment({
  // Do not declare sandboxReuse: every Attempt gets a fresh Sandbox with no leftover state.
  maxConcurrency: 8,   // You still get parallelism.
  // ...
});

​

Related reading

• Eval Groups — explicitly declare members and ordering within a group while retaining concurrency between groups.
• Rerun and Carry Results — see how reused Sandboxes adopt terminal results and when to use --rerun.
• Concurrency and Execution Order — get the same speedup through concurrency when you do not reuse.
• Sandbox Provider Configuration — create Provider templates, images, and snapshots.