Reuse Sandboxes Across a Batch of Evals

• Reuse of last results still applies. When an eval’s configuration is the same as last time and it already has passed or failed, the old result is reused directly and no Sandbox is created for it. Only Attempts that really need to run this time enter the shared Sandbox.
• NiceEval cannot see changes to callback function bodies. When a Sandbox Plugin’s name, instance, behaviorRevision, declared identity, or preparation steps change, old results are invalidated and rerun. When you only changed code inside a callback, update behaviorRevision or identity as well, or use --rerun all.
• Interruption does not roll back external state. Resuming is equivalent to running straight through only when state shared across Attempts can be restored to just after the last completed Attempt. When that is not possible, start over with a clean copy of the state.
• State outside the working directory remains. $HOME, /tmp, global installs, and background processes all carry over to the next eval. Preparation code must accept this; see “Write preparation code correctly” below.
• It cannot be used together with --keep-sandbox.

Turn it on: three fields in the Experiment

import { defineExperiment } from "niceeval";
import { e2bSandbox } from "niceeval/sandbox";
import { codexAgent } from "niceeval/adapter";

export default defineExperiment({
  evals: ["react-datepicker/"],
  agent: codexAgent(),
  sandbox: e2bSandbox({
    template: "acme-evals",
    lifetimeMs: 60 * 60_000,   // ① Sandbox lifetime: required with reuse, or it errors before dispatch
  }),
  sandboxReuse: true,          // ② Declare reuse
  maxConcurrency: 3,           // ③ How many Sandbox lanes run in parallel; each lane takes several Attempts in turn
  timeoutMs: 20 * 60_000,
});

Sandboxes are not shared across terminals

export default defineExperiment({
  sandbox: e2bSandbox({ template: "mempal" })
    .before(async (sandbox, context) => {
      await restoreMempal(sandbox);
      context.onCleanup(async (sandbox) => {
        await saveMempal(sandbox);
      });
    }),
  sandboxReuse: true,
  maxConcurrency: 1,
  sharedState: { key: "mempal/codex/cohort-a" },
});

Lifecycle: what runs how many times

Write preparation code correctly: layer by “what it changes with”

• Heavy dependencies every experiment needs (Agent CLIs, language runtimes): bake them into the Provider’s image / template / snapshot, or write them as the lowest-frequency .before() preparation steps.
• Preparation shared by the whole batch (installing toolchains, cloning shared repositories, warming build caches): write them as low-frequency .before() preparation steps. On Docker, unchanged preparation steps can be reused directly on the next run. These steps are still passed through once per Attempt; you cannot ask for them to run only once per Sandbox.
• Materials only this eval needs (its own repository, data, dependencies): put them in the eval’s own before or test(t). They run again after each reset, so they must be code that is correct when repeated.
• Background processes started or ports taken: the eval’s after cleans them up itself; resetting workdir does not kill processes.

Idempotence is a hard requirement: one bad, one good

# Bad: skip the whole block when a probe succeeds. When rustup is half-installed (cargo present, no default toolchain set),
# the probe passes, installation is skipped, the next cargo build fails, and the whole Attempt errors.
if ! command -v cargo; then curl https://sh.rustup.rs | sh; fi

# Good: declare the target state directly; repeating it any number of times gives the same result.
rustup default 1.79.0
npm install          # Installs everything per package.json/lockfile and is itself safe to repeat

Agent-native Plugins: installation is handled by the Adapter

• before scripts must be safe to repeat. They run again on the carried-over $HOME every Attempt. A script that registers a hook in global configuration must produce the same configuration no matter how many times it runs. Before turning on reuse, run two or three evals through one Sandbox lane first; problems only show up from the second eval onward.
• Where plugin data lives. The plugin installation directory is reinstalled and overwritten every Attempt. Data that should carry over to the next eval can only be written outside the installation directory. Whether carried-over data affects the next eval is a question your experiment design has to answer.

A batch with mixed Sandbox configurations

A typical rhythm

# 1. After changing shared configuration, look at the plan before spending money
pnpm exec niceeval exp smoke memory/commit0 --dry

# 2. Go through quickly with an experiment that declares sandboxReuse
pnpm exec niceeval exp smoke memory/commit0

# 3. One eval fails: rerun it alone with an experiment that does not declare reuse, keeping the scene
pnpm exec niceeval exp baseline memory/commit0/tool-first --keep-sandbox

# 4. After the fix, verify with an experiment that does not declare reuse to get trustworthy results
pnpm exec niceeval exp baseline memory/commit0

When you cannot make it idempotent, use concurrency instead of reuse

export default defineExperiment({
  // No sandboxReuse: every Attempt uses a fresh Sandbox with no leftovers
  maxConcurrency: 8,   // Still runs in parallel
  // ...
});

Related reading

• Eval Groups: list group members explicitly; a group shares a Sandbox while groups run in parallel as usual.
• Rerun and Reuse: how last results are reused with Sandbox reuse, and when to use --rerun.
• Concurrency and Execution Order: how to get the same speedup with concurrency without reuse.
• Sandbox Provider Configuration: how to make each Provider’s template / image / snapshot.