> ## Documentation Index > Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Reuse Sandboxes with Eval Groups > Use defineEvalGroup to declare compatible members explicitly: one Sandbox is reused in stable ID order within a group, other groups keep running in parallel, and shared preparation belongs in the right Sandbox Layer. When a batch of evals shares a heavy toolchain, you usually need two things at once: do not create a Sandbox repeatedly for every task, but do not make the whole batch serial merely to reuse one. `defineEvalGroup()` puts compatible evals inside one physical reuse boundary. Attempts that really dispatch within a group run serially in stable order and reuse one Sandbox; other groups and ungrouped evals can still use other concurrency slots. | What you get | Runtime behavior | | -------------------------------------- | -------------------------------------------------------------------------------------------- | | Pay for common preparation fewer times | Sandbox creation and group-level `setup()` run only once per group instance | | Stable serialization within a group | The Runner orders normalized eval IDs and dispatches only one Attempt in the group at a time | | Preserve throughput between groups | Other groups and ungrouped evals can run in parallel | | Keep results comparable individually | Every Attempt still has its own Assertions, Verdict, usage, and File Changes | Eval groups schedule only Attempts that need to execute for real in this run. Carried results never enter the group's Sandbox; `--rerun`, `attempts`, and early exit keep their existing Experiment behavior. ## Choose the right reuse mode first | Task | Use | | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | | One or a few compatible eval groups need to reuse the same Sandbox while other groups should keep running in parallel | `defineEvalGroup()` | | All selected regular evals in an Experiment can run in any order and you only need several reuse lanes | `sandboxReuse: true` | | Every Attempt needs a fresh `$HOME`, `/tmp`, and background processes | Do not reuse Sandboxes; adjust concurrency only | One Experiment cannot both select eval groups and declare `sandboxReuse: true`. Eval groups already own their reuse boundary; when both appear, NiceEval reports `eval-group-sandbox-reuse-conflict` before the Provider creates a Sandbox. ## Step 1: Put the group file next to its members Place `eval-group.ts` in a named directory under `evals/`. The directory path becomes the eval-group ID: ```text theme={null} evals/ └── workflow/ ├── eval-group.ts # eval-group ID: workflow ├── 01-index/ │ └── eval.ts # eval ID: workflow/01-index └── 02-query/ └── eval.ts # eval ID: workflow/02-query ``` NiceEval discovers only `evals/**/eval-group.ts`. Do not name it `*.eval-group.ts` and do not put it at `evals/eval-group.ts`; the latter has no usable group ID. ## Step 2: Use a factory result to declare members Each member continues to default-export the result of `defineEval()` or `defineScoreEval()`. The group file imports those results, then lists compatible members in one closed set: ```ts theme={null} // evals/workflow/eval-group.ts import { defineEvalGroup } from "niceeval"; import buildIndex from "./01-index/eval.ts"; import queryIndex from "./02-query/eval.ts"; export default defineEvalGroup({ evals: [buildIndex, queryIndex], onUnavailable: "stop-group", }); ``` `evals` must not be empty. It accepts only objects actually returned by the factories—not eval IDs, directory prefixes, `glob`, `tag`, or `selector`. NiceEval does not collect files from the directory automatically either. Adding, removing, or reordering members is explicit in the `eval-group.ts` diff. Every member must have the same evaluation kind: all `defineEval()` results or all `defineScoreEval()` results. Discovery rejects a mixed group before Experiment filtering or Sandbox planning, and lists both sets of Eval IDs so you can split the group. The `evals` array declares members, not business order. The Runner always sorts normalized eval IDs stably; changing only the array positions does not change scheduling behavior or the group fingerprint. Put a result dependency such as “build first, then query” into one eval. Eval groups do not implement a business-order API. `onUnavailable` is required. `"stop-group"` stops later dispatch in the group when the physical Sandbox cannot be created, reset, or prepared; `"replace-sandbox"` retires the current instance first, then lets the next slot try to establish a replacement instance. Omitting the policy fails while loading `eval-group.ts`, so the Runner does not have to guess the cost and side effects after failure for its author. An eval can belong to at most one group, and it cannot appear more than once in the same group. Experiments and the CLI still select evals: selecting `workflow/02-query` does not pull `workflow/01-index` into this run. ## Step 3: Let the Experiment provide a reusable Sandbox Most projects let an Experiment choose the Provider and `template`; the eval group owns only the reuse queue: ```ts theme={null} // experiments/codex.ts import { defineExperiment } from "niceeval"; import { codexAgent } from "niceeval/adapter"; import { e2bSandbox } from "niceeval/sandbox"; export default defineExperiment({ evals: ["workflow/"], agent: codexAgent(), sandbox: e2bSandbox({ template: "codex", lifetimeMs: 60 * 60_000, }), maxConcurrency: 4, }); ``` Do not write `sandboxReuse: true` here. `maxConcurrency: 4` caps the whole Experiment; it does not run four Attempts from one group at once. It lets other eval groups or ungrouped evals use the remaining concurrency slots. Eval groups support only Sandbox Agents and Providers that support reuse. Direct Agents cannot run eval groups. ## Step 4: Check the group ID and selection result first Confirm selection with `--dry`; it creates no Sandbox: ```bash theme={null} npx niceeval exp codex --dry ``` The minimal project above shows: ```text theme={null} plan: 2 attempts · 2 evals × 1 run configuration · attempts 1 codex workflow/01-index [group workflow] new codex workflow/02-query [group workflow] new ``` After confirming that the two selected evals display the same group ID, run: ```bash theme={null} npx niceeval exp codex ``` The current scheduler orders members by normalized eval ID. When `attempts` is greater than 1, consecutive Attempts for one member enter the group lane before the next member. This is a stable scheduling rule, not a cross-eval data-dependency contract. Slots that early exit or result carry do not dispatch are skipped directly. ## Put common preparation on the eval group One Sandbox plan can receive declarations from the Experiment, eval group, and eval, but only one of the three layers can provide a template. The usual split is: | Declaration location | What belongs there | Execution frequency | | ---------------------------------------- | ---------------------------------------------------------------------- | ----------------------------- | | The Experiment's `sandbox` | Provider `template` that changes with the Agent or model configuration | Once per Sandbox in the group | | The eval group's `sandbox` + `.before()` | Shared toolchain, common checkout, and warmed build cache | Reused by action fingerprint | | A member's own `sandbox` + `.before()` | Starter files, dependencies, and public materials for only that task | Planned for each real Attempt | An eval group can provide its own `SandboxLayer`: ```ts theme={null} import { defineEvalGroup } from "niceeval"; import { changeFrequency, sandboxLayer, shell } from "niceeval/sandbox"; import buildIndex from "./01-index/eval.ts"; import queryIndex from "./02-query/eval.ts"; export default defineEvalGroup({ evals: [buildIndex, queryIndex], sandbox: sandboxLayer().before(shell({ id: "group-toolchain", command: "npm install --global tsx@4.22.4", changeFrequency: changeFrequency.rare, })), onUnavailable: "stop-group", }); ``` The group command context and Agent Context can both read `ctx.evalGroup.id` and `ctx.evalGroup.definitionHash`. Shared functions can also serve ungrouped evals, so the type keeps `evalGroup` optional. When you need to isolate a cache or service namespace outside `workdir`, first check that the field exists, then derive a key from the group ID. Members cannot provide a `template`, but they can contribute command-only `.before()` / `.after()` actions. Experiment, group, member, and Agent actions enter the same dependency graph. Once dependencies are satisfied, the smallest `changeFrequency` runs first, so a stable group action can precede a more volatile Experiment action. ## Do not treat eval groups as task dependency graphs Between Attempts, NiceEval resets `workdir` to the state after common preparation. A file written to `workdir` by one task is not an input to the next; `$HOME`, `/tmp`, global installations, and background processes can remain. Do not reuse a Sandbox if you cannot accept those leftovers. An eval group also does not guarantee that the preceding task executes. Result carry, CLI filtering, budget exhaustion, and interruption can leave members out of this run's Sandbox. When a later step must read a file from an earlier step, put both steps in one eval. Put evals in the same reusable group only when their shared state belongs outside `workdir` and every eval can independently produce a valid Verdict. ## Common errors | Error | Fix | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `eval-group-member-unresolved` | Import the default object returned by the factory from the eval entry point; do not reconstruct it or write a string ID. | | `eval-group-member-overlap` | Put every eval in only one group and list it once. | | `eval-group-member-layer` | Remove the Provider template from the member; keep command-only `.before()` / `.after()` actions. | | A load-time error says `onUnavailable` is missing | Explicitly choose `"stop-group"` or `"replace-sandbox"`; do not omit the failure policy. | | `eval-group-sandbox-reuse-conflict` | Remove `sandboxReuse: true` from the Experiment. | | `eval-group-direct-agent` | Use a Sandbox Agent or stop selecting grouped evals in this Experiment. | | `sandbox.reuse-unavailable` | Switch to a Provider that supports Sandbox reuse. | | `eval-group-incompatible` | Make members receive the same physical Sandbox plan, or split groups by compatible `template`. | ## Continue reading * [Reuse Complete Evaluation Conditions with Plugins](/docs/tutorials/plugins)—combine declarations needed by a group, Experiment, or member into an explicit occurrence while preserving the eval group's Docker Sandbox reuse boundary. * [Reuse Sandboxes](/docs/tutorials/sandbox-reuse)—the lifecycle and leftover-state boundary for ordinary batches that suit `sandboxReuse: true`. * [Tune Concurrency](/docs/tutorials/concurrency)—how eval groups work with global and Experiment concurrency caps. * [Configure Sandbox Providers](/docs/tutorials/sandbox-providers)—choose a `template`, set lifetime, and write a `SandboxLayer`. * [Rerun and Carry Results](/docs/tutorials/rerun-and-cache)—which Attempts enter this run's group queue.