Use Eval Groups to Let Related Evals Share a Sandbox

Your eval set has 20 Rust tasks and 30 Python tasks. Every Rust task needs an environment with cargo and dependencies installed, which takes five minutes to set up. Installing it for every task is too slow; chaining all 50 tasks with sandboxReuse is also too slow, since the Python tasks have no reason to wait for the Rust tasks. Eval groups solve this. Use defineEvalGroup() to put the 20 Rust tasks into one group: the group shares one Sandbox and runs its evals in ID order, while the Python tasks and other groups run in parallel as usual.

What you get	How it behaves at run time
Pay for shared preparation fewer times	Each group Sandbox is created only once, and group-level preparation runs only once
Stable order within the group	The group is sorted by eval ID, and only one Attempt runs at a time
Groups still run in parallel	Different groups and ungrouped evals can run at the same time
Results can still be compared individually	Each Attempt still has its own assertions, verdict, usage, and file changes

An eval group only schedules the Attempts that really need to run this time. Evals that reuse last results do not enter the group Sandbox; the rules for --rerun, attempts, and early exit are the same as without groups.

Choose the right kind of reuse first

Task	How
One or a few groups of compatible evals should share a Sandbox while other groups run in parallel as usual	`defineEvalGroup()`
All evals the Experiment selects can run in any order, and you only want a few lanes that share Sandboxes	`sandboxReuse: true`
Every Attempt needs a fresh `$HOME`, `/tmp`, and background processes	Do not reuse Sandboxes; only tune concurrency

The same Experiment cannot both select eval groups and declare sandboxReuse: true. Both decide which evals share a Sandbox; when both appear, the run reports eval-group-sandbox-reuse-conflict before creating any Sandbox.

Step 1: Put the group file next to its members

Create a directory under evals/ and put the members and eval-group.ts in it. The directory’s path is the group ID:

evals/
└── workflow/
    ├── eval-group.ts          # Group ID: workflow
    ├── 01-index/
    │   └── eval.ts            # Eval ID: workflow/01-index
    └── 02-query/
        └── eval.ts            # Eval ID: workflow/02-query

NiceEval only recognizes the file name evals/**/eval-group.ts. Naming it *.eval-group.ts means it is not discovered; putting it at evals/eval-group.ts does not work either, because it has no enclosing directory to serve as the group ID.

Step 2: List the members in the group file

Each member default-exports the return value of defineEval() or defineScoreEval() as usual. The group file imports these return values and lists them in evals:

// evals/workflow/eval-group.ts
import { defineEvalGroup } from "niceeval";
import buildIndex from "./01-index/eval.ts";
import queryIndex from "./02-query/eval.ts";

export default defineEvalGroup({
  evals: [buildIndex, queryIndex],
  onUnavailable: "stop-group",
});

When writing evals:

It cannot be empty, and only accepts imported return values. Eval IDs, directory prefixes, glob, tag, and selector are not allowed.
NiceEval does not collect files from the same directory automatically. Adding or removing members always shows up in the diff of eval-group.ts.
Members must be either all defineEval() or all defineScoreEval(). When they are mixed, the run reports an error before planning Sandboxes and lists the eval IDs of both kinds, so you can split them into two groups.
An eval belongs to at most one group, and cannot be listed twice in the same group.

Array order does not decide execution order. Within a group, evals always run sorted by eval ID, and moving array positions does not change scheduling. When you need a dependency such as “build first, then query”, write both steps in the same eval. Ordering members by business sequence is not supported yet. onUnavailable is required. It decides what happens when the group Sandbox cannot be created, reset, or prepared:

"stop-group": the rest of this group’s evals are no longer dispatched.
"replace-sandbox": discard the current Sandbox and try creating a new one for the next Attempt.

Without it, loading eval-group.ts reports an error. Whether to spend more money rebuilding or to stop after a failure is your decision; NiceEval does not guess for you. Which evals run is still decided by the Experiment and the command line. When only workflow/02-query is selected, workflow/01-index is not pulled into this run automatically.

Step 3: Have the Experiment provide the Sandbox

In most projects, the Experiment chooses the Provider and template, and the eval group only decides “which evals share a Sandbox”:

// experiments/codex.ts
import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/adapter";
import { e2bSandbox } from "niceeval/sandbox";

export default defineExperiment({
  evals: ["workflow/"],
  agent: codexAgent(),
  sandbox: e2bSandbox({
    template: "codex",
    lifetimeMs: 60 * 60_000,
  }),
  maxConcurrency: 4,
});

Do not write sandboxReuse: true here. maxConcurrency: 4 is the cap for the whole Experiment and does not make the same group run four at once; the remaining concurrency capacity goes to other groups and ungrouped evals. Eval groups only work with Agents that run in a Sandbox and Providers that support reuse. An Agent without a Sandbox cannot run eval groups.

Step 4: Confirm the grouping before running

First look at the plan with --dry; it creates no Sandbox and calls no model:

pnpm exec niceeval exp codex --dry

Confirm that both member evals appear in the plan, then check whether they belong to the same group. In the machine-readable plan, each item carries evalGroupId:

pnpm --silent exec niceeval exp codex --dry --json

Both evals’ evalGroupId should be workflow. When the field is missing, first check that the group file’s name and location are evals/workflow/eval-group.ts. After confirming, run:

pnpm exec niceeval exp codex

Within the group, evals queue by eval ID. When attempts is greater than 1, the Attempts of one eval run back to back before the next eval’s turn. Positions that do not need to run because of early exit or result reuse are skipped directly. This is only a stable queueing rule; it cannot be used to pass data between evals. After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view.

Put shared preparation on the eval group

Sandbox preparation can be written in three places: the Experiment, the eval group, and the eval. Only one of the three can choose template. A common split:

Written on	What fits there	How often it runs
The Experiment’s `sandbox`	A Provider `template` that varies with the Agent or model	Once per group Sandbox
The eval group’s `sandbox` + `.before()`	Toolchains, shared repositories, and build cache warm-up used by the whole group	Reuses the last preparation result when content has not changed
A member’s own `sandbox` + `.before()`	Starting code, dependencies, and public materials that belong only to this eval	Every Attempt that really runs

Write the eval group’s preparation steps as sandboxLayer():

import { defineEvalGroup } from "niceeval";
import {
  changeFrequency,
  sandboxLayer,
  shell,
} from "niceeval/sandbox";
import buildIndex from "./01-index/eval.ts";
import queryIndex from "./02-query/eval.ts";

export default defineEvalGroup({
  evals: [buildIndex, queryIndex],
  sandbox: sandboxLayer().before(shell({
    id: "group-toolchain",
    command: "npm install --global tsx@4.22.4",
    changeFrequency: changeFrequency.rare,
  })),
  onUnavailable: "stop-group",
});

Members cannot choose template, but they can add their own .before() / .after() commands. The preparation steps of the Experiment, eval group, members, and Agent are placed in one order: dependencies first, then ascending changeFrequency. So a rarely changing step in the eval group can come before a frequently changing step in the Experiment. Lifecycle callbacks and Agent Context within a group can read ctx.evalGroup.id and ctx.evalGroup.definitionHash. The same function may also serve ungrouped evals, so evalGroup is an optional field. When you need to isolate caches or service names by group outside workdir, first check that the field exists, then build names from the group ID.

Do not rely on group order to pass results

After each Attempt, NiceEval restores workdir to its state when shared preparation finished. Files an earlier eval wrote into workdir are not visible to a later one; $HOME, /tmp, global installs, and background processes may stay behind. When you cannot accept those leftovers, do not reuse Sandboxes. An earlier eval may not even run. Result reuse, command-line filtering, an exhausted budget, and interruption can all keep a member out of the Sandbox this time. When a later step must read a file from an earlier step, write both steps in the same eval. Only put evals in the same group when what they share already lives outside workdir and each eval can reach a verdict on its own.

Common errors

Error	Fix
`eval-group-member-unresolved`	Import the object the member file originally default-exports; do not reconstruct the object or write a string ID
`eval-group-member-overlap`	Put each eval in only one group, and list it only once
`eval-group-member-layer`	Remove the Provider template from the member and keep only `.before()` / `.after()` commands
Loading reports a missing `onUnavailable`	Write `"stop-group"` or `"replace-sandbox"` explicitly
`eval-group-sandbox-reuse-conflict`	Remove `sandboxReuse: true` from the Experiment
`eval-group-direct-agent`	Switch to an Agent that runs in a Sandbox, or stop this Experiment from selecting grouped evals
`sandbox.reuse-unavailable`	Switch to a Provider that supports Sandbox reuse
`eval-group-incompatible`	Make the members’ Sandbox configuration consistent, or split them into several groups by `template`

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting