> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Eval Groups to Let Related Evals Share a Sandbox

> When several groups of evals each need a heavy toolchain, group them with defineEvalGroup: each group shares one Sandbox and runs in sequence, while different groups run in parallel as usual.

Your eval set has 20 Rust tasks and 30 Python tasks. Every Rust task needs an environment with cargo and dependencies installed, which takes five minutes to set up. Installing it for every task is too slow; chaining all 50 tasks with `sandboxReuse` is also too slow, since the Python tasks have no reason to wait for the Rust tasks.

Eval groups solve this. Use `defineEvalGroup()` to put the 20 Rust tasks into one group: the group shares one Sandbox and runs its evals in ID order, while the Python tasks and other groups run in parallel as usual.

| What you get | How it behaves at run time |
| - | - |
| Pay for shared preparation fewer times | Each group Sandbox is created only once, and group-level preparation runs only once |
| Stable order within the group | The group is sorted by eval ID, and only one Attempt runs at a time |
| Groups still run in parallel | Different groups and ungrouped evals can run at the same time |
| Results can still be compared individually | Each Attempt still has its own assertions, verdict, usage, and file changes |

An eval group only schedules the Attempts that really need to run this time. Evals that reuse last results do not enter the group Sandbox; the rules for `--rerun`, `attempts`, and early exit are the same as without groups.

## Choose the right kind of reuse first

| Task | How |
| - | - |
| One or a few groups of compatible evals should share a Sandbox while other groups run in parallel as usual | `defineEvalGroup()` |
| All evals the Experiment selects can run in any order, and you only want a few lanes that share Sandboxes | `sandboxReuse: true` |
| Every Attempt needs a fresh `$HOME`, `/tmp`, and background processes | Do not reuse Sandboxes; only tune concurrency |

The same Experiment cannot both select eval groups and declare `sandboxReuse: true`. Both decide which evals share a Sandbox; when both appear, the run reports `eval-group-sandbox-reuse-conflict` before creating any Sandbox.

## Step 1: Put the group file next to its members

Create a directory under `evals/` and put the members and `eval-group.ts` in it. The directory's path is the group ID:

```text theme={null}
evals/
└── workflow/
    ├── eval-group.ts          # Group ID: workflow
    ├── 01-index/
    │   └── eval.ts            # Eval ID: workflow/01-index
    └── 02-query/
        └── eval.ts            # Eval ID: workflow/02-query
```

NiceEval only recognizes the file name `evals/**/eval-group.ts`. Naming it `*.eval-group.ts` means it is not discovered; putting it at `evals/eval-group.ts` does not work either, because it has no enclosing directory to serve as the group ID.

## Step 2: List the members in the group file

Each member default-exports the return value of `defineEval()` or `defineScoreEval()` as usual. The group file imports these return values and lists them in `evals`:

```ts theme={null}
// evals/workflow/eval-group.ts
import { defineEvalGroup } from "niceeval";
import buildIndex from "./01-index/eval.ts";
import queryIndex from "./02-query/eval.ts";

export default defineEvalGroup({
  evals: [buildIndex, queryIndex],
  onUnavailable: "stop-group",
});
```

When writing `evals`:

* It cannot be empty, and only accepts imported return values. Eval IDs, directory prefixes, `glob`, `tag`, and `selector` are not allowed.
* NiceEval does not collect files from the same directory automatically. Adding or removing members always shows up in the diff of `eval-group.ts`.
* Members must be either all `defineEval()` or all `defineScoreEval()`. When they are mixed, the run reports an error before planning Sandboxes and lists the eval IDs of both kinds, so you can split them into two groups.
* An eval belongs to at most one group, and cannot be listed twice in the same group.

Array order does not decide execution order. Within a group, evals always run sorted by eval ID, and moving array positions does not change scheduling. When you need a dependency such as "build first, then query", write both steps in the same eval. Ordering members by business sequence is not supported yet.

`onUnavailable` is required. It decides what happens when the group Sandbox cannot be created, reset, or prepared:

* `"stop-group"`: the rest of this group's evals are no longer dispatched.
* `"replace-sandbox"`: discard the current Sandbox and try creating a new one for the next Attempt.

Without it, loading `eval-group.ts` reports an error. Whether to spend more money rebuilding or to stop after a failure is your decision; NiceEval does not guess for you.

Which evals run is still decided by the Experiment and the command line. When only `workflow/02-query` is selected, `workflow/01-index` is not pulled into this run automatically.

## Step 3: Have the Experiment provide the Sandbox

In most projects, the Experiment chooses the Provider and `template`, and the eval group only decides "which evals share a Sandbox":

```ts theme={null}
// experiments/codex.ts
import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/adapter";
import { e2bSandbox } from "niceeval/sandbox";

export default defineExperiment({
  evals: ["workflow/"],
  agent: codexAgent(),
  sandbox: e2bSandbox({
    template: "codex",
    lifetimeMs: 60 * 60_000,
  }),
  maxConcurrency: 4,
});
```

Do not write `sandboxReuse: true` here. `maxConcurrency: 4` is the cap for the whole Experiment and does not make the same group run four at once; the remaining concurrency slots go to other groups and ungrouped evals.

Eval groups only work with Agents that run in a Sandbox and Providers that support reuse. An Agent without a Sandbox cannot run eval groups.

## Step 4: Confirm the grouping before running

First look at the plan with `--dry`; it creates no Sandbox and calls no model:

```bash theme={null}
pnpm exec niceeval exp codex --dry
```

Confirm that both member evals appear in the plan, then check whether they belong to the same group. In the machine-readable plan, each item carries `evalGroupId`:

```bash theme={null}
pnpm --silent exec niceeval exp codex --dry --json
```

Both evals' `evalGroupId` should be `workflow`. When the field is missing, first check that the group file's name and location are `evals/workflow/eval-group.ts`. After confirming, run:

```bash theme={null}
pnpm exec niceeval exp codex
```

Within the group, evals queue by eval ID. When `attempts` is greater than 1, the Attempts of one eval run back to back before the next eval's turn. Positions that do not need to run because of early exit or result reuse are skipped directly. This is only a stable queueing rule; it cannot be used to pass data between evals.

After the run, view results in the terminal with `pnpm exec niceeval show`, or in the browser with `pnpm exec niceeval view`.

## Put shared preparation on the eval group

Sandbox preparation can be written in three places: the Experiment, the eval group, and the eval. Only one of the three can choose `template`. A common split:

| Written on | What fits there | How often it runs |
| - | - | - |
| The Experiment's `sandbox` | A Provider `template` that varies with the Agent or model | Once per group Sandbox |
| The eval group's `sandbox` + `.before()` | Toolchains, shared repositories, and build cache warm-up used by the whole group | Reuses the last preparation result when content has not changed |
| A member's own `sandbox` + `.before()` | Starting code, dependencies, and public materials that belong only to this eval | Every Attempt that really runs |

Write the eval group's preparation steps as `sandboxLayer()`:

```ts theme={null}
import { defineEvalGroup } from "niceeval";
import {
  changeFrequency,
  sandboxLayer,
  shell,
} from "niceeval/sandbox";
import buildIndex from "./01-index/eval.ts";
import queryIndex from "./02-query/eval.ts";

export default defineEvalGroup({
  evals: [buildIndex, queryIndex],
  sandbox: sandboxLayer().before(shell({
    id: "group-toolchain",
    command: "npm install --global tsx@4.22.4",
    changeFrequency: changeFrequency.rare,
  })),
  onUnavailable: "stop-group",
});
```

Members cannot choose `template`, but they can add their own `.before()` / `.after()` commands. The preparation steps of the Experiment, eval group, members, and Agent are placed in one order: dependencies first, then ascending `changeFrequency`. So a rarely changing step in the eval group can come before a frequently changing step in the Experiment.

Lifecycle callbacks and Agent Context within a group can read `ctx.evalGroup.id` and `ctx.evalGroup.definitionHash`. The same function may also serve ungrouped evals, so `evalGroup` is an optional field. When you need to isolate caches or service names by group outside `workdir`, first check that the field exists, then build names from the group ID.

## Do not rely on group order to pass results

After each Attempt, NiceEval restores `workdir` to its state when shared preparation finished. Files an earlier eval wrote into `workdir` are not visible to a later one; `$HOME`, `/tmp`, global installs, and background processes may stay behind. When you cannot accept those leftovers, do not reuse Sandboxes.

An earlier eval may not even run. Result reuse, command-line filtering, an exhausted budget, and interruption can all keep a member out of the Sandbox this time. When a later step must read a file from an earlier step, write both steps in the same eval. Only put evals in the same group when what they share already lives outside `workdir` and each eval can reach a verdict on its own.

## Common errors

| Error | Fix |
| - | - |
| `eval-group-member-unresolved` | Import the object the member file originally default-exports; do not reconstruct the object or write a string ID |
| `eval-group-member-overlap` | Put each eval in only one group, and list it only once |
| `eval-group-member-layer` | Remove the Provider template from the member and keep only `.before()` / `.after()` commands |
| Loading reports a missing `onUnavailable` | Write `"stop-group"` or `"replace-sandbox"` explicitly |
| `eval-group-sandbox-reuse-conflict` | Remove `sandboxReuse: true` from the Experiment |
| `eval-group-direct-agent` | Switch to an Agent that runs in a Sandbox, or stop this Experiment from selecting grouped evals |
| `sandbox.reuse-unavailable` | Switch to a Provider that supports Sandbox reuse |
| `eval-group-incompatible` | Make the members' Sandbox configuration consistent, or split them into several groups by `template` |

## Read next

* [Reuse Complete Eval Conditions with Plugins](/docs/tutorials/plugins): write a full set of conditions needed by groups, experiments, or members once, and attach it where needed.
* [Reuse Sandboxes](/docs/tutorials/sandbox-reuse): when the whole batch suits sharing Sandboxes, the lifecycle and leftover boundaries of `sandboxReuse: true`.
* [Tune Concurrency](/docs/tutorials/concurrency): how eval groups work with the global and Experiment concurrency caps.
* [Sandbox Provider Configuration](/docs/tutorials/sandbox-providers): choose a `template`, set lifetimes, and write preparation steps.
* [Rerun and Reuse](/docs/tutorials/rerun-and-cache): which Attempts enter the group queue this time.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.