> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiment: who to evaluate and how to run

> An experiment is a commit-ready run configuration: which agent to evaluate, which model to use, which flags to enable, and how many times to run the same eval suite. Why evals and experiments are separate — late binding lets the same eval run against different subjects.

An **experiment** chooses which agent to evaluate, which model to use, which flags to enable, and how many times to run. Use it for A/B tests, model comparisons, and feature comparisons — `flags` are exactly the feature flags of an A/B test, and one experiment is one set of flag values.

## What an experiment contains

An experiment is a commit-ready TypeScript file under `experiments/`, and it comes down to a few fields:

```ts theme={null}
// experiments/concise.ts
export default defineExperiment({
  description: "Concise prompt variant",
  agent: webAgent({ baseUrl: "https://staging.example.com" }), // configured agent instance
  model: "gpt-5.4",                    // forwarded to the Adapter through ctx.model
  flags: { promptVariant: "concise" }, // forwarded through ctx.flags; visible in evals as t.flags
  attempts: 3,                         // how many times to run each eval (to counter randomness)
});
```

* `agent`: who to evaluate. Put a configured instance here — the subject-under-test URL and credentials go to the Adapter factory, not into other fields on the experiment.
* `model` / `flags`: pass-through values. [NiceEval](https://niceeval.com/) does not interpret them. They are forwarded through `ctx` to the Adapter as-is, and the Adapter forwards them with the request and applies the switch as needed — this is exactly the channel [Tier](/docs/explanation/tier) uses for model comparisons (Tier 1) and feature A/B tests (Tier 3).
* `attempts`, `budget`, concurrency, `sandbox`, and other run parameters: how to run, and how much. See [Write experiments](/docs/tutorials/write-experiment) for the full field list.

An experiment also has a pair of experiment-level hooks, `setup` / `teardown`. They run once for the whole experiment, on your own machine, to start and stop services shared across all attempts — for example, a tunnel to an internal service, or a mock server dedicated to the experiment. `setup` runs before the first attempt in this experiment is dispatched; `teardown` runs after all attempts finish (including on interruption), but only if `setup`'s point in the timeline was already reached. To prepare an environment inside the Sandbox on a per-experiment basis (install binaries, warm things up, carry state across attempts), attach that to the spec on the `sandbox` field instead — the object returned by factories such as `dockerSandbox({ source: { type: "image", image: "node:24-slim" } })` can chain `.setup()` / `.teardown()`. For where the boundary between the two lies and how to write each, see [Write experiments · Start experiment-shared services](/docs/tutorials/write-experiment#start-experiment-shared-services) and [Sandbox providers · Lifecycle](/docs/tutorials/sandbox-providers#lifecycle).

When evals in the same experiment need different pre-built environments, an eval only declares a provider-neutral `environment` profile; the `environments` table in the sandbox spec then maps that profile to a Docker image, an E2B template, or a Vercel Sandbox snapshot. This keeps task requirements in the eval and provider artifacts in the spec, so one experiment still covers every eval and the comparison doesn't have to be split apart. See [Write experiments · Use different pre-built environments for different evals](/docs/tutorials/write-experiment#let-different-evals-use-different-prebuilt-environments) for how to write this.

## Matrix comparisons

Write one experiment file per variant you want to compare: two models means two files that differ by one `model` line; a prompt A/B means two files that differ by one parameter. Run the same eval suite once under each experiment and you get a comparable cross-section of pass rate, cost, and latency — stack them side by side in `niceeval view`. For what's worth comparing and how to read the results, see [Experiment matrix](/docs/tutorials/experiments).

## Related reading

* [Write experiments](/docs/tutorials/write-experiment) — The full `defineExperiment` field set: attempts, budget, concurrency, and sandbox.
* [Experiment matrix](/docs/tutorials/experiments) — How to organize comparisons across agent / model / flags, and how to read the results.
* [Evals](/docs/explanation/evals) — The other half: what an eval is, plus its lifecycle and verdicts.
* [Tier](/docs/explanation/tier) — Which tier `model` and `flags` each take effect at.