Skip to main content
An experiment chooses which agent to evaluate, which model to use, which flags to enable, and how many times to run. Use it for A/B tests, model comparisons, and feature comparisons — flags are exactly the feature flags of an A/B test, and one experiment is one set of flag values.

​
What an experiment contains

An experiment is a commit-ready TypeScript file under experiments/, and it comes down to a few fields:
  • agent: who to evaluate. Put a configured instance here — the subject-under-test URL and credentials go to the Adapter factory, not into other fields on the experiment.
  • model / flags: pass-through values. NiceEval does not interpret them. They are forwarded through ctx to the Adapter as-is, and the Adapter forwards them with the request and applies the switch as needed — this is exactly the channel Tier uses for model comparisons (Tier 1) and feature A/B tests (Tier 3).
  • attempts, budget, concurrency, sandbox, and other run parameters: how to run, and how much. See Write experiments for the full field list.
An experiment also has a pair of experiment-level hooks, setup / teardown. They run once for the whole experiment, on your own machine, to start and stop services shared across all attempts — for example, a tunnel to an internal service, or a mock server dedicated to the experiment. setup runs before the first attempt in this experiment is dispatched; teardown runs after all attempts finish (including on interruption), but only if setup’s point in the timeline was already reached. To prepare an environment inside the Sandbox on a per-experiment basis (install binaries, warm things up, carry state across attempts), attach that to the spec on the sandbox field instead — the object returned by factories such as dockerSandbox({ source: { type: "image", image: "node:24-slim" } }) can chain .setup() / .teardown(). For where the boundary between the two lies and how to write each, see Write experiments · Start experiment-shared services and Sandbox providers · Lifecycle. When evals in the same experiment need different pre-built environments, an eval only declares a provider-neutral environment profile; the environments table in the sandbox spec then maps that profile to a Docker image, an E2B template, or a Vercel Sandbox snapshot. This keeps task requirements in the eval and provider artifacts in the spec, so one experiment still covers every eval and the comparison doesn’t have to be split apart. See Write experiments · Use different pre-built environments for different evals for how to write this.

​
Matrix comparisons

Write one experiment file per variant you want to compare: two models means two files that differ by one model line; a prompt A/B means two files that differ by one parameter. Run the same eval suite once under each experiment and you get a comparable cross-section of pass rate, cost, and latency — stack them side by side in niceeval view. For what’s worth comparing and how to read the results, see Experiment matrix.
  • Write experiments — The full defineExperiment field set: attempts, budget, concurrency, and sandbox.
  • Experiment matrix — How to organize comparisons across agent / model / flags, and how to read the results.
  • Evals — The other half: what an eval is, plus its lifecycle and verdicts.
  • Tier — Which tier model and flags each take effect at.