Skip to main content
An Experiment specifies which Agent to evaluate, which model to use, which flags to enable, and how many times to run. Use it for A/B tests, model comparisons, and feature comparisons. flags are the feature flags in an A/B test; an Experiment is one set of flag values.

​
What an Experiment contains

An Experiment is a check-in-able TypeScript file under experiments/. Its core fields are:
  • agent: who to evaluate. This is an already configured instance. Pass the subject’s URL and authentication to the Adapter factory, not to other Experiment fields.
  • model / flags: pass-through values. NiceEval does not interpret them; it passes them through ctx to the Adapter, which forwards them with the request so the application can switch as needed. They are the channels for model comparison (Tier 1) and feature A/B tests (Tier 3) in Tier.
  • Run parameters such as attempts, budget, concurrency, and sandbox: how to run and how much to run. See Write Experiments for the full field set.
An Experiment also has an Experiment-level pair of setup / teardown Hooks. They run once for the entire Experiment on your own machine, so they can start and stop services shared by every Attempt, such as a tunnel to an internal service or an Experiment-specific mock server. setup runs before the first Attempt in this Experiment is dispatched. teardown runs after every Attempt finishes, including on interruption, but only if the timeline reached the point where setup should run. To prepare an environment inside a Sandbox for an Experiment, attach actions to the .before() chain on its sandbox field. Stable actions form a reusable preparation prefix by fingerprint. A dynamic callback can register cleanup immediately with context.onCleanup(). For the boundary between host Hooks and Sandbox actions, see Write Experiments — Start a service shared by an Experiment and Sandbox providers — Prepare the Sandbox in a deterministic order. When Evals in the same Experiment need different prebuilt environments, put the corresponding template-bearing sandbox layer directly on each Eval and omit the Experiment’s template. Reuse repeated layers with ordinary TypeScript helpers; there is no environment-profile registry. One Experiment can still cover every Eval because link planning pairs each Eval layer with the Experiment layer independently. See Write Experiments — Give different Evals different prebuilt environments.

​
Matrix comparisons

Write one Experiment file for every variant you want to compare. Two models mean two files differing by one model line; a Prompt A/B test means two files differing by one parameter. Run the same Eval suite under each Experiment. Fixed Inspection operations compare published Attempts within the same PublicationCutoff, while each Experiment keeps its own Eval set and denominator rather than fabricating a common intersection. See Experiment Matrix for what is useful to compare and how to read the results. Put values used only to group reports, and not to change what happens in an Attempt, in labels. They are immutable Run configuration context and are not passed to the Adapter or Eval. Running again after changing labels creates a Run with the new labels and can reference existing Attempts under the policy for that run. Historical Runs retain the labels they had at the time.
  • Write Experiments — The complete defineExperiment fields: attempts, budget, concurrency, and Sandbox.
  • Experiment Matrix — How to organize comparisons across Agent / model / flags and read the results.
  • Evals — The other half: what an Eval is, its lifecycle, and its Verdicts.
  • Tier — The tier in which model and flags each take effect.