Experiment: what to evaluate and how to run it

An Experiment says which Agent to run against, which model to use, which flags to turn on, and how many times to run. Use it to compare models, prompts, and feature toggles. flags are the feature flags of an A/B test, and one Experiment is one set of flag values.

What an Experiment contains

An Experiment is a check-in-able TypeScript file under experiments/. It has only a few core fields:

// experiments/concise.ts
export default defineExperiment({
  description: "Concise prompt variant",
  agent: webAgent({ baseUrl: "https://staging.example.com" }),  // A configured agent instance
  model: "gpt-5.4",                    // Passed through to the Adapter as ctx.model
  flags: { promptVariant: "concise" },  // Passed through as ctx.flags; t.flags in the eval
  attempts: 3,                         // How many times each eval runs (to counter randomness)
});

agent: what to evaluate. Put an already configured Adapter instance here. Pass the system’s address and authentication to the Adapter when you create the instance, not in other Experiment fields.
model / flags: experiment conditions handed to the Adapter. flags only accepts flat strings, finite numbers, or booleans, and NiceEval does not interpret their meaning. When the Adapter declares parseFlags, the values are validated and normalized first, then handed to the Adapter through ctx. The Adapter forwards them with the request, and the application switches behavior based on the values. Model comparison (Tier 1) and feature A/B tests (Tier 3) both use this path. See Integration tiers.
Run parameters such as attempts, budget, concurrency, and sandbox: how to run and how much. For the full field set, see Write Experiments.

An Experiment also has a pair of Experiment-level Hooks, setup / teardown. They run once for the whole Experiment on your own machine, and start and stop services shared by all Attempts, such as a tunnel to an internal service or a mock server just for this Experiment. setup runs before the first Attempt of this Experiment starts. teardown runs after all Attempts finish, and also runs when the run is interrupted. But teardown runs only if setup has already run. To prepare the environment inside a Sandbox for the whole Experiment, attach the preparation steps to .before() on the sandbox field. Steps whose content does not change can be reused next time. Dynamic callbacks can register cleanup with context.onCleanup(). See Start shared services for an Experiment and Sandbox · Prepare the Sandbox in a deterministic order. When different eval cases in the same Experiment need different prebuilt environments, write the sandbox configuration that names the prebuilt environment on each eval case, and leave it out of the Experiment. Reuse repeated configuration with ordinary TypeScript functions; no extra registration is needed. The same Experiment can still cover all eval cases, and NiceEval merges each eval case’s configuration with the Experiment’s configuration one by one. See Write Experiments · Different eval cases need different environments.

Matrix comparisons

Write one Experiment file for each variant you want to compare. Two models means two files that differ only in the model line. A prompt A/B test means two files that differ in one parameter. Run the same set of eval cases under each Experiment, then compare the results published at the same moment. Each Experiment computes its denominator from the eval cases it ran, and NiceEval does not force a common intersection. For what is worth comparing and how to read the results, see Write Experiments. Put values used only to group reports, which do not affect the run, in labels. They are recorded on the Run and are not passed to the Adapter or eval cases. Running again after changing labels creates a Run with the new labels, and unaffected earlier results are reused as usual. Historical Runs keep the labels they had at the time.

Write Experiments — Compare models and prompts, read comparison results, and common fields.
Start shared services for an Experiment — How to write setup / teardown and when they run.
Eval cases — The other half: what an eval case is, its lifecycle, and its verdicts.
Tier — The tier in which model and flags each take effect.

Core Model

Interaction and Scoring

Execution Model

Experiment: what to evaluate and how to run it

What an Experiment contains

Matrix comparisons

What an Experiment contains

Matrix comparisons

Related reading