Skip to main content
An Experiment decides what you run against, including the Agent under test, the model, and the run conditions. It is kept separate from Evals, so the same set of Evals can compare different configurations. For the complete approach, see Write an Experiment.

​
Experiment fields

​
models

Selects a model for each named model slot of a custom Adapter; cannot be combined with the single-model shorthand.

​
description

A one-sentence description shown in the view and CLI; purely informational, it does not affect scheduling or scoring.

​
agent

Shorthand for a conversational Adapter; exactly one of agent and adapter must be provided.

​
adapter

The Adapter implementation this Experiment uses.

​
model

A single model (decided by the Experiment when the agent leaves it empty); omitted = use the agent’s native default. To compare models, write several Experiment files instead of an array.

​
reasoningEffort

The model’s reasoning effort (such as “low”/“medium”/“high”; valid values depend on the specific model/adapter); omitted = use the agent’s native default. Passed to the adapter and eval through ctx.reasoningEffort.

​
judgeRuntime

This Experiment’s Judge execution configuration. It overrides only the model / endpoint / credential selector / call budget; the rubric is owned by the Match, and the material and consumption threshold come from the Assertion. Each field resolves in the order Experiment → Eval → Config → built-in default.

​
flags

Experiment conditions (the feature flags in an A/B), declared by the Experiment file; must be a flat scalar object (strings, finite numbers, or booleans only, validated when defineExperiment parses it). Passed to the adapter through ctx.flags, exposed to the eval as t.flags, and copied verbatim into the result snapshot’s ExperimentRunInfo.flags.

​
labels

Report grouping labels: the Experiment’s coordinates on each comparison axis (such as { line: "codex", memory: "mempal" }). Values are string | number (validated at parse time). The line between labels and flags is “does it change what happens in the attempt”: labels are not passed to ctx / t (the agent and eval cannot see them) and do not take part in the comparability configuration (changing them does not invalidate existing results); they are only projected verbatim into the snapshot’s ExperimentRunInfo.labels for report dimensions (label() / numericLabel()) to group by. The default report recognizes the line key: when any Experiment in the group declares it, scatter points are grouped by line and connected. See docs/feature/experiments/library.md “labels”.

​
attempts

How many times to repeat the same eval (each result counts as one attempt); defaults to 1 when omitted or when CLI --attempts overrides it.

​
earlyExit

Within a repetition (attempts > 1), whether to skip the remaining repeats after one attempt passes; defaults to false when omitted (attempts run in full to measure the complete pass rate). Turn it on explicitly to save cost when you only want to know whether it can pass.

​
evals

Which evals this Experiment covers: "*" for all, a string array matched by id prefix, or a custom predicate (which receives each discovered and fanned-out read-only EvalDescriptor, with no path / execution fields exposed); omitted is equivalent to "*". The predicate is evaluated once per candidate eval of this invocation, and the result becomes the in-memory selectedEvalIds plan, not a filter called repeatedly at runtime (see docs/feature/eval/library.md “EvalDescriptor” and docs/feature/experiments/library.md “evals: iterate discovery results, choose your own”).

​
timeoutMs

Overrides the project-level / CLI per-attempt timeout (milliseconds), only for this Experiment.

​
sandbox

The Sandbox declaration layer this Experiment contributes. It is linked pairwise with the same field of each selected Eval; in each pair, exactly one side provides a template-bearing layer.

​
sandboxCache

The Host’s execution cache policy; unlike the Sandbox declaration layer, this field does not take part in identity computation.

​
plugins

The list of Plugin instances this Experiment declares explicitly, normalized by defineExperiment().

​
sandboxReuse

Reuse Sandboxes within the same Run; such runs are isolated from history carry in both directions.

​
sharedState

Declares shared external state this Experiment needs exclusively. Invocations with the same key in the same project Coordination domain are mutually exclusive from before Experiment/Sandbox setup until Sandbox teardown, provider finalizers, and Experiment teardown have all finished; waiters do not create a Sandbox first, and continue their own plan after acquiring the lease. It is not a state store, a transaction, or a cross-machine lock.

​
budget

This Experiment’s estimated spend limit (USD). The scheduler accumulates the estimatedCostUSD of finished attempts; that value is computed from the model, token usage, and the runtime/config pricing table, independent of the observed usage.costUSD reported by the Provider / Adapter, which does not drive the budget. Once the estimate reaches the limit, the Experiment’s remaining attempts that have not started are skipped and run:budgetExceeded is reported once (attempts already in flight still finish).

​
maxConcurrency

This Experiment’s own concurrency limit within this Invocation: the scheduler throttles only this Experiment’s attempts; other Experiments in the batch are unaffected and still run under the global concurrency (CLI / env / config / Sandbox default). Use it to serialize Experiments with shared state (such as memory accumulated across evals: maxConcurrency: 1 guarantees attempts run one by one in eval order), or to slow down an Experiment that hits provider limits. A slot shares the attempt’s lifetime: it is held from before Sandbox creation until teardown and Sandbox destruction finish, and any wait along the way (including turn retry backoff) keeps holding it. maxConcurrency: 1 is therefore a strict critical section that the next attempt of the same Experiment cannot enter early. It does not span Invocations; for shared external state across Invocations, declare sharedState.

​
classifyFailure

This Experiment’s failure classifier: it recognizes your own shared-infrastructure causes of death that surface in third-party error shapes (such as connections refused by your own tunnel host), and returns undefined for “unknown, hand off to the rest of the chain”. Failures in any per-attempt phase of this Experiment are all routed to it; on the send failure chain it runs before the adapter’s classifySendFailure, because filtering by your own coordinates is more specific than the protocol’s generic shape, so the scope axis wins when both claim the failure. The classifier must be fast, pure, and must not throw (a throw falls back to undefined and is swallowed); it declares only the decision axis and the reason word, and retry and gating policy belong to the executor. See docs/feature/error-classification/library.md “Experiment / eval authors: declaring the blast radius of a cause of death”.

​
setup

The setup side of the Experiment-level lifecycle hook pair: at most once per run, on the host side, for host resources that are “one per Experiment, shared by all attempts” (tunnels, mock servers, license leases). Triggered by the first attempt of this Experiment that passes dispatch admission (memoized: concurrent attempts wait for the same result; not run when all results are carried in). setup returns no value; write products to module-level variables, and teardown and agent / sandbox hooks in the same file read them from the closure; the runner does not mediate values. If setup throws, every Attempt of this Experiment gets an errored Verdict (code "experiment-setup-failed", phase "experiment.setup"); other Experiments in the batch are unaffected. The function body does not enter the fingerprint; after changing hook logic, use --rerun all to explicitly rerun everything. See docs/feature/experiments/architecture.md “Experiment-level lifecycle”.

​
teardown

The teardown side of the Experiment-level lifecycle hook pair: runs after all of this Experiment’s attempts finish (also when the run is interrupted), if and only if execution reached the setup point. A setup throw does not exempt it (a half-initialized scene also needs cleanup; teardown should guard closure variables that may be unassigned); having no setup declared does not prevent it from firing; it is skipped when no attempt is dispatched. A throw or exceeding the 30s cleanup limit records only a run-level diagnostic (experiment-teardown-failed) and does not change the Verdict.