Organize Experiments: Compare Models, Prompts, and Agents

You wrote a new system prompt and want to know whether it is better or worse than the old one. The evals already exist; now you need to run them against both versions and look at the results side by side. That is what Experiments are for. An eval only says “what to send and what counts as correct”; an Experiment says “what to run it against”: which Agent, which model, which flags, how many times, and how much to spend before stopping. Keeping them separate means the same evals can compare any two configurations.

A minimal experiment

// experiments/local.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";

export default defineExperiment({
  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
});

The file path is the Experiment ID, here local. agent holds an already configured Adapter instance; connection parameters such as the address and auth of the application under test are passed to the Adapter factory and are not written anywhere else.

pnpm exec niceeval exp local

Compare two models

One Experiment file is one configuration. To compare two models, write two files and change only the model line:

experiments/
  models/gpt-5.4.ts
  models/deepseek-v4-pro.ts

// experiments/models/gpt-5.4.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../../agents/web-agent.ts";

export default defineExperiment({
  description: "gpt-5.4",
  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
  model: "gpt-5.4",
  attempts: 3,
});

model is passed to the Adapter as ctx.model. The Adapter must put it into the request it sends to the application, or the application will not really switch models; see Connect Your Agent. attempts: 3 runs each eval 3 times. The Agent answers differently every time, so a pass rate from a single run does not say much.

Compare two prompt versions

The prompt version lives inside the application, and NiceEval cannot see it. The approach is the same as A/B testing in a product: the application accepts a flag and switches the prompt based on it; the Experiment assigns the flag’s value.

// experiments/prompts/v1.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../../agents/web-agent.ts";

export default defineExperiment({
  description: "System Prompt v1 (baseline)",
  agent: webAgent({ baseUrl: "https://staging.example.com" }),
  model: "gpt-5.4",
  flags: { promptVariant: "v1" }, // The only difference between the two files
  attempts: 3,
  budget: 5,
});

experiments/prompts/v2.ts changes promptVariant to "v2" and is otherwise identical word for word. The less the two configurations differ, the more the score difference can be attributed to the prompt. The Adapter forwards ctx.flags to the application unchanged:

// agents/web-agent.ts
async send(input, ctx) {
  const r = await fetch(`${baseUrl}/api/turn`, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ message: input.text, model: ctx.model, flags: ctx.flags }),
    signal: ctx.signal,
  });
  // ...translate the reply into events
}

flags is a flat object whose values can only be strings, finite numbers, or booleans. Nested objects, arrays, null, and explicit undefined are all rejected before the run. Evals can read the same values with t.flags. When you need to check that flags are valid or fill in defaults, declare a synchronous parseFlags(input: unknown) on the Adapter. Its return value decides the type of flags in the experiment, and the normalized value takes part in the result reuse decision. The most common mistake is writing the same flags or the same model in both files, for example both reading the default of the same environment variable. Runs and reports look normal, but the two columns compare the same configuration.

Run a group, or just one cell

Positional arguments select Experiments by ID prefix. Directories only organize IDs and batch selection:

pnpm exec niceeval exp prompts      # All experiments under prompts/
pnpm exec niceeval exp prompts/v2   # Only the v2 cell

When one cell’s result looks odd and needs reproducing on its own, write its full ID; you do not need to run the whole group first or move other files away. Follow it with an eval ID prefix to run only a few evals:

pnpm exec niceeval exp prompts/v2 refund-policy

To confirm before running which evals run and how many times, add --dry; see Select and Preview the Run Scope.

See the comparison

After both configurations run, view each in the terminal:

pnpm exec niceeval show --experiment prompts/v1 --experiment prompts/v2

Each Experiment’s Summary lists these items:

  Observed   12/12
  Verdicts   8 passed; 4 failed; 0 errored; 0 skipped
  Pass rate  66.67%

Observed is the number of Attempts with results out of the planned number. When the two numbers differ, some Attempts have no result yet, and the pass rate cannot be compared directly with the other side. Besides the pass rate, look at duration, tokens, cost, and the kinds of failures. A configuration with a slightly higher pass rate at double the cost is not necessarily worth switching to. To compare two Runs side by side, see Compare Pinned Runs; to view in the browser, use niceeval view.

Make the comparison meaningful

Keep the set of evals unchanged during the comparison; otherwise the two sides are not comparing the same evals.
For highly non-deterministic Agents, especially Coding Agents, run each configuration at least 3 times.
Change one variable at a time. If you switch the model and the prompt together, you cannot tell which one deserves the credit.
Group failures by cause; do not just look at the total score.

Run only some evals

Each Experiment chooses which evals to run with evals, defaulting to all. You can write an array of ID prefixes, or a function:

evals: (eval) => eval.id.startsWith("coding/") && eval.tags.includes("coding"),

eval.id is the in-project ID derived from the file path, not an absolute path, so you can test it directly with startsWith.

Common fields

Field	Purpose
`agent`	Choose and configure the Adapter. Required
`model`	A single model name, passed to the Adapter through `ctx.model`
`reasoningEffort`	Reasoning effort, for example `"high"`, passed through `ctx.reasoningEffort` / `t.reasoningEffort`
`flags`	Experiment conditions, passed through `ctx.flags` / `t.flags`
`attempts`	The maximum number of runs per eval
`earlyExit`	Stop the remaining Attempts after one pass. Off by default; running all of them gives the full pass rate
`evals`	`"*"`, an array of ID prefixes, or a function returning a Boolean
`timeoutMs`	Timeout per Attempt
`budget`	This configuration’s spending cap, in US dollars
`maxConcurrency`	How many Attempts this configuration runs at once
`labels`	Annotations used only for report grouping; neither the Adapter nor evals can read them
`sandbox`	The Sandbox used by Coding Agents; see Sandbox
`plugins`	Reuse a complete set of run conditions; see Plugin
`setup` / `teardown`	Start and stop services shared by the whole experiment; see Start Shared Experiment Services
`sharedState`	An exclusive key for sharing state across runs; see Start Shared Experiment Services

For the full set of fields, see the defineExperiment reference.

Different evals need different environments

A batch of real tasks may need different runtime versions, for example some tasks need Python 3.9. In that case, write the Sandbox directly on the eval, so the task and the environment it needs live together:

// evals/astropy-2021.eval.ts
export default defineEval({
  sandbox: e2bSandbox({ template: "codex-python39" }),
  async test(t) {
    // Send the task and verify the result
  },
});

When several evals share the same environment, reuse it with an ordinary function:

// evals/shared/python39.ts
import { e2bSandbox } from "niceeval/sandbox";

export const python39 = () => e2bSandbox({ template: "codex-python39" });

The Experiment then only needs to specify the Agent, and the same Experiment can still cover every eval. Before creating any Sandbox, NiceEval checks that each eval’s environment is compatible with the Experiment’s configuration.

One application with several models

When the application has different internal roles such as planning, replying, and reviewing, each using its own model, assign them by purpose with models:

export default defineExperiment({
  adapter: workflowAdapter,
  models: {
    planner: { model: "provider/model-a" },
    responder: { model: "provider/model-b" },
    reviewer: { model: "provider/model-a" },
  },
});

The Adapter reads this configuration from ctx.models and hands it to the application, and on each model request it writes the matching modelRole in ctx.recordUsage. That way usage and cost can be counted separately by purpose; roles that are never called incur no cost. models cannot be written together with model or reasoningEffort. Agent-style Adapters use only a single model. After the run, use niceeval show @<locator> --usage to see each model’s usage and cost in one run. When Judge usage was not reported, it shows as unavailable, and the total cost gives only the known part.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting