Reuse Run Conditions with Plugins

When several Experiments share Skills, MCP servers, or preparation steps, define one Plugin and attach it through the plugins array. The Sandbox template remains in the Eval, Eval Group, or Experiment sandbox field.

Define an instance or factory

An object definition returns an instance. A function definition returns a single-argument factory. name and identity are required; key defaults to "default".

import { definePlugin } from "niceeval/plugin";
import { writeText } from "niceeval/sandbox";

export const memory = definePlugin(({ version }: { version: string }) => ({
  name: "example.memory",
  identity: { version },
  agent: { skills: [{ kind: "local", path: "./skills/memory" }] },
  sandbox: { before: writeText({ id: "memory-mode", path: "/tmp/memory-mode", text: version }) },
  experiment: {
    before: (ctx) => ctx.progress({ message: "Prepare memory service" }),
    after: (ctx) => ctx.progress({ message: "Release memory service" }),
  },
}));

export default defineExperiment({
  adapter: codexAgent(),
  plugins: [memory({ version: "1" })],
});

before and after accept one element or an array. Compose shared conditions with array spread: plugins: [...basePlugins, condition]. Each effective (name, key) pair must be unique for an Attempt, including attachments across Experiment, Eval Group, and Eval.

Choose a scope

Field	Timing	Allowed attachment
`agent`	Merged into a built-in Agent factory	Experiment
`experiment`	Once per Run with fresh Attempts	Experiment
`group`	Once per Group lane with fresh Attempts	Experiment, Eval Group
`sandbox`	Every fresh Attempt, after reset	Experiment, Eval Group, Eval
`attempt`	Every fresh Attempt	Experiment, Eval Group, Eval

When all results carry, no hooks run and no Sandbox is created. Before hooks run in author and Plugin declaration order. After hooks run in reverse activation order. Host after failures produce warnings and do not skip remaining cleanup. Agent after failures fail the Attempt while remaining after hooks and finalizers still run. Declare requires: { evalGroup: true } to access ctx.evalGroup.id directly. Selecting an Eval outside a Group fails before resources are created, including when all results would carry.

Merge Agent extensions

Use agent.skills and agent.mcpServers for common extensions. Use agent.codex, agent.claudeCode, and agent.bub for factory-specific settings. Factory configuration comes first, followed by Plugins in array order. Duplicate Skills, MCP names, native plugins, or config files fail validation. Bub does not accept MCP servers. Custom Adapters do not accept agent extensions. A Plugin with Agent-specific fields must include the selected Agent.

Rerun when conditions change

identity declares the condition version. Managed Action content, Skill content, and configuration file content also affect identity; MCP headers and env do not. Changing only a callback body in an Experiment module does not change Plugin identity. Update identity or use --rerun all when its behavior changes. Eval modules retain their source-closure rules. Plugin identity does not populate ctx.flags; declare runtime parameters explicitly in flags.

pnpm exec niceeval debug compare/codex memory/smoke
pnpm exec niceeval show --exp compare/codex
pnpm exec niceeval show --exp compare/codex --json

debug shows expanded steps, actual Action order, and Plugin sources. Experiment information in show and view lists Plugin name, key, and identity.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Reuse Run Conditions with Plugins

Define an instance or factory

Choose a scope

Merge Agent extensions

Rerun when conditions change

Next steps