# Eval an AI Agent Application with a Hand-Written Adapter
Source: https://niceeval.com/docs/examples/ai-agent-application
A runnable AI SDK v6 web agent eval project covering a hand-written adapter, tool calls, image understanding, multi-turn sessions, model comparison, and dual observability.
This project shows how to use `defineAgent` to connect an existing HTTP agent to NiceEval. It's a good fit for applications that need control over their own protocol and event mapping.
* [View the full source](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/ai-sdk)
* Local directory: `examples/zh/ai-sdk/`
If your application uses AI SDK v7's UI Message Stream, prefer the built-in `uiMessageStreamAgent` instead. See the corresponding non-intrusive example: [AI SDK v7](/docs/examples/integrations/ai-sdk-v7).
## What this project demonstrates
| Part | Implementation |
| - | - |
| System under test | AI SDK v6 web agent, HTTP `POST /api/turn` |
| Adapter | `defineAgent` + custom `AgentEvent` → `StreamEvent` mapping |
| Evals | Weather tool, image understanding, multi-turn text, multi-turn image context |
| Experiment | Each file pins one model; the group compares DeepSeek against GPT |
| Observability | The app keeps reporting to Langfuse while also sending this turn's OTLP data to NiceEval |
The adapter is only responsible for connecting and translating. The URL and model belong to the experiment; evals only describe the interaction and what a good result looks like. This separation lets the same evals be reused across different models and deployment instances.
## Directory structure
```text theme={null}
examples/zh/ai-sdk/
├── src/ # web agent under test
├── adapter/adapter.ts # hand-written NiceEval adapter
├── evals/ # tool, image, and multi-turn evals
├── experiments/ # model comparison
└── niceeval.config.ts
```
## Run it
This example calls real models — there's no mock mode. Copy the environment variables first and fill in credentials for an OpenAI-compatible service:
```bash theme={null}
cd examples/zh/ai-sdk
pnpm install
cp .env.example .env
pnpm dev
```
In a second terminal, run:
```bash theme={null}
cd examples/zh/ai-sdk
pnpm exec niceeval run list
pnpm exec niceeval exp compare-models weather-tool
pnpm exec niceeval view
```
## Next steps
* [Connect Your Agent](/docs/tutorials/connect-your-agent): Start from a minimal adapter and connect your own protocol.
* [Write Send](/docs/tutorials/write-send): Gradually fill in sessions, tools, HITL, and OTel.
* [Write Experiments](/docs/tutorials/write-experiment): Organize model and configuration comparisons.
# Eval Coding Agent Extensions
Source: https://niceeval.com/docs/examples/coding-agent-extensions
Evaluate whether Skills, prompts, and plugin benchmarks improve a coding agent's task results with a real workspace and baseline Experiment.
This example treats extension content as an Experiment variable: the same coding agent runs the same batch of real development tasks, and you compare task success rate, cost, latency, and behavior differences.
* [View the full source](https://github.com/CorrectRoadH/coding-agent-skill)
* [View the Fixtures guide](/docs/tutorials/fixtures)
## Two experiments
| Experiment | Comparison | What it tests |
| - | - | - |
| Zod Skill | `with-skill` vs `baseline` | Whether the Skill makes the agent reliably use `z.object().safeParse()`, instead of falling back to handwritten validation |
| Ponytail Benchmark | Baseline / Caveman / Ponytail / YAGNI One-liner | Whether a full decision Skill is more effective than a bare agent, a short-style Skill, or a one-line prompt |
The Ponytail group is migrated from a third-party plugin's agentic benchmark, but what this repository actually compares is the content and prompts injected into the agent. It does not prove whether a native plugin wrapper or install protocol is correct. For native Skill/plugin installation and configuration, see [Official Adapters](/docs/reference/official-adapters).
## Experiment design
Every Arm holds the same model, Sandbox, task set, Runs, and budget fixed, and only varies the injected content. The eval doesn't read which Arm is currently active, and it doesn't lower its acceptance bar for any experiment group.
```text theme={null}
coding-agent-skill/
├── skills/ # content injected by each Arm
├── workspaces/ts-starter/ # starting project for each eval
├── evals/ # real development tasks and verification
├── experiments/ # baseline and experiment Arms
```
Verification looks at final artifacts first: project tests, hidden probes, source code, and diffs. Only treat Skill load or tool calls as a Gate when the underlying events are stable and complete.
## Run it
```bash theme={null}
git clone https://github.com/CorrectRoadH/coding-agent-skill.git
cd coding-agent-skill
pnpm install
cp .env.example .env
docker info
pnpm exec niceeval exp ponytail-baseline
pnpm exec niceeval exp caveman
pnpm exec niceeval exp ponytail
pnpm exec niceeval exp yagni-oneliner
pnpm exec niceeval view --run --run
```
## What to reuse from this example
* Use an Experiment to express the comparison between having the extension and not, without letting the eval know which condition is active.
* Keep the prompt from leaking the answer; check tests, source code, diffs, and behavior evidence during verification instead.
* Use task success rate as the primary metric, and tokens, cost, latency, and behavior as explanatory metrics.
* [Evaluation kinds](/docs/tutorials/evaluation-kinds) owns the choice between Gate, Soft, and Judge; this page doesn't duplicate the assertion API.
# NiceEval Examples
Source: https://niceeval.com/docs/examples/index
Pick a runnable NiceEval example by subject under test: Agent Framework integrations, full AI agent evals, coding-agent extension benchmarks, and real projects.
Every example here has real, runnable source. Choose by subject under test — you don't need to read them in order.
If you haven't run your first eval yet, read [Quickstart](/docs/tutorials/quickstart) first. If you already know your task, go straight to the matching [Tutorials](/docs/tutorials/connect-your-agent) page.
## Connect an Existing Agent Framework
Each of the five examples below keeps both the plain application before integration and the full project after integrating NiceEval. The code diffs on each page are generated from those two sources.
| Subject under test | Integration path | Example coverage |
| - | - | - |
| [AI SDK v7](/docs/examples/integrations/ai-sdk-v7) | Built-in `uiMessageStreamAgent` | UI Message Stream, multi-turn, tools, HITL |
| [Claude Agent SDK](/docs/examples/integrations/claude-sdk) | `createClaudeSdkEventStream` | Native event stream, multi-turn, tools, HITL |
| [Codex SDK](/docs/examples/integrations/codex-sdk) | `createCodexThreadEventStream` | Coding tasks, files and commands, usage; the SDK doesn't support HITL |
| [pi-agent-core](/docs/examples/integrations/pi-sdk) | `createPiAgentEventStream` | Native event stream, multi-turn, tools, HITL; the SDK has no OTel |
| [LangGraph](/docs/examples/integrations/langgraph) | Hand-written custom SSE frame mapping | Python application, multi-turn, tools, HITL |
Every example's source lives under [`examples/zh/origin/`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin) and [`examples/zh/tier1/`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1). When you need OTel or Experiment Flags, follow the same project into `tier2/` and `tier3/`. Review each run's results locally with `niceeval show`, `query`, or `view`.
## Full Eval Projects
[Simulated X social platform](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/llm-x) shows how to evaluate a text and image generation application.
Personas, profiles, posts, and social interactions are generated by an LLM; the evals call application methods directly and check state and content relationships.
For the integration approach, see [Evaluate a custom LLM application](/docs/tutorials/custom-application).
An AI SDK v6 web agent covering tool calls, image understanding, multi-turn sessions, model comparison, and dual observability.
Use a real workspace and controlled experiments to measure how Skills, prompts, and plugin benchmarks affect task outcomes.
## Real Projects
### Coding Agent Memory Evals
[coding-agent-memory-evals](https://github.com/CorrectRoadH/coding-agent-memory-evals) uses the same batch of development tasks and the same model to compare task success rate, cost, latency, and behavior differences between a coding agent with persistent memory and one without.
* [View the source code](https://github.com/CorrectRoadH/coding-agent-memory-evals)
### Terminal-Bench
[terminal-bench](https://github.com/NiceEval/terminal-bench) runs the audited Harbor Terminal-Bench task set on NiceEval with the original `run-tests.sh` verifiers, and compares coding agents and models side by side.
* [View the source code](https://github.com/NiceEval/terminal-bench)
## Scope of Examples
* Single-snippet API usage belongs in a task tutorial or the Reference, not here.
* Toy projects without runnable source code are not included.
* Roadmap items and integrations that aren't implemented yet are not included.
* The same project doesn't get duplicated across multiple case pages just because it's also called a Skill, a Plugin, or a Hook.
# Integrate AI SDK v7 with NiceEval non-invasively
Source: https://niceeval.com/docs/examples/integrations/ai-sdk-v7
The complete before/after diff for connecting an AI SDK v7 chat app to NiceEval through its HTTP interface: zero app-side code changes.
Comparison targets:
* **before**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/ai-sdk-v7](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/ai-sdk-v7) -- a regular AI SDK v7 chat app (HTTP server + React chat UI), before any eval integration.
* **after**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7) -- the same app after integrating NiceEval.
**Integration path**: built-in **`uiMessageStreamAgent`** -- the official non-invasive adapter for the AI SDK UI Message Stream protocol (the standard SSE backend used by `useChat`). The adapter file only declares where the endpoint lives and how to pass `model` in the request body. Session replay, HITL approval (rewriting and resending `needsApproval` tool parts), and tool/message events built directly from protocol frames are all handled by the factory. The protocol frames do not include usage data, so this example does not assert usage. App-side `src/backend/*` files are byte-for-byte unchanged. Use `tier2/` for OTel waterfalls and `tier3/` for feature A/B.
All integration code changes (measured from both directories when generated):
## File list
```text theme={null}
ai-sdk-v7/
├── package.json modified
├── tsconfig.json modified
├── pnpm-workspace.yaml modified
├── niceeval.config.ts added
├── agents/
│ └── ai-sdk-v7.ts added
├── evals/
│ ├── basic-qa.eval.ts added
│ ├── hitl-approve.eval.ts added
│ ├── hitl-deny.eval.ts added
│ ├── session-isolation.eval.ts added
│ └── weather-tool.eval.ts added
└── experiments/
├── assistant.ts added
└── compare-models/
├── deepseek-v4-flash.ts added
└── deepseek-v4-pro.ts added
```
## App-side changes (dependency declarations only)
## Added adapter, evals, and experiments
# Integrate Claude Agent SDK with NiceEval non-invasively
Source: https://niceeval.com/docs/examples/integrations/claude-sdk
The complete before/after diff for connecting a Claude Agent SDK assistant backend to NiceEval: zero app-side code changes; everything is added on the eval side.
Comparison targets:
* **before**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/claude-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/claude-sdk) -- a standalone `@anthropic-ai/claude-agent-sdk` HTTP service, before any eval integration.
* **after**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/claude-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/claude-sdk) -- the same app after integrating NiceEval.
**Integration path**: official converter -- mapping Claude Agent SDK native `SDKMessage` events into standard events is handled by `createClaudeSdkEventStream` (exported from `"niceeval/adapter"`). The adapter only keeps transport glue and HITL stop-turn detection (`calculate` is gated through the official `canUseTool` callback). You start the app yourself in its normal way (`pnpm start`); evals do not manage the process. App-side `src/backend/*` files are byte-for-byte unchanged.
All integration code changes (measured from both directories when generated):
## File list
```text theme={null}
claude-sdk/
├── package.json modified
├── tsconfig.json modified
├── pnpm-workspace.yaml modified
├── niceeval.config.ts added
├── agents/
│ └── claude-sdk.ts added
├── evals/
│ ├── basic-qa.eval.ts added
│ ├── hitl-approve.eval.ts added
│ ├── hitl-deny.eval.ts added
│ ├── session-isolation.eval.ts added
│ └── weather-tool.eval.ts added
└── experiments/
└── assistant.ts added
```
## App-side changes (dependency declarations only)
## Added adapter, evals, and experiments
# Integrate Codex SDK with NiceEval non-invasively
Source: https://niceeval.com/docs/examples/integrations/codex-sdk
The complete before/after diff for connecting a Codex SDK coding-agent backend to NiceEval: zero app-side code changes.
Comparison targets:
* **before**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/codex-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/codex-sdk) -- a standalone `@openai/codex-sdk` HTTP service, before any eval integration.
* **after**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/codex-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/codex-sdk) -- the same app after integrating NiceEval.
**Integration path**: official converter -- mapping Codex native `ThreadEvent` events into standard events is handled by `createCodexThreadEventStream` (exported from `"niceeval/adapter"`). Message text, tool items (`command_execution` / `mcp_tool_call` / `file_change` -> paired `operation.started` / `operation.finished` events), and `turn.completed` usage all come from this stream, leaving only transport glue in the adapter. There is no HITL because Codex SDK does not support it. The evals run real coding tasks (writing files and running commands in a working directory), and assertions read the disk directly. You start the app yourself (`pnpm start`); evals do not manage the process. App-side `src/backend/*` files are byte-for-byte unchanged. Use `tier2/` for OTel waterfalls and `tier3/` for feature A/B.
All integration code changes (measured from both directories when generated):
## File list
```text theme={null}
codex-sdk/
├── package.json modified
├── tsconfig.json modified
├── pnpm-workspace.yaml modified
├── niceeval.config.ts added
├── agents/
│ └── codex-sdk.ts added
├── evals/
│ ├── basic-qa.eval.ts added
│ ├── create-file.eval.ts added
│ ├── run-command.eval.ts added
│ └── session-isolation.eval.ts added
└── experiments/
└── codex-sdk.ts added
```
## App-side changes (dependency declarations only)
## Added adapter, evals, and experiments
# Integrate LangGraph with NiceEval non-invasively
Source: https://niceeval.com/docs/examples/integrations/langgraph
The complete before/after diff for connecting a pure Python LangGraph app with LangSmith OTel export to NiceEval.
Comparison targets:
* **before**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/langgraph](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/langgraph) -- a pure Python `create_agent` (LangChain 1.x / LangGraph) HTTP service, before any eval integration.
* **after**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/langgraph](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/langgraph) -- the same app after integrating NiceEval.
**Integration path**: handwritten frame mapping -- custom JSON frames from `server.py` are translated one by one into standard events (`tool-input` -> `operation.started`, `tool-output` -> `operation.finished`, accumulated `text-delta` -> `message`, `tool-approval-request` -> stop turn + `input.requested`). HITL stop-turn state is stored through a typed `ctx.session` slot. The subject app is Python, while the eval side is a separate TypeScript project. App-side `src/backend/*.py` files are byte-for-byte unchanged. Use `tier2/` for OTel waterfalls and `tier3/` for feature A/B.
All integration code changes (measured from both directories when generated):
## File list
```text theme={null}
langgraph/
├── package.json added
├── tsconfig.json added
├── pnpm-workspace.yaml added
├── niceeval.config.ts added
├── agents/
│ └── langgraph.ts added
├── evals/
│ ├── basic-qa.eval.ts added
│ ├── hitl-approve.eval.ts added
│ ├── hitl-deny.eval.ts added
│ ├── session-isolation.eval.ts added
│ └── weather-tool.eval.ts added
└── experiments/
└── langgraph.ts added
```
## Added TypeScript-side scaffolding (the app itself is unchanged)
## Added adapter, evals, and experiments
# Integrate pi-agent-core with NiceEval non-invasively
Source: https://niceeval.com/docs/examples/integrations/pi-sdk
The complete before/after diff for connecting a pi-agent-core (@earendil-works) assistant backend to NiceEval: the app side only adds one devDependency.
Comparison targets:
* **before**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/pi-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/pi-sdk) -- a standalone `@earendil-works/pi-agent-core` HTTP service, before any eval integration.
* **after**: [https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/pi-sdk](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/pi-sdk) -- the same app after integrating NiceEval.
**Integration path**: official converter -- mapping pi native `AgentEvent` events into standard events is handled by `createPiAgentEventStream` (exported from `"niceeval/adapter"`). The adapter only keeps transport glue: which URL the app uses and which endpoint handles approvals (`calculate` goes through HITL approval). You start the app yourself in its normal way (`pnpm start`); evals do not manage the process. App-side `src/backend/*` files are byte-for-byte unchanged.
All integration code changes (measured from both directories when generated):
## File list
```text theme={null}
pi-sdk/
├── package.json modified
├── tsconfig.json modified
├── pnpm-workspace.yaml modified
├── niceeval.config.ts added
├── agents/
│ └── pi-sdk.ts added
├── evals/
│ ├── basic-qa.eval.ts added
│ ├── hitl-approve.eval.ts added
│ ├── hitl-deny.eval.ts added
│ ├── session-isolation.eval.ts added
│ └── weather-tool.eval.ts added
└── experiments/
└── assistant.ts added
```
## App-side changes (dependency declarations only)
## Added adapter, evals, and experiments
# Adapter: connect an Agent to NiceEval
Source: https://niceeval.com/docs/explanation/adapter
An Adapter is the integration layer you write. This page explains what the send function receives and returns, how to pass the configuration of the system under test, the three integration tiers, and where the capabilities on t come from.
In [NiceEval](https://niceeval.com/), an `Adapter` is the integration layer between the Runner and the system under test. An Adapter implements the `send` function: it sends eval-side actions such as `t.send()` to your application, then translates the application's response into [NiceEval](https://niceeval.com/)'s standard `Turn`.
Read next: [Write send](/docs/tutorials/write-send)
## Adapters
If the system under test uses the standard OpenAI Chat Completions or Responses protocol, use an official Adapter directly. For your own frontend/backend protocol (HTTP, gRPC, or WebSocket all work), write an Adapter. It knows how to authenticate, how to call your application, and how to translate the response into the standard event stream.
One complete `t.send()` round trip crosses this boundary twice:
On the way out: a drive verb in the eval case (`t.send` / `t.sendFile` / `t.respond`) builds the `TurnInput` and this session line's `ctx`, then calls the Adapter's `send` once.
On the way back: the Adapter translates the application's raw response into a `Turn`. Its `events` are an array of event objects in the order they happened (real values appear below).
The URL, authentication, and protocol details of the system under test are passed in through the Adapter factory's configuration. `defineExperiment` holds an already configured agent instance. The Adapter consumes this configuration in the `send` closure, then forwards per-turn dynamic values (such as `ctx.model`, `ctx.flags`, and `ctx.telemetry`) to the application with the request:
```ts theme={null}
// agents/web-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";
interface WebAgentOptions {
baseUrl: string;
apiKey?: string;
}
export function webAgent(options: WebAgentOptions): Agent {
const baseUrl = options.baseUrl.replace(/\/$/, "");
return defineAgent({
name: "web-agent",
evidenceCoverage: completeEvidenceCoverage,
async send(input, ctx) {
ctx.progress({ message: "Waiting for Web Agent" });
const response = await fetch(`${baseUrl}/api/turn`, {
method: "POST",
headers: {
"content-type": "application/json",
...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}),
...ctx.telemetry?.headers,
},
body: JSON.stringify({
message: input.text,
files: input.files,
model: ctx.model,
flags: ctx.flags,
}),
signal: ctx.signal,
});
if (!response.ok) {
ctx.diagnostic({
code: "agent-http-error",
level: "error",
message: `Web Agent returned HTTP ${response.status}`,
data: { status: response.status },
});
return {
status: "failed",
events: [{ type: "error", message: `HTTP ${response.status}` }],
};
}
const body = await response.json();
return {
status: "completed",
events: body.events,
data: body.data,
usage: body.usage,
};
},
});
}
// experiments/staging.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";
export default defineExperiment({
agent: webAgent({
baseUrl: "https://staging.example.com",
apiKey: process.env.STAGING_AGENT_API_KEY,
}),
model: "gpt-5.4",
flags: { promptVariant: "concise" },
});
```
## Integration tiers
Integration has three tiers based on where the Adapter connects and what extra observability data it gets: **Tier 1 connects only send** (no application code changes, and the full assertion set is already available at this tier), **Tier 2 is send + OTel** (the application sends spans to [NiceEval](https://niceeval.com/) in exchange for the call waterfall in `niceeval view`), and **Tier 3 changes the application and adds experiment flags** (feature A/B). For what each tier costs, what it provides, and when to move up, see [Tier](/docs/explanation/tier).
The eval-side drive APIs, `t.send()`, `t.sendFile()`, `t.newSession()`, and the HITL methods `t.respond()` / `t.respondAll()`,
all call the Adapter's `send`.
For how this converges and how to wire it inside send, see [Write send](/docs/tutorials/write-send).
## The send function
Whether the Adapter connects to an HTTP service or a CLI inside a Sandbox, the interface it exposes to the Runner is exactly the same:
```ts theme={null}
interface Agent {
readonly name: string;
send(input: TurnInput, ctx: AgentContext): Promise;
// Optional: setup / teardown (the lifecycle that runs once per Sandbox).
// Optional observability members do not take part in send; see "OTel integration"
}
```
`progress` is short-lived runtime state and is not saved to disk. `diagnostic` is a bounded warning/error that needs review after the run, and it is saved with the Attempt. Neither can specify a lifecycle phase, and neither changes `Turn.status` automatically. Throw an exception when the connection fails or the response cannot be parsed. The Runner records it as `errored` and prints a locator in the terminal, which you can use to see the details of this failure.
`send` is the only function you need to implement. Its signature has only three types, covered separately below.
The `setup` / `teardown` here are the lifecycle of how the Agent sets up its own connection. A Sandbox Agent can also contribute command-only `.before()` actions through its own `sandbox` field, for example to write the Adapter's public `.env`. The Experiment or eval case selects the one Provider template. For ordering and caching, see [Sandbox · Prepare the Sandbox in a deterministic order](/docs/tutorials/sandbox-providers#prepare-the-sandbox-in-order).
### Input: `TurnInput`
```ts theme={null}
interface TurnInput {
readonly text: string; // Present on every turn: the text sent. On a response turn = the answer text (multiple joined with newlines)
readonly files?: readonly InputFile[]; // Only on t.sendFile turns: attached files (base64)
readonly responses?: readonly InputResponse[]; // Only on response turns (t.respond / t.respondAll): one structured answer per request
readonly outputSchema?: JsonSchema; // Only when this turn declared output: the structured-output requirement, lowered to JSON Schema
}
interface InputFile {
readonly filename?: string; // Optional, for the adapter / model to refer to
readonly mimeType: string; // For example, image/png
readonly dataBase64: string; // Base64 content, JSON-friendly, can go straight into a request body
}
interface InputResponse {
readonly requestId: string; // Required: which input.requested request this answers; aligns answers when several requests pause together
readonly optionId?: string; // Either this or text: the answer matched an id in the request's options (approve / deny…)
readonly text?: string; // Either this or optionId: a free-text answer (the request has no options, or the answer is not any option)
}
```
Whether the eval side calls `t.send()`, `t.sendFile()`, or `t.respond()`, the Adapter receives one ordinary `send`. The difference is all in the fields:
If your application interface does not accept files, ignore `files`.
| Eval side | `text` | `files` | `responses` | `outputSchema` |
| - | - | - | - | - |
| `t.send(text)` | The sent text | — | — | Present only when `output` was declared |
| `t.sendFile(path, text?)` | A note (can be an empty string) | Present | — | Same as above |
| `t.respond(...)` / `t.respondAll(...)` | Answer text | — | One per request | Same as above |
Ignore `files` if the interface does not accept files.
Forward `outputSchema` if the application interface accepts a schema (Chat Completions `response_format`, Responses `text.format`). It is fine if it does not: validation happens in the Runner anyway (see the `Turn.data` rules below).
#### Inputs for the different answers
In a HITL response turn, a person's decision arrives in structured form with `input.responses`. The Adapter does not need to parse `text` to guess which answer belongs to which request, or whether it counts as approval. Every answer carries `requestId`. `optionId` and `text` are mutually exclusive: when the answer matches an id in the request's `options`, it is `optionId` (the eval side has already checked that it exists, so a typo cannot slip through silently). Otherwise the whole answer is free text in `text`. The four typical forms:
```ts theme={null}
// ① One pending request, the answer matches an option (approve / deny are the same, only optionId differs)
await t.respond("approve");
// → { text: "approve",
// responses: [{ requestId: "req_1", optionId: "approve" }] }
// ② Several requests pause together; the object form names the request explicitly
await t.respond({ request, optionId: "deny" });
// → { text: "deny",
// responses: [{ requestId: request.id, optionId: "deny" }] }
// ③ The answer is not any option → free text (requests waiting for more information look like this)
await t.respond("Change the recipient to ceo@corp.com");
// → { text: "Change the recipient to ceo@corp.com",
// responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] }
// ④ respondAll: one answer per pending request, all with the same optionId
await t.respondAll("approve");
// → { text: "approve\napprove",
// responses: [{ requestId: "req_1", optionId: "approve" },
// { requestId: "req_2", optionId: "approve" }] }
```
On the Adapter side, hand each decision back to the application by `requestId`; do not guess by order. A call rejected by a person sets the tool `operation.finished` `status` to `"rejected"`, not `"failed"`.
Rejection is a human decision, not a tool failure. This way `noFailedActions()` is not tripped by mistake, and `calledTool(toolMatch(..., { status: "rejected" }))` can assert it precisely.
### Context: `AgentContext`
```ts theme={null}
interface AgentContext {
// Feedback and runtime context present on every turn
readonly signal: AbortSignal; // Runner timeout and cancellation: pass it to every request you make
readonly session: AgentSession; // State slots for this session line
progress(update: { message: string; current?: number; total?: number }): void;
diagnostic(input: {
code: string;
level: "warning" | "error";
message: string;
data?: Readonly>;
dedupeKey?: string;
}): void;
// Three fields that appear by integration tier (see "Integration tiers")
readonly model?: string; // Tier 1 model comparison: experiment.model passed through. Forward it if the application interface accepts model selection
readonly reasoningEffort?: string; // Reasoning effort: experiment.reasoningEffort passed through, owned the same way as model
readonly telemetry?: Telemetry; // Tier 2: this turn's W3C trace context (headers, a new traceparent each turn)
readonly flags: Readonly>; // Tier 3 feature A/B: experiment.flags passed through. {} when not configured
readonly experimentId?: string; // Experiment id derived from the path, same source as experimentId in results. undefined when not run through an experiment
}
interface SandboxAgentContext extends AgentContext {
readonly sandbox: Sandbox; // Available only to Agents built with defineSandboxAgent
}
interface AgentSession {
// Session continuation: the server keeps history (the shape for interfaces that accept a session id)
readonly id?: string; // The session id recorded on this line. undefined on a new session line
capture(id: string | undefined): void; // Record a returned id. Only takes effect if none was recorded yet
// Adapter-private state: client-side history, a paused HITL turn, and so on
get(slot: SessionSlot): T | undefined;
set(slot: SessionSlot, value: T): void;
take(slot: SessionSlot): T | undefined; // Clears on read, consumed once
}
```
The Runner binds a separate lifecycle scope for the Agent's `setup`, each `send`, and `teardown`. The Adapter only reports progress/diagnostic from inside the current callback, and cannot pass a phase, color, or output stream. `progress` is not saved to disk. `diagnostic` is committed to Record as an Attempt-owned diagnostic-channel event. When it cannot continue, throw; the Runner saves a structured error and forms an `errored` verdict on the `niceeval.verdict` channel. The Attempt lifecycle state is only `active` / `completed` / `abandoned`; a verdict token is not an Attempt state.
`ctx` has no flags for you to check. The three tier fields are all pass-through: `model` and `flags` are declared by the Experiment and handed over unchanged by the Runner. The Adapter only forwards them to the application with the request and does not interpret their meaning. `telemetry` appears only when OTel integration is configured; in `send`, just spread `headers` into the request headers. The receiving endpoint is fixed in `defineConfig` and pointed to when the application starts, not passed from here. See [OTel integration](/docs/tutorials/connect-otel). `experimentId` is a stable identifier derived from the path. You can use it in dynamic Sandbox callbacks to isolate cross-Attempt state per experiment. See [Sandbox · Prepare the Sandbox in a deterministic order](/docs/tutorials/sandbox-providers#prepare-the-sandbox-in-order).
`session` is the state owned by this session line. [NiceEval](https://niceeval.com/) promises only one thing about it: **every `send` on the same session line gets the same `ctx.session`, and a new session line (the eval case's first turn, or after `t.newSession()`) gets a brand new one.** Continue server-side history with `id` / `capture`. For client-side history and paused HITL turns, the Adapter creates a typed slot at module scope with `createSessionSlot(name)`, then reads and writes it with `get` / `set` / `take`. `take` clears the value as it reads it. Slots are isolated by symbol identity, so they never collide even with the same name.
### Return: `Turn`
```ts theme={null}
interface Turn {
readonly events: StreamEvent[]; // ★ The standard event stream: every Adapter's core output
readonly data?: JsonValue; // This turn's structured output. Fill it only when input.outputSchema is present; the Runner validates it against the declaration
readonly status: "completed" | "failed" | "waiting"; // waiting = the turn paused for a person (HITL)
readonly usage?: Usage;
}
```
`events` is an **ordinary JS array**: one object for each thing that happened in this turn, in the real order it happened. For example, in a turn asking "What's the temperature in Beijing today?", the agent looks up the weather and then answers, and `events` looks like this:
```ts theme={null}
[
{ type: "operation.started", operationId: "c1",
operation: { kind: "tool", name: "get_weather", input: { city: "Beijing" } } },
{ type: "operation.finished", operationId: "c1", kind: "tool",
output: { temp: 21 }, status: "completed" },
{ type: "message", role: "assistant", text: "Beijing is 21°C today." },
]
```
There are ten object variants in total (`message`, `operation.*`, `input.requested`, and so on; see the [Event stream reference](/docs/reference/events) for the full list). Assertions read exactly this array: `t.calledTool("get_weather")` counts started tool operations, and `t.reply` takes the last assistant `message`. Note that each `send` returns only **this turn's** array. Joining turns into a whole session line is the Runner's job; see the next section. Most of the time you do not write these objects by hand either: an official converter returns a complete `Turn` with `events`, `usage`, and `status` filled in. For choosing a converter, see [Write send](/docs/tutorials/write-send).
`data` is not a pocket for "whatever". It has only one rule: **the eval case declares what it wants with a schema on send. `data` exists only when declared, and it has the declared type when you get it.**
```ts theme={null}
// Eval side: the output declaration is both a requirement for the application and the type source of turn.data
const turn = await t.send("What is the total on this invoice?", {
output: z.object({ amount: z.number(), currency: z.string() }),
});
turn.data.amount; // Type is number, not unknown; no manual narrowing
```
The declaration is lowered to JSON Schema and reaches the Adapter through `input.outputSchema`. After the Adapter gets the application's response, it puts the structured output in `data` (for example, `JSON.parse(reply.content)`). **The Runner enforces validation**: if `data` does not match the declaration, the turn is immediately `failed` and reports the difference. There is no path where a wrong object slips in and an assertion silently passes. Conversely, a turn that did not declare `output` has no `data`: an application that only returns text never touches this field. Do not copy the raw response body into it to fill space.
### After send returns: where the four Turn fields go
Once `send` returns, the Adapter's work is done; everything else is the Runner's job. Each of the four fields has a clear destination:
| Turn field | What the Runner does with it | Where the eval author sees it |
| - | - | - |
| `events` | Appends it to this session line and normalizes it into messages and logical tool occurrences. The last assistant `message` also updates `t.reply` | Scoped assertions such as `turn.calledTool()` and `turn.event(...)`; `t.events` can be inspected directly |
| `status` | Records this turn's final state. On `"waiting"`, it collects this turn's `input.requested` into the pending list | `t.succeeded()` or `t.check(turn.status, equals("waiting"))`. After `"waiting"`, continue with `t.respond()` |
| `usage` | Adds it up per turn for the session line and the whole run | `maxTokens` / `maxCost` assertions and usage in reports |
| `data` | Validates it against the `output` schema declared on send; a mismatch makes the turn `failed` immediately. Saved only if it passes | `turn.data` (strongly typed by the declaration), `t.check(turn.data, equals(...))` or `matches(...)` |
The handle you get from `await t.send()` is a view of this one Turn: turn-level assertions see only this turn's `events` and fields, and `t`-level assertions see what has accumulated on the whole session line. So the Adapter does not need to "cooperate" with any assertion. Fill these four fields correctly, and assertions, scoring, and reports all happen downstream automatically.
## Where capabilities come from: proof by construction, not a questionnaire
`defineAgent` builds a Direct Agent: the Runner calls a function, SDK, or service endpoint directly, and the target can be local or remote.
A Direct Agent's `setup`, `send`, and `teardown` never get a Sandbox. `defineSandboxAgent` builds a
Sandbox Agent, and only its callbacks get a `SandboxAgentContext` with a real `sandbox`.
| What it unlocks on `t` | What proves it | What you write |
| - | - | - |
| `t.send`, `t.sendFile`, `t.check`… | Every Agent has them | Nothing |
| Multiple `t.send()` calls, `t.reply`, `t.newSession()` | `send` uses the `ctx.session` continuation accessors (typed slot or `id` + `capture()`); without them every turn is a new conversation | Nothing; the accessors are already on `ctx` |
| `t.respond()`, status checks, and other HITL | send has returned `"waiting"` + an `input.requested` event | Nothing; doing it means having it |
| Positive assertions such as `turn.calledTool()` / `turn.toolOrder()` | events contain paired `operation.started` / `operation.finished` | Nothing; if the events exist, you can assert |
| Negative assertions such as `t.notCalledTool()` / `t.usedNoTools()` being **trustworthy** | Events come from an official converter with a completeness proof | Nothing; the proof travels with the converter's return value |
| `t.sandbox`, `t.sandbox.fileChanged()`, and similar | Built with `defineSandboxAgent` | Nothing |
| Trace channel and the `niceeval view` waterfall | A `tracing` block exists (spans only go into the waterfall, not into assertions) | Nothing; you write the block anyway |
The negative-assertion row is the only one with levels. `notCalledTool` asserts that something "did not happen", which requires "all events". That holds only when the source itself has a completeness contract (direct SDK native event streams, AI SDK `result.steps`, Responses `output`). A handwritten mapping has no proof, so a negative assertion warns at runtime that it is not trustworthy instead of silently passing. So there is no gap of "declared but cannot deliver" either: trust follows the source of the events, not the author's diligence. For exact obligations per capability, see the [Capability reference](/docs/reference/capabilities).
## Related reading
* [Write send](/docs/tutorials/write-send) — The hands-on tutorial: from sending one message to a full integration in seven steps, each unlocking a set of assertions.
* [Connect your Agent](/docs/tutorials/connect-your-agent) — The integration overview: minimal integration, parameter channels, and the incremental map.
* [Drive](/docs/explanation/drive) — The eval-side view: how to use `t.send()`, `t.newSession()`, and HITL.
* [Assert](/docs/explanation/assert) — The full assertion vocabulary driven by the standard event stream.
* [Overall model](/docs/explanation/overview) — How eval cases, Experiments, Adapters, the Runner, and Record connect into one run.
# NiceEval Assertions and Matches
Source: https://niceeval.com/docs/explanation/assert
A Match describes how to compare a value. Calling a check method registers one Assertion immediately, and the methods chained after it only configure that one.
NiceEval evaluates by Assertion, and one Assertion is one check. When you call `t.check(value, match)`, `t.judge(value, definition)`, `turn.succeeded()`, or `calledTool(...)`, NiceEval registers one Assertion immediately.
There is no step where you first create a check object and then hand it to another API to run. The handle returned by the call only configures the Assertion just registered, for example to add a label, set a gate, or score it.
```ts theme={null}
import { includes } from "niceeval/expect";
t.check(turn.message, includes("Completed")).label("Reply says it completed");
turn.succeeded().label("Turn completed");
turn.calledTool("get_weather").label("Looks up weather");
t.judge({ question, answer: turn.message }, answerQuality)
.gate(0.8).label("Answer quality");
```
Each Assertion's `key` and `label` can each be set at most once. Setting either again is an error, even when both writes use the same value.
## Matches only compare
Everything `niceeval/expect` exports is a Match, and a Match is one comparison rule. It is reusable and never changes; the same input always gets the same result, and it has no side effects. A Match itself carries no gate, score, or control flow. Those are all configured on the handle.
`t.check(value, match)` takes only these two arguments. It reads `value` at the call and registers the Assertion.
```ts theme={null}
import { equals, includes, matches, satisfies, toolMatch } from "niceeval/expect";
t.check(turn.message, includes("Order confirmed"));
t.check(turn.data, equals({ status: "refund", amount: 42 }));
t.check(turn.data, matches(OrderSchema));
t.check(turn.message.length, satisfies("Reply is not empty", (length) => length > 0));
```
Whether the result is `matched` or `mismatched` only says whether the value satisfies this rule. It does not say whether the rule itself is written correctly. When a task allows several correct results, the rule must cover all of them.
`equals(expected)` does a deep equality comparison; for strings, it requires an exact match. If the task accepts several equivalent forms, first parse or normalize the output into the value the task cares about, then compare. Compare strings directly only when the literal form is itself the task requirement.
## Check on a Turn, a Session, or the whole eval
Methods such as `succeeded` and `calledTool` register one Boolean Assertion when called. The object you call them on decides which evidence they see:
* `turn.*` sees only that one Turn.
* `session.*` sees everything in that Session up to the call.
* `t.*` sees every Session that has started at the time of the call.
```ts theme={null}
const turn = await t.send("Run checks and fix the problem.");
turn.succeeded().label("Turn succeeded");
turn.calledTool("shell").label("Uses shell");
turn.calledTool(toolMatch("read_file").exactly(2)).label("Reads twice");
```
`calledTool` does not take a count argument. Put the count on `toolMatch`, for example `.exactly(2)`. Conditions on `input`, `output`, and `status` also go in `toolMatch`: use `jsonMatch` for JSON structure and `commandMatch` for command arguments. `notCalledTool` accepts the same matcher, but without a count.
## Pass/fail and scored
`defineEval` is pass/fail. Boolean checks directly affect the Attempt's verdict: one `mismatched` makes the final verdict `failed`, but later checks still run and register.
Continuous-value checks and Judges give measurements between 0 and 1. By default, measurements do not affect the verdict. After you call `.gate(minimum)` on the handle, falling below the minimum gives `failed`.
```ts theme={null}
t.judge({ task, answer: turn.message }, actionableQuality)
.gate(0.8).label("Actionable steps");
```
`defineScoreEval` is scored. It has both a verdict and an accumulated `score`. The score is the sum of each item's contribution, with no implied total or percentage.
By default, an Assertion only records its result and does not score. Adding `.score(n)` makes it contribute. A satisfied Boolean adds `n` and an unsatisfied one adds 0; a measurement `m` contributes `m * n`. `t.score(n)` adds points directly, and its returned handle can only set `key` and `label`.
```ts theme={null}
turn.calledTool("search"); // Saves only the evaluation
turn.calledTool("search").score(2); // +2 when matched
t.check({ task, answer: turn.message }, completeness).score(5); // +4 when the measurement is 0.8
t.score(5); // Directly +5
```
In a scored eval, a measurement can score without a gate. When you need a quality gate, call `.gate(minimum)`. It does not affect that item's score, so the same Assertion can have both a gate and a score.
A low score alone is not a failure. Only when a gate is set and not met is the verdict `failed`, and the score already earned is kept. A check without `.score()` does not show `+0`. A scored eval with no scoring items normally gets `score: 0`.
## Stop later steps
`.orStop()` stops later steps when the check is not satisfied. It returns a Promise and must be awaited:
* A Boolean check uses its own condition.
* For a measurement with no gate yet, use `.orStop(minimum)` to set the minimum for stopping.
* If `.gate(minimum)` is already written, use `.orStop()` with no argument to reuse the same minimum.
```ts theme={null}
const turn = await t.send("Search for information and explain the conclusion.");
await turn.succeeded().orStop();
const search = await t.check(turn.message, includes("conclusion")).orStop();
```
Stopping only affects code after the `await` that has not run yet, and that code produces no results. Ordinary JavaScript side effects that already happened are not undone, and concurrent tasks started earlier are not cancelled.
The verdict after a stop depends on the reason:
* Stopping because a gate was not met gives `failed`.
* A measurement that uses only `.orStop(minimum)` without a gate and falls below the minimum only stops later steps. It does not give `failed` for that reason.
* Stopping because of `unavailable` or `errored` gives `errored`.
After the first stop, this Assertion's configuration can no longer change. Awaiting `.orStop()` with no argument again returns the same Promise.
## Judge
`defineJudge` defines a Judge. `t.judge(material, definition)` only accepts a Judge definition and is shorthand for `t.check(material, definition)`; both take the same execution path. Both pass/fail and scored evals can use `.gate(minimum)`, and scored evals can also use `.score(n)`. One Assertion calls the Judge only once. For material, configuration, and failure behavior, see [Judge](/docs/explanation/judge).
## Missing evidence and saved results
Missing evidence must not be treated as an ordinary failure to match. When the result is `unavailable` or `errored`, the reason and redacted evidence are kept.
Each Assertion's result is also more than "passed" or "did not pass". It saves three kinds of content:
* The value being checked: safe structured content, or a stable reference to it.
* Which rule was used: the identity and complete configuration of the Match or evaluation function.
* The evaluation result, plus the information needed for gates, scoring, and display.
With `t.check(a, b)` and `t.judge(material, definition)`, you provide the value being checked and the rule. Methods such as `calledTool` and `succeeded` get the value and build the Match for you, and save the same things.
Two examples:
* `t.check(await t.sandbox.runCommand(...), commandSucceeded())` saves the command name, arguments, exit status, and duration of that command, plus redacted stdout and stderr (content or references).
* `calledTool(...)` saves every tool call it saw: the call identity, redacted input, status, references to output or errors, whether the record is complete, and which events matched. When nothing matched, it still saves the scope it looked at and the candidate calls, not just `false`.
So `query`, `view`, and the source view can all build text and interfaces directly from the saved result, without rerunning the Match or calling the Judge again. The `expected` and `received` shown in interfaces are generated at read time from this content; they are not just two stored strings.
Assertion only defines what must be saved. Record decides how it is written to disk. When the data needed for a decision cannot be saved safely, the result must be `unavailable`; it cannot keep only `matched` or `mismatched`.
## Related reading
* [Match Reference](/docs/reference/expect) — Built-in Matches, composition, and custom Matches.
* [Eval API](/docs/reference/define-eval) — The complete field set for `t`, Session, Turn, and handles.
* [Pass/fail and scored evals](/docs/tutorials/evaluation-kinds) — The complete form of both kinds of eval.
* [Judge](/docs/explanation/judge) — Judge and model configuration.
# How to drive an Agent in NiceEval: send, sessions, and HITL
Source: https://niceeval.com/docs/explanation/drive
t.send() and the Turn it returns, t.sendFile(), multi-turn conversations, t.newSession(), and human-in-the-loop (HITL) handled through respond().
Before writing an assertion or Judge check, make the Agent do something. **Drive** is the part of `test(t)` that sends input and receives results: `t.send()`, `t.sendFile()`, `t.newSession()`, and the HITL methods `t.respond()` / `t.respondAll()`. Drive produces a **Turn**. Every assertion and Judge check in [NiceEval](https://niceeval.com/) reads data from a Turn; see [Assert](/docs/explanation/assert) and [Judge](/docs/explanation/judge).
## `t.send()` and the `Turn` it returns
`t.send(input)` is the Adapter's basic execution verb. Under the hood, it calls the Adapter's `send(input, ctx)` and normalizes the result into a `Turn`:
```ts theme={null}
const turn = await t.send("What's the weather like in Brooklyn today?");
turn.message; // The last assistant message
turn.data; // Structured output: present only if this send declared output; its type follows that declaration (below)
turn.status; // "completed" | "failed" | "waiting"
turn.events; // This turn's StreamEvent[]
turn.usage; // { inputTokens, outputTokens, ... }, only if the Adapter reported it
turn.succeeded().label("Turn completed"); // This turn did not fail or remain on an unanswered HITL request
```
`t.reply` is shorthand for the main session's last assistant message. It is equivalent to the `turn.message` from the latest `t.send()`. `t.events` is the full accumulated event stream for the main session.
When you need structured output, declare `output` on `send` using a Standard Schema such as zod. The declaration both asks the application for that output and supplies the type of `turn.data`:
```ts theme={null}
const turn = await t.send("What is the total amount on this invoice?", {
output: z.object({ amount: z.number(), currency: z.string() }),
});
t.check(turn.data.amount, equals(3200)); // turn.data is { amount: number; currency: string }
```
The Runner validates the value against the declaration immediately after the Adapter returns. If `data` does not match, the turn is immediately `failed` and reports the difference. A turn that did not declare `output` has no `turn.data`. It is not a pocket for “whatever the Agent happened to return”; the field exists only when you declare it and is strongly typed when it does.
`turn.succeeded()` looks only at this turn: its `status` and whether it is waiting for an unanswered HITL request. If later turns depend on an earlier one succeeding, use `await turn.succeeded().orStop()`.
## One turn with a file: `t.sendFile()`
`t.sendFile(path, text?)` reads a local file relative to the eval case's directory, infers a MIME type from the extension, and attaches it to this turn's input as a data URL:
```ts theme={null}
const turn = await t.sendFile("fixtures/invoice.png", "What is the total amount on this invoice?");
t.check(turn.message, includes("$"));
```
## Multi-turn conversations
Each `await t.send(...)` is a new turn in the same conversation. Assign each response to a local variable so you can assert that individual turn as well as use run-level `t` assertions:
```ts theme={null}
const draft = await t.send("Help me draft a follow-up email.");
draft.succeeded().label("Draft completed");
t.check(draft.message, includes("Best regards"));
t.judge({ request: draft.input, draft: draft.message }, toneQuality)
.gate(0.6).label("Professional tone");
await t.send("Okay, send it.");
t.calledTool("send_email").label("Sent");
```
Whether multi-turn `t.send()` calls truly continue their context depends on whether the Adapter's `send` uses `ctx.session` session-continuation storage, either a typed slot or `id` + `capture()`. Without it, every turn is a new conversation. See [Adapter](/docs/explanation/adapter) and [Write Send](/docs/tutorials/write-send).
## Independent sessions: `t.newSession()`
`t.newSession()` opens a second conversation line parallel to the main session, without interfering with it. Its returned session handle has the same drive API (`send` / `sendFile` / `respond` / `respondAll`) and scoped-assertion vocabulary, but sees only that session's events:
```ts theme={null}
await t.send("My name is Xiaoming.");
await t.send("What is my name?");
t.check(t.reply, includes("Xiaoming")); // The main session remembers
const fresh = t.newSession();
await fresh.send("What is my name?");
t.check(fresh.reply, satisfies("No memory leak", (reply) => !reply.includes("Xiaoming")));
```
Do not assume that `t.newSession()` inherently guarantees isolation. The Runner only guarantees that the new session line starts with empty state. Opening a fresh conversation with the application is the Adapter's job. An Adapter that ignores its session state and always resumes the same underlying context makes `t.newSession()` silently share state without an error. When you write an Adapter, verify isolation with an eval case like the one above before you trust it.
## Human-in-the-loop (HITL)
Some Agents stop during a turn to wait for approval or missing information instead of finishing immediately. That turn ends with `status: "waiting"` and has one or more `input.requested` events that say what it needs. For the full mental model—handshake sequence, Adapter obligations, and `rejected` semantics—see [HITL](/docs/explanation/hitl). This section explains how to use it on the eval side.
```ts theme={null}
import { equals } from "niceeval/expect";
const draft = await t.send("Draft a follow-up email, but do not send it until I confirm.");
t.check(draft.status, equals("waiting")).label("Paused for approval");
const request = t.requireInputRequest({
prompt: /send/,
optionIds: ["approve", "reject"],
});
await t.respond({ request, optionId: "approve" });
t.calledTool("send_email").label("Sent");
```
`t.requireInputRequest(filter)` turns a pending HITL request into a concrete value you can inspect and answer. It throws if zero or more than one pending request matches, so use as many filter fields as you can—`id`, `prompt`, `display`, `action`, `optionIds`, or `input`—to disambiguate.
`t.respond(...)` answers the request and emits the next turn. Each argument is one answer. The string form is only for when exactly one request is pending. When several requests pause together, name each one explicitly with `{ request, optionId }` or `{ request, text }`; argument order is never used to guess. Under the hood it is another ordinary `send`: answer text goes to `input.text`, while structured answers go to `input.responses` one per request. A response that selects a request option carries `{ requestId, optionId }`; a free-text response carries `{ requestId, text }`. The Adapter does not need to parse the text: `requestId` identifies which answer belongs to which request. See [Inputs for the different answers](/docs/explanation/adapter#inputs-for-the-different-answers) for how each response looks to the Adapter.
If the current turn has multiple pending requests of the same kind that should all receive the same answer, such as approving a batch of file changes one by one, use `t.respondAll(optionId)` to answer them at once. It validates `optionId` against every pending request first. If the option is absent from a request's `options`, it throws instead of silently sending the wrong answer:
```ts theme={null}
await t.send("Submit this batch of changes for approval one by one.");
t.requireInputRequest({ display: /approval/ });
await t.respondAll("approve");
t.succeeded().label("All completed");
```
## Related reading
* [HITL](/docs/explanation/hitl) — The complete concept of paused turns waiting for people: handshake sequence and obligations on both sides.
* [Assert](/docs/explanation/assert) — The assertion vocabulary that reads `Turn.events` and `Turn.data`.
* [Judge](/docs/explanation/judge) — Assess open-ended output with a rubric and material that has meaningful fields.
* [Adapter](/docs/explanation/adapter) — Where capabilities come from: what unlocks `t.newSession()`, HITL, and tool-related assertions.
# Eval cases in NiceEval: lifecycle, verdicts, and files
Source: https://niceeval.com/docs/explanation/evals
An eval case says what to send and what counts as correct, without binding to a system under test. Learn how eval cases are discovered, scheduled, scored, and reported.
An eval case says what to send and what counts as correct. It is usually exported from a `*.eval.ts` file and declared with `defineEval`.
It reads like a unit test, but the output of the system under test is nondeterministic. The same eval case runs several times according to the Experiment's `attempts`. Each run is one Attempt, and together they give a pass rate or a score.
## What an eval case contains
```ts theme={null}
import { defineEval } from "niceeval";
import { includes, jsonMatch, toolMatch } from "niceeval/expect";
export default defineEval({
description: "Brooklyn weather query",
async test(t) {
await t.send("What's the weather like in Brooklyn today?");
t.succeeded();
t.calledTool(
toolMatch("get_weather", { input: jsonMatch({ city: "Brooklyn" }) }).exactly(1),
);
t.check(t.reply, includes("sunny"));
},
});
```
Core fields:
| Field | Description |
| - | - |
| `description` | A human-readable description shown in reports |
| `sandbox` | Optional. This eval case's own Sandbox configuration; omit it when the Experiment already names the prebuilt environment |
| `test(t)` | The interaction and assertion logic |
An eval case does not declare which Agent to use, so the same eval case can run against different Agents under different Experiments. The Agent is an Experiment field, and the CLI does not override it ad hoc.
Do not write `id` or `name` by hand. [NiceEval](https://niceeval.com/) derives the ID from the file path.
## Path is identity
`evals/weather/brooklyn.eval.ts` has the ID `weather/brooklyn`. Positional arguments after the Experiment name filter by ID prefix:
```bash theme={null}
npx niceeval exp local weather
npx niceeval exp local weather/brooklyn
```
IDs follow the directory structure, so they are stable and readable. Filtering matches by string prefix: `terminal-swe-bench` also matches `terminal-swe-bench-astropy-1`, and the next character does not have to be `/`.
## Lifecycle
The Runner loads `*.eval.ts` files and Fixture directories under `evals/`.
It combines concurrency, result reuse, `attempts`, and earlyExit to produce the Attempts for this run.
`t.send()` calls the Adapter selected by the Experiment and receives a standard `Turn`.
NiceEval records every check: value assertions, assertions on the Turn and Session, Judge scores, and each item's score contribution.
A pass/fail eval combines the check results into a final verdict; a scored eval accumulates a score.
The terminal and Reporters output results; the Run and published Attempts are written to `.niceeval/record.sqlite`.
## Four verdicts
Each Attempt of a pass/fail eval gets exactly one of four verdicts:
Every required Boolean check is satisfied, every measurement with `.gate(minimum)` reaches its minimum, and no execution error occurred.
At least one required Boolean check is not satisfied, or a measurement with `.gate(minimum)` falls below its minimum.
An execution exception, a timeout, or a mistake in the eval code means this run cannot reach a trustworthy conclusion.
The eval case skipped itself, usually with `t.skip(reason)`.
A scored eval is read mainly by its accumulated score, and higher is better. With no scoring items, the score is `0`. Reports show the score by default and do not count it toward the pass rate, but the verdict is still kept. A result with a `failed` verdict does not enter the success ranking even if its score was fully computed.
The `points` in `.score(points)` is the weight of that item, not the total score. Reports do not convert scores to percentages automatically. When one Experiment contains both kinds of evals, the report shows the pass rate and the score separately.
## Verdicts and measurements
In pass/fail evals, Boolean checks affect the verdict by default. In scored evals, Boolean checks are only recorded by default. Continuous measurements and Judge scores are numbers between 0 and 1. Both kinds of evals use `.gate(minimum)` to set a minimum, and falling below it gives `failed`. In a scored eval, the score already earned is kept when a gate is not met. See [Assertion and Match](/docs/explanation/assert) for the complete rules.
## The `*.eval.ts` convention
Only files ending in `.eval.ts` are discovered. Directories only form ID prefixes:
```text theme={null}
evals/
└─ billing/
└─ refund.eval.ts # id: billing/refund
```
## Array exports and data-driven tests (dataset fan-out)
A file can also default-export an array of `defineEval(...)` calls, generating multiple eval cases from the same logic:
```ts theme={null}
export default rows.map((row) =>
defineEval({
description: row.task,
async test(t) {
await t.send(row.prompt);
t.check(t.reply, equals(row.expected));
},
}),
);
```
This generates IDs such as `sql/0000` and `sql/0001`. See [Data-driven tests](/docs/tutorials/dataset-fanout).
## Related reading
* [Experiment](/docs/explanation/experiment) — The other half: what to evaluate and how to run it, and why it is separate from eval cases.
* [Assert](/docs/explanation/assert) — Assertion and verdict rules.
# Experiment: what to evaluate and how to run it
Source: https://niceeval.com/docs/explanation/experiment
An Experiment is a check-in-able run configuration: which Agent and model the same set of eval cases runs against, which flags are on, and how many times to run. Eval cases and Experiments are kept separate so the same eval cases can run against different systems.
An Experiment says which Agent to run against, which model to use, which flags to turn on, and how many times to run. Use it to compare models, prompts, and feature toggles. `flags` are the feature flags of an A/B test, and one Experiment is one set of flag values.
## What an Experiment contains
An Experiment is a check-in-able TypeScript file under `experiments/`. It has only a few core fields:
```ts theme={null}
// experiments/concise.ts
export default defineExperiment({
description: "Concise prompt variant",
agent: webAgent({ baseUrl: "https://staging.example.com" }), // A configured agent instance
model: "gpt-5.4", // Passed through to the Adapter as ctx.model
flags: { promptVariant: "concise" }, // Passed through as ctx.flags; t.flags in the eval
attempts: 3, // How many times each eval runs (to counter randomness)
});
```
* `agent`: what to evaluate. Put an already configured Adapter instance here. Pass the system's address and authentication to the Adapter when you create the instance, not in other Experiment fields.
* `model` / `flags`: experiment conditions handed to the Adapter. `flags` only accepts flat strings, finite numbers, or booleans, and NiceEval does not interpret their meaning. When the Adapter declares `parseFlags`, the values are validated and normalized first, then handed to the Adapter through `ctx`. The Adapter forwards them with the request, and the application switches behavior based on the values. Model comparison (Tier 1) and feature A/B tests (Tier 3) both use this path. See [Integration tiers](/docs/explanation/tier).
* Run parameters such as `attempts`, `budget`, concurrency, and `sandbox`: how to run and how much. For the full field set, see [Write Experiments](/docs/tutorials/write-experiment#common-fields).
An Experiment also has a pair of Experiment-level Hooks, `setup` / `teardown`. They run once for the whole Experiment on your own machine, and start and stop services shared by all Attempts, such as a tunnel to an internal service or a mock server just for this Experiment.
`setup` runs before the first Attempt of this Experiment starts. `teardown` runs after all Attempts finish, and also runs when the run is interrupted. But `teardown` runs only if `setup` has already run.
To prepare the environment inside a Sandbox for the whole Experiment, attach the preparation steps to `.before()` on the `sandbox` field. Steps whose content does not change can be reused next time. Dynamic callbacks can register cleanup with `context.onCleanup()`. See [Start shared services for an Experiment](/docs/tutorials/experiment-services) and [Sandbox · Prepare the Sandbox in a deterministic order](/docs/tutorials/sandbox-providers#prepare-the-sandbox-in-order).
When different eval cases in the same Experiment need different prebuilt environments, write the `sandbox` configuration that names the prebuilt environment on each eval case, and leave it out of the Experiment. Reuse repeated configuration with ordinary TypeScript functions; no extra registration is needed. The same Experiment can still cover all eval cases, and NiceEval merges each eval case's configuration with the Experiment's configuration one by one. See [Write Experiments · Different eval cases need different environments](/docs/tutorials/write-experiment#different-evals-need-different-environments).
## Matrix comparisons
Write one Experiment file for each variant you want to compare. Two models means two files that differ only in the `model` line. A prompt A/B test means two files that differ in one parameter. Run the same set of eval cases under each Experiment, then compare the results published at the same moment. Each Experiment computes its denominator from the eval cases it ran, and NiceEval does not force a common intersection. For what is worth comparing and how to read the results, see [Write Experiments](/docs/tutorials/write-experiment#compare-two-models).
Put values used only to group reports, which do not affect the run, in `labels`. They are recorded on the Run and are not passed to the Adapter or eval cases. Running again after changing `labels` creates a Run with the new labels, and unaffected earlier results are reused as usual. Historical Runs keep the `labels` they had at the time.
## Related reading
* [Write Experiments](/docs/tutorials/write-experiment) — Compare models and prompts, read comparison results, and common fields.
* [Start shared services for an Experiment](/docs/tutorials/experiment-services) — How to write `setup` / `teardown` and when they run.
* [Eval cases](/docs/explanation/evals) — The other half: what an eval case is, its lifecycle, and its verdicts.
* [Tier](/docs/explanation/tier) — The tier in which `model` and `flags` each take effect.
# Human-in-the-loop (HITL): pause a turn, answer, and continue
Source: https://niceeval.com/docs/explanation/hitl
What human-in-the-loop is, how it appears in Claude Code, Codex, and AI SDK applications, and how NiceEval turns it into an eval path: a waiting turn pauses, input.requested says what it needs, and t.respond answers for the human so the same turn can continue.
HITL (human-in-the-loop) is when an Agent pauses during execution and hands a decision back to a person. It does not continue until it receives an answer. This is not an error; it is an intentional safety or information gate in the Agent product. A person approves a sensitive operation, supplies missing information, or chooses between several options.
You probably use it every day:
* **Claude Code** pauses before it runs a risky command or changes a file outside its permissions, then asks “Allow this?” It executes only after approval, and takes a different path after rejection.
* **Codex** uses approval modes such as suggest and auto-edit to decide which actions need confirmation first. This is the same kind of gate.
* **AI SDK applications** often surface a tool call to the frontend when a tool has no `execute`. A person confirms it in the UI, the result returns through `addToolResult`, and the model continues. This is the most common approval-gate pattern in custom applications.
For a subject under test, this “pause and wait for a person” branch needs regression coverage like any other behavior. Does the expected action happen after approval? Does it truly not happen after rejection? What if it asks the wrong person or pauses at the wrong point? Eval cases run automatically, so they have no person in the loop. [NiceEval](https://niceeval.com/) reduces the “person” to three programmable actions: use `t.check(turn.status, equals("waiting"))` to confirm the Agent paused, use `t.requireInputRequest()` to inspect what it is waiting for, then use `t.respond()` to answer in place of the person. Approval, rejection, and extra information become paths you can write into an eval case.
## How a paused turn is represented: `waiting` + `input.requested`
The `Turn.status` returned from `send` has three values: `"completed"` when it has finished, `"failed"` when it has errored, and `"waiting"` when **the turn has not finished and the Agent is waiting for human input**. `waiting` is not a failure; it is a suspension. Once an answer arrives, the same turn continues without resending the request or opening a new conversation.
A paused turn must also say what it is waiting for. Every question that needs an answer emits an `input.requested` event with a stable request `id`, the action where it paused (`action`), and optional decisions in `options`, such as approve / deny. Both status and event are necessary: `waiting` tells the eval case that it should answer, while `input.requested` tells it what to answer.
## One complete handshake
`t.respond` has no special channel. It is simply **another ordinary `send`**, with the human decision delivered in structured input:
Two easy mistakes to avoid:
* **A response turn is not a new conversation.** It happens on the same session line, so `ctx.session` remains the same. The Adapter uses it to recover the paused state from the previous turn.
* **Do not infer an answer from text.** `input.responses` carries `{ requestId, optionId }` for each request, or `{ requestId, text }` for a free-text answer. The eval side has already verified that an `optionId` exists in the request's options. A typo throws instead of silently reaching the application. See [Inputs for the different answers](/docs/explanation/adapter#inputs-for-the-different-answers).
## On the eval side: three actions
```ts theme={null}
import { equals, toolMatch } from "niceeval/expect";
const draft = await t.send("Send a weekly status email to my manager.");
t.check(draft.status, equals("waiting")); // ① Assert that this turn paused
const req = t.requireInputRequest({ action: "send_email" }); // ② Get the pending request
await t.respond({ request: req, optionId: "approve" }); // ③ Answer for the person and emit the next turn
t.calledTool(toolMatch("send_email", { status: "completed" }));
```
* `t.check(draft.status, equals("waiting"))` asserts that the turn really stopped in `waiting`. Conversely, `draft.succeeded()` fails on a waiting turn. A paused turn is an explicit state; it is never treated as “finished.”
* `t.requireInputRequest(filter)` picks exactly one pending request by matching fields such as `id`, `prompt`, `action`, or `optionIds`. It throws when zero or more than one request matches, which disambiguates several simultaneously paused requests.
* `t.respond(...)` answers the request and emits the next turn. When several requests of the same kind need the same answer, such as approving a batch of changes one by one, `t.respondAll(optionId)` handles them at once.
Approval and rejection are two branches of one gate. Covering it means writing an eval case for both: the expected action happens after approval, and it does not happen after rejection. For the complete API-by-API use, see [Drive](/docs/explanation/drive#human-in-the-loop-hitl).
## On the Adapter side: two obligations and one continuation
HITL asks an Adapter to do exactly three things. The first two happen in the paused turn; the third happens in the next turn when the answer arrives:
1. **Return `status: "waiting"` honestly when a turn pauses.** Do not report a suspension as `"completed"`; otherwise `t.check(turn.status, equals("waiting"))` fails and `succeeded()` can pass incorrectly.
2. **Emit one `input.requested` event for every question waiting for an answer.** Keep `id` stable and populate as many fields as possible. eval-side checks and alignment depend on them.
3. **On the next `send`, return the decision first, then continue.** Take each decision from `input.responses` by `requestId` rather than guessing by order, then continue from where the previous turn paused instead of resending the request.
The paused state, such as a partially read SSE stream, lives in `ctx.session` on this session line. The Adapter declares a private typed slot with `createSessionSlot()`, saves it on pause with `ctx.session.set(slot, pending)`, and recovers it on the response turn with `ctx.session.take(slot)`. `take` clears the value as it retrieves it. See step 5 of [Write Send](/docs/tutorials/write-send#step-5-hitl) for the complete streaming + HITL skeleton.
Like other [NiceEval](https://niceeval.com/) capabilities, HITL does not have a Boolean declaration: **if you do it, you have it**. Once `send` has returned `"waiting"` and emitted `input.requested`, `t.respond` works. If it has not, the eval case fails explicitly at the status check or `requireInputRequest()` rather than silently passing. See [Adapter](/docs/explanation/adapter#where-capabilities-come-from-proof-by-construction-not-a-questionnaire) for how capability follows from construction.
## Rejection is not a failure
A human veto and a tool error are different things. The event stream distinguishes them with different statuses: an operation that a person rejects has `"rejected"` as the tool `operation.finished` `status`, not `"failed"`. That way `t.noFailedActions()` is not tripped by a legitimate human veto, while `t.calledTool(toolMatch("deploy", { status: "rejected" }))` precisely asserts “the call started and a person stopped it.” This is exactly what an eval case for the rejection branch needs to assert.
## Which Agents do not need HITL
Not every Agent has this branch. If it does not exist, skip it entirely: the Adapter does not need to handle `waiting`, and the eval case does not use `t.respond`.
* **Agents that finish every turn in one pass:** question-answering, retrieval, translation, and other send-and-reply services have no point where they wait for a person.
* **Runtime modes with the approval gate disabled:** Claude Code and Codex are HITL products, but when they are subjects under test in a Sandbox, they usually run fully automatically and skip permission confirmation. This is how the built-in `claude-code`, `codex`, and `bub` Sandbox Agents run, so none uses HITL.
* **Approval that happens outside the execution loop:** for example, a ticket system may have a person review an Agent's output. That is another system's flow. It does not go through `send`, so an eval case cannot and should not evaluate it.
There is one test: use HITL only if your Agent's execution loop has a branch that pauses for a person, then continues after an answer, and you want to write approval and rejection into an eval case.
## Related reading
* [Drive](/docs/explanation/drive#human-in-the-loop-hitl) — The complete eval-side usage: status checks, `t.requireInputRequest()`, and `t.respond()` / `t.respondAll()`.
* [Adapter](/docs/explanation/adapter#inputs-for-the-different-answers) — The structured input that reaches an Adapter, in four typical forms.
* [Write Send](/docs/tutorials/write-send) — Adapter-side practice: typed session slots and the complete streaming + HITL skeleton.
* [Connect Your Agent](/docs/tutorials/connect-your-agent) — The integration overview: minimal integration, parameter channels, and the incremental map.
# Judge in NiceEval: assess open-ended output
Source: https://niceeval.com/docs/explanation/judge
Use a rubric, anchors, and explicit material to assess open-ended output, adding images when needed.
Judge uses a separate grading model to assess factual consistency, summary faithfulness, and explanation quality. It is a managed evaluator. Ordinary custom
`ScoreMatch` values stay pure functions; `defineJudge`, the built-in factories, and the advanced managed `defineScoreMatch` all take the same Assertion path.
## Declare a scoring definition
Write the criterion at module scope. `rubric` describes the dimensions to evaluate, and `anchors` give calibration points on a continuous quality scale:
```ts theme={null}
import { defineJudge } from "niceeval";
const explainsRisk = defineJudge({
name: "explains-risk",
rubric: "Based on change, evaluate whether explanation covers compatibility, rollback, and data risks.",
anchors: [
{ measurement: 0, description: "Does not explain a concrete risk or rollback plan" },
{ measurement: 0.5, description: "Explains some risks but omits one key point" },
{ measurement: 1, description: "Fully explains compatibility, rollback, and data risks" },
],
});
```
When `anchors` is omitted, NiceEval uses 0 and 1 as the two default endpoints. A measurement can be any finite value in `[0,1]`;
the endpoints do not limit the output to two classes, and they are not model confidence.
## Select evaluation material
Pass the values needed for the decision directly to `t.judge`. The eval's `judge` only selects the model configuration for this eval:
```ts theme={null}
export default defineEval({
judge: "judge-model",
async test(t) {
const change = "Modify a persistent field and provide a rollback plan";
const turn = await t.send(`Explain this change: ${change}`);
t.judge({ change, explanation: turn.message }, explainsRisk)
.gate(0.8)
.label("Risk explanation quality");
},
});
```
Material uses strings, finite numbers, booleans, null, arrays, and plain objects. Field names should express application meaning, such as `change` and
`explanation`; you do not need to join the object into text first. NiceEval creates a bounded JSON snapshot when the Assertion is registered, so later changes to the original object do not change the request.
When you need to inspect image content, put `judgeImage({ body, mediaType })` in the material. It freezes PNG or JPEG bytes without creating another Judge or sending an extra request. The image and text are sent in the same Judge call and saved in the same Assertion's evidence. For the setup steps, see [Add image material](/docs/tutorials/verify-judge#attach-image-material).
`t.judge(material, explainsRisk)` and `t.check(material, explainsRisk)` each register one measurement Assertion. They share the material snapshot, budget, evaluation, audit, result saving, and handle; there is no Match allowlist.
## Use built-in Judges directly
Root `t`, Session, and Turn all provide built-in methods that return an Assertion's `MeasurementAssertionHandle`. For example:
```ts theme={null}
import { eventMatch } from "niceeval/expect";
const answer = await t.send("List the risks and rollback steps.");
answer.closeQA("Does this turn explain the risks and actionable rollback steps?")
.gate(0.8).label("Rollback explanation");
// For a special check, select all assistant messages explicitly.
answer.closeQA(
eventMatch("message", { role: "assistant" }),
"Do all messages clearly explain the risks?",
).gate(0.8);
```
`closeQA(selector, question, options?)` sends all matched material to one Judge call in its original order. The question is an acceptance question answered only from the material, not a preset answer.
An Agent's `closeQA(question, options?)` uses the complete recorded history of that Turn, Session, or Attempt by default, including messages and tool facts. It does not pick only the last reply or successful operations.
An explicit EventMatch selects from the three event views. For questions about tool inputs and receipts, use a ToolMatch or a custom MaterialMatch.
All these forms compose a scoring Match and pass it to the same `check`. Application assertions such as `usedNoTools()` wrap a fixed criterion and do not accept another Match.
The result is 1 when the condition holds, 0 when it does not, and `unavailable` when evidence is insufficient. A complete empty set gives 0 without calling the model; missing or oversized material is not filled with zero or truncated before scoring.
An Adapter provides the application ctx through `create`, and the synchronous reader in `defineMaterialMatch` reads the current read-only ctx when the Assertion is registered.
A selector's predicate judges one item only; `and` constrains the same item. Use `t.check(selector).gate()` for ordinary existence checks and `t.closeQA(selector, question).gate(0.8)` for a question about the whole set.
`factuality`, `faithfulness`, `instructionFollowing`, and `pairwisePreference` take explicit material, build a Match, and pass it to the same `check`.
## Configure the Judge Provider
Project configuration selects the service explicitly with a named Provider factory. This example connects to OpenAI or an explicitly compatible gateway:
```ts theme={null}
import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";
export default defineConfig({
judgeRuntime: OpenAIProvider({
model: "judge-model",
baseUrl: "https://gateway.example.com/v1",
apiKeyEnv: "JUDGE_GATEWAY_KEY",
timeoutMs: 180_000,
maxOutputTokens: 512,
}),
});
```
| Factory | Service | Default key variable |
| - | - | - |
| `OpenAIProvider` | OpenAI or an explicitly compatible gateway | `OPENAI_API_KEY` |
| `VercelProvider` | Vercel AI Gateway | `AI_GATEWAY_API_KEY` |
| `OpenRouterProvider` | OpenRouter | `OPENROUTER_API_KEY` |
| `TypesafeProvider` | TypeSafe System One | `TYPESAFE_API_KEY` |
All four factories are exported from `niceeval/judge`. The Vercel factory means AI Gateway, not Vercel Sandbox or a general AI SDK model object.
When one eval only needs a different model, write just this field:
```ts theme={null}
export default defineEval({
judge: "another-model",
async test(t) {
const answer = await t.send("What is the capital of France?");
t.factuality({ input: "What is the capital of France?", output: answer.message, expected: "Paris" }).gate(1);
},
});
```
This keeps the global Provider's endpoint, credential source, and execution limits, and replaces only `model`.
Evals and Experiments can also provide a complete Provider, which replaces the service, default model, and limits as a whole.
There is no rule that guesses the service from the hostname, model prefix, or an existing key.
NiceEval uses the selected Provider's managed protocol request to get a finite `[0,1]` measurement and a public rationale. Chat services use forced-function requests, and TypeSafe uses `/systemone`. With no model or key configured, no network request is sent and the Assertion is `unavailable`. Transport failures or timeouts are `unavailable`; HTTP 400, protocol incompatibility, or invalid responses are `errored`. Static configuration validation does not probe the network.
TypeSafe uses the `/systemone` protocol. It supports score, classify, and batch classify, but not extract.
So `faithfulness()` on `TypesafeProvider` gives `unavailable`; it is not converted to an overall score or a different denominator.
## Thresholds and scores
Both pass/fail and scored evals call `.gate(minimum)` on the registered handle to form a quality gate. Scored evals can also call
`.score(points)`, and the contribution is the measurement times points. Gate and score can be combined in either order, and both run the Judge only once.
Successful requests save the fixed material actually sent. A path, URL, alt text, or ordinary base64 string is not visual input; only explicit image material becomes the image part of the model API request.
# NiceEval's overall model: the parts of an evaluation
Source: https://niceeval.com/docs/explanation/overview
What eval cases, Experiments, Adapters, Sandboxes, the Runner, and Record each handle, how they connect into one run, and why the same eval runs more than once.
An evaluation answers one question: can the system under test do this task correctly with this configuration? NiceEval splits that question into several parts, and each part handles one thing.
```text theme={null}
Eval case (what counts as correct) Experiment (against what, and how to run)
\ /
\ /
Runner (schedules Attempts, scores)
│
Adapter (connects to the system under test)
│
System under test (direct, or running in a Sandbox)
│
Record (stores each Attempt's result)
│
show (terminal) · view (browser) · query (JSON)
```
## Eval cases: what counts as correct
Eval cases live in `evals/*.eval.ts`. They say what to send and what to check. They do not say which Agent or model to run against, so the same eval case can run against different systems under test.
Checks come in two kinds. Deterministic facts get deterministic rules, such as which tool was called or whether the tests passed. Open-ended quality goes to a Judge, a grading model that scores against the criteria you write. See [Eval cases](/docs/explanation/evals) and [Judge](/docs/explanation/judge).
## Experiment: against what, and how to run
Experiments live in `experiments/`. They say which Adapter, which model, which flags, and how many times each eval runs. To compare two models, write two Experiments that differ only in `model` and run each against the same set of eval cases. See [Experiment](/docs/explanation/experiment).
## Adapter and Sandbox: how to reach the system under test
An Adapter is connection code you write. It sends the actions in an eval case to the system under test, then translates the response into events NiceEval can read, such as replies, tool calls, and usage. You can start by implementing only `send` and add OTel or flags when you need them. See [Adapter](/docs/explanation/adapter) and [Integration tiers](/docs/explanation/tier).
A Coding Agent needs to change code and run commands on a real file system, so NiceEval first creates an isolated Sandbox and starts the Agent inside it. Directly connected Web Agents and business applications do not need a Sandbox.
## Runner and Attempt: why the same eval runs more than once
LLM output is nondeterministic. A question answered correctly this time may be answered wrong next time. One run cannot tell you whether a change really helped. So an Experiment uses `attempts` to set how many times each eval case runs, and each run is called an Attempt.
The Runner selects eval cases from the Experiment, expands them into a batch of Attempts, and schedules them within the concurrency limits. Each Attempt gets one verdict: `passed`, `failed`, `errored`, or `skipped`. Together, a batch of Attempts gives the pass rate of that eval under that configuration. Scored evals accumulate a score instead of reading a pass rate. See [Pass/fail and scored evals](/docs/tutorials/evaluation-kinds).
Read `failed` and `errored` separately. `failed` means the system under test replied normally but got it wrong. `errored` means no trustworthy result came back, for example because the service did not start or a credential expired. It points to an environment problem, not to the capability of the system under test.
## Record: results stay for comparison and investigation
When an Attempt finishes, its result is saved together with the conversation, tool calls, file changes, duration, and cost in the project's `.niceeval/record.sqlite`. This is Record.
Keeping results serves two purposes. One is investigation: go back to a failure and see what the system under test actually did. The other is reuse: on the next run, results unaffected by your change are reused and only the affected part runs again. See [Rerun and reuse](/docs/tutorials/rerun-and-cache).
There are three ways to read results. `show` drills down in the terminal, and both you and Coding Agents use it to investigate. `view` lets you browse conversations and timing in a browser. `query` outputs JSON for scripts and CI. All three reach the same conclusions. See [How run results flow](/docs/explanation/results-flow).
## Next steps
* First time using NiceEval: read the [Quickstart](/docs/tutorials/quickstart).
* To learn about one part: follow the link at the end of each section above.
* To look up fields and commands: go to the [Reference](/docs/reference/define-eval).
# Record
Source: https://niceeval.com/docs/explanation/record
Learn what the project results file .niceeval/record.sqlite stores, how to copy it for someone else, and what is validated when it is read.
You want to hand a run's results to a colleague for review, or archive a baseline to compare against later. All of NiceEval's run results live in one file: `.niceeval/record.sqlite` in your project. This is Record.
## What Record contains
Runs, in-progress summaries, published Attempts, and their attachments and content are all stored in this file. Cache, credentials, and user-level service state are not.
A Run can be found as soon as it is created. An Attempt can be read once it finishes and publishes, without waiting for its Run to end. Published Attempts are never modified again. `query`, `show`, and `view` only read published results.
## Hand results to someone else
After a run has fully ended, copy the file directly:
```sh theme={null}
cp .niceeval/record.sqlite ./baseline.record.sqlite
```
Copy only `record.sqlite` itself. Do not copy the WAL, checkpoint, or staging files next to it, and do not modify the SQLite content after copying.
When you receive someone else's file, point `--record` at it:
```sh theme={null}
pnpm exec niceeval show --record ./baseline.record.sqlite
pnpm --silent exec niceeval query explain --record ./baseline.record.sqlite --request request.json
```
`view` only reads the current project's `.niceeval/record.sqlite` and does not accept `--record`.
## External files are validated before reading
`query` and `show` treat the file named by `--record` as untrusted input. The command reads it only when the format version matches exactly, the SQLite integrity check passes, and the internal data is consistent. When validation fails, the command reports an error. It does not migrate, patch, or read part of the file.
When a file is rejected, first confirm it is a complete single file copied after the run ended, and that no other process was writing during the copy. For specific errors and next steps, see [Debug failed eval results](/docs/troubleshooting/debugging#cannot-open-results).
Record can contain sensitive business content, such as raw conversations and file changes. Before sharing, you are responsible for permissions, redaction, and retention.
## What Record does not do
Record only stores results. It does not compute statistics, comparisons, or pages. NiceEval's fixed built-in queries read the published results, then hand the complete query result to `query`, `show`, or `view`. External code never gets a database connection, a reader object, or SQL access.
To choose a way to read results, see [CLI](/docs/reference/cli).
# NiceEval's post-run data flow
Source: https://niceeval.com/docs/explanation/results-flow
Understand where run results are stored, and what each of the three read entry points, query, show, and view, is responsible for.
After a run ends, you might read results in the terminal, browse conversations in a browser, or have a CI script read JSON. All three see the same results with different layouts. This page explains where results are stored, how they are read out, and what each of the three read entry points is responsible for.
```text theme={null}
.niceeval/record.sqlite in the project (published Runs and Attempts)
↓
NiceEval's fixed built-in queries
↓
query (JSON) | show (terminal) | view (browser)
```
## Results are stored in one file
Each project's run results are stored in `.niceeval/record.sqlite`. This is Record. When `niceeval exp` starts, it creates a Run and records every Attempt planned for this run. Each Attempt publishes on its own as soon as it finishes and can be read right away, without waiting for the whole Run to end. Published Attempts and ended Runs are never modified again.
After a run has fully ended, the file itself can be copied or archived. To read a copy someone gave you, add `--record ` to `query` or `show`. NiceEval fully validates the file first, then opens it read-only.
## NiceEval fixes the read rules
When results are read, NiceEval's fixed built-in queries decide which Runs to select, up to which point to read, which results are missing, what issues exist, which evidence to use, and how to compare. A query returns plain data. It never hands the caller a database connection, a file path, or a reader object.
The built-in queries cover Runs, Attempts, traces, diffs, sources, artifacts, and comparisons between Runs. You cannot customize reads with ad hoc SQL, JSON paths, or formulas.
## Three read entry points
* `niceeval show` starts with an overview in the terminal, then drills into one Attempt's conversation, timing, and changes. Both you and Coding Agents use it to investigate problems.
* `niceeval query` outputs structured JSON for scripts and CI. Agents also use it when they need exact fields.
* `niceeval view` is for people in a browser. It opens an access-protected Insight interface that listens only on a local address.
All three entry points read the same query result. Each one only handles its own layout: JSON, terminal text, or browser page. None of them selects results on its own or recomputes pass rates or scores, so the same Run reaches the same conclusions in all three places.
For the full commands, see [CLI](/docs/reference/cli). For how the results file is copied and validated, see [Record](/docs/explanation/record).
# How a run produces readable results
Source: https://niceeval.com/docs/explanation/runner
Learn how the Runner creates a Run, publishes Attempts one by one, and why read commands only see published results.
You start `niceeval exp` and want to look at the results that are already in before it finishes. Or you want to know which results still count when a run is interrupted halfway. This page explains how the Runner turns a run into results you can read.
## A Run fixes what to run when it starts
The Runner creates one Run for each selected Experiment. At creation it records the `runId`, this run's `invocationId`, and every Attempt planned for this run. The plan does not change afterward, so the denominator of the pass rate is fixed at the start.
## Each Attempt publishes on its own when it finishes
When an Attempt finishes and its results are fully written, it publishes on its own. Once published, it can be read, compared, reused in the next run, or explicitly adopted with `niceeval accept`, even while its Run is still `active`.
Unpublished Attempts, summaries that are not finished computing, and late writes never appear in any read command. So every result you see mid-run is complete, and you never read half-written data.
## Get this run's Run ID when it ends
When `niceeval exp` ends, the terminal's `NEXT` panel lists the Runs it created and gives two commands for reading them:
```sh theme={null}
pnpm exec niceeval show --run
pnpm exec niceeval view --run
```
With `--json`, the last line is this run's receipt. Its `createdRunIds` lists the new Runs, and `publicationCutoff` marks how far publication had reached when this run ended. Do not parse `publicationCutoff`; pass it to later commands as is. Scripts and CI first run `niceeval query discover` to find the available queries, then send a complete request.
## The browser page does not jump to new results by itself
When `niceeval view` opens, it reads the results published at that moment and stays on them. When new Attempts publish later, the page prompts you to refresh. Only after you confirm does it switch to the latest results. This way the page does not change in the middle of a comparison.
The Runner never hands the database, the working directory used during execution, or unpublished records to read commands. Which results to select, what is missing, which evidence to use, and how to compare are all decided by NiceEval's fixed built-in queries. See [How run results flow](/docs/explanation/results-flow).
# Integration tiers: three levels of investment, three sets of capabilities
Source: https://niceeval.com/docs/explanation/tier
Integration has three tiers based on where the Adapter connects and what additional observability data it receives: Tier 1 connects only send, Tier 2 adds OTel, and Tier 3 changes the application to expose Experiment flags. Learn what each tier provides and when to move up.
Connecting [NiceEval](https://niceeval.com/) is not an all-or-nothing project. Based on where the Adapter connects and what additional observability data it receives, integration has three tiers. Each tier adds one kind of investment to the previous tier and provides a new set of capabilities; every eval case you have already written keeps working unchanged.
## Tier 1: connect only `send`
You do not change a line of application code. Integration is a single Adapter file whose `send` communicates through the same interface your user-facing frontend already uses.
**The full assertion set is available at this tier.** That includes text assertions and Judge checks, structured-output validation, multi-turn and session isolation, HITL approval flows, tool assertions from event mapping, and usage assertions from the usage returned by `send`. Every verdict comes from the `Turn` returned by `send`; the next two tiers add no assertions. If the application interface exposes model selection, **model comparison** Experiments also belong here: the Experiment's `model` reaches the Adapter through `ctx.model` and only needs to be forwarded with the request.
Most teams start here, and many need nothing more.
## Tier 2: `send` + OTel
Use the same `send` and event mapping, but have the application also send its OTel spans to [NiceEval](https://niceeval.com/). If the application is already instrumented with AI SDK telemetry, LangGraph, OpenLLMetry / OpenInference, or its own `gen_ai` spans, this requires no application change. Otherwise, add generic OTel initialization. That is observability work, not a change customized for eval cases.
This tier provides **observability**: the call waterfall in `niceeval view`. Each model call and tool execution inside the application appears in a per-turn timeline with its latency and tokens. Assertions are unaffected: spans go only into the waterfall, not the event stream or assertions. See [OTel Integration](/docs/tutorials/connect-otel) for the path.
## Tier 3: application changes + Experiment flags
To evaluate a **feature A/B test** such as “which Prompt, tool set, or feature toggle works better?”, the comparison point is inside the application, where the first two tiers cannot reach it. This tier changes application code to expose variants as externally selectable configuration. An Experiment's `flags` pass through `ctx.flags` to the Adapter; the Adapter sends them to the application with the request, using an HTTP header, request-body field, or environment variable. The application chooses the variant from those parameters.
What changes is the application, not the integration surface. The Adapter stays the same and still calls the application only through its external interface.
## Non-intrusive is the baseline for the first two tiers
The first two tiers are **non-intrusive**. You start the application in its normal way, such as `pnpm start` or a deployment wherever you run it. The eval side does not spawn the application process or open another port. The Adapter only communicates with **the same interface your user-facing frontend already uses**, such as an HTTP endpoint or SSE stream. If it cannot connect, it reports a clear “start the application first” error. The only difference is observability data: Tier 1 has only the events returned by `send`, while Tier 2 has an additional copy of spans sent to [NiceEval](https://niceeval.com/), so the waterfall appears.
## How to move up
The three tiers are progressive rather than mutually exclusive. Start at Tier 1 for a baseline and model comparisons. Move to Tier 2 when you need the call waterfall. Move to Tier 3 when you need to compare variants inside the application. Each upgrade only adds something to the Adapter or application. See [Experiment](/docs/explanation/experiment) for how to organize comparisons on the Experiment side.
## Related reading
* [Connect Your Agent](/docs/tutorials/connect-your-agent) — The integration overview: minimal integration and parameter channels.
* [Adapter](/docs/explanation/adapter) — The contract itself: what `send` receives and returns, and how `ctx.model` / `ctx.telemetry` / `ctx.flags` appear at each tier.
* [OTel Integration](/docs/tutorials/connect-otel) — The full Tier 2 path.
* [Experiment](/docs/explanation/experiment) — How to declare model / flags comparisons in an Experiment.
# What is NiceEval
Source: https://niceeval.com/docs/introduction
Write TypeScript evals for AI Agents and applications: assert tool calls, replies, code changes, and business outcomes, compare configurations, and investigate failures.
You revised a prompt, switched models, wrote a new Skill for Claude Code, or tuned an NPC's prompt in a game. Did it get better?
A few manual trials cannot reliably answer that. An Agent's responses vary, and one lucky answer proves little. Fixing one case can quietly break another.
NiceEval helps answer that question. It focuses on Agents and also evaluates any LLM-powered application. You define success in TypeScript, and NiceEval handles the rest:
* Connect to the subject under test and send it tasks.
* Repeat runs as needed and score each one.
* Save conversations, tool calls, file changes, duration, and cost for every run.
* Compare configurations in the terminal or browser, then inspect a specific failure to see what happened.
Everything runs on your own machine and in your CI. No account required.
## What an eval looks like
This eval checks that a weather assistant actually calls `get_weather` when asked about the weather, instead of making up an answer.
```ts theme={null}
// evals/weather-tool.eval.ts
import { defineEval, defineJudge } from "niceeval";
import { includes } from "niceeval/expect";
const groundedAnswer = defineJudge({
name: "grounded-weather-answer",
rubric: "Assess whether the answer gives specific weather information based on the question instead of refusing or answering vaguely.",
});
export default defineEval({
description: "Call the weather tool and answer based on its result",
judge: groundedAnswer,
async test(t) {
const turn = await t.send("What's the weather like in Beijing today?");
turn.succeeded().label("Turn succeeded");
t.calledTool("get_weather").label("Calls the weather tool");
t.check(turn.message, includes("weather")).label("Describes the weather");
const second = await t.send("And Shanghai tomorrow?");
t.check(second.message, includes("Shanghai")).label("Second turn follows the context");
t.judge({ question: turn.input, answer: turn.message }, groundedAnswer).gate(0.7)
.label("Answer is grounded");
},
});
```
Use deterministic rules for facts such as which tool was called or whether a reply contains a word. For open-ended quality, such as whether an answer is grounded, use a Judge model to score it. A score below 0.7 fails this check.
The Agent and model to run belong in an Experiment, separate from the eval. You can therefore use the same evals to compare two models or two prompt versions.
## What evals are for
Evals look like unit tests when you write them and work like experiments when you run them. The same set of evals can serve several purposes:
* **Prevent regressions**: rerun after changing prompts, models, or dependencies and check whether the pass rate drops. See [Rerun affected evals after changes](/docs/tutorials/rerun-and-cache).
* **Compare alternatives**: run the same evals against two models, two prompts, or with and without a Skill, then compare each case. See [Write an experiment](/docs/tutorials/write-experiment) and [Compare runs](/docs/tutorials/compare-runs).
* **Gate CI**: make a PR fail when its evals fail. See [CI integration](/docs/tutorials/ci-integration).
* **Build coverage over time**: turn failures from real use into new evals so every future change checks them.
## What you can evaluate
Use AI SDK, LangGraph, Pi, or your own Agent loop, in any language. If it exposes a callable API, you can connect it with an Adapter.
Run Claude Code or Codex in a Sandbox with a real repository and task, then score it with the project's own tests. Measure the effect of Skills, Plugins, or memory.
If your application exposes operations such as posting, replying, or NPC actions, call them directly from evals and check structured results and world state.
For applications that are not conversational Agents, evals call the application's own methods. This eval comes from the AI social application in [`examples/zh/llm-x`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/llm-x):
```ts theme={null}
// evals/social-journey.eval.ts — evaluates an AI-powered social application
export default x.defineEval({
description: "AI characters respond to posts and replies, and the social world stays consistent",
async test(t) {
const initial = await t.visitDiscoveryPage();
const post = await t.post({ intent: "Invite everyone to photograph the city at night", withImage: true });
t.check(post, authoredPost(initial.viewerId)).label("Post belongs to the current player");
const reply = await t.reply({ postId: post.id, intent: "Add that we meet at the riverside path entrance" });
const responses = await t.waitForReplies(reply.id);
t.check(responses.length, greaterThan(0)).label("AI characters respond");
t.check(worldMaterial(await t.refreshFeed()), coherentSocialWorld()).label("Social relationships survive a refresh");
},
});
```
There are two ways to connect the subject under test:
NiceEval calls your Agent or application API through an Adapter. Docker is not required. Use this for conversational Agents and applications that expose business operations.
Coding Agents need a real file system to edit code and run commands. NiceEval creates an isolated Sandbox, then starts the Agent inside it.
## How NiceEval differs from DeepEval, LangFuse, and Braintrust
DeepEval is a mature Python evaluation framework with a broad library of metrics. NiceEval makes different choices:
* **Native TypeScript**: evals, Adapters, and Experiments are typed TypeScript code.
* **Assert what happens along the way**: an Agent may need several conversation turns, tool calls, file reads, and code changes to complete a task. The final answer is only part of it. NiceEval asserts these facts directly, without requiring a golden dataset first.
* **Score in a real environment**: coding Agents run in a Sandbox and are scored using the project's own tests and file changes, beyond the reply alone.
* **Built for controlled experiments**: separate evals from Experiments to compare models, prompts, and extensions with the same cases.
Observability platforms such as LangFuse and Braintrust answer “What happened in production?” Evals answer “Is this behavior good enough?” NiceEval focuses on the latter and the local cycle of writing evals, running them, reviewing results, and improving the Agent. They can coexist: keep using those platforms for production traces and send NiceEval results to Braintrust.
## Terms you will see often
| Term | Meaning |
| - | - |
| [Eval](/docs/explanation/evals) | A test in `evals/*.eval.ts` that defines what to send and what counts as success. |
| [Experiment](/docs/explanation/experiment) | A run configuration that chooses the Agent, model, and number of runs. |
| [Adapter](/docs/explanation/adapter) | Code that connects to the subject under test, sends requests, and translates replies into events NiceEval can read. |
| [Sandbox](/docs/tutorials/sandbox-providers) | An isolated runtime environment. Coding Agents need it; directly connected applications do not. |
| Attempt | One execution of an eval. Running the same eval five times produces five Attempts. |
Other terms are explained where they first appear.
## Next
The [Quickstart](/docs/tutorials/quickstart) walks you through running your first eval and seeing results in the terminal and browser in about ten minutes.
# Built-in Agent Capabilities Reference
Source: https://niceeval.com/docs/reference/builtin-agents
What NiceEval's built-in claude-code, codex, and bub Adapters can do, which assertions they support, and their known limits.
The three built-in Sandbox Agents—`claude-code`, `codex`, and `bub`—do not all provide the same capabilities. There is no capability-declaration layer: construction and actual behavior prove them. See the [Capabilities reference](/docs/reference/capabilities). This page inventories what each built-in Agent provides, plus two built-ins for AI SDK applications: the non-intrusive HTTP Adapter `uiMessageStreamAgent`, including HITL, and the result converter `turnFromAiSdk`.
## Capability overview
| Capability | Assertion / API that corresponds to it |
| - | - |
| Send and receive messages (base contract) | `t.send()` (can be called multiple times), `t.reply`, `t.check(turn.data, equals(...))` / `matches(...)`, and status-based `t.succeeded()` |
| Event-stream completeness | Scoped assertions including `calledTool` / `usedNoTools` / `maxToolCalls` / `noFailedActions` / `event`; `toolOrder` / `eventOrder` accept matchers only on a Turn or Session. For messages, use `t.check(turn.message, includes(...))` or `t.check(turn.message, pattern(...))`. **A negative assertion such as `notCalledTool` is trustworthy only with a complete event stream.** |
| Session continuation | Cross-turn memory assertions and session isolation with `t.newSession()` |
| HITL | `t.check(turn.status, equals("waiting"))`, `t.requireInputRequest()`, `t.respond()` / `t.respondAll()` |
| `tracing` | Trace decoding and the call waterfall in `niceeval view` |
## What each built-in Agent provides
| Agent | Send and receive | Event stream | Session continuation | HITL | Tracing | Notes |
| - | - | - | - | - | - | - |
| `claude-code` | ✅ | ✅ | ✅ (`claude --resume `) | ❌ | ✅ (`http/protobuf` → OTLP, beta Flag) | The built-in parser emits raw `compaction` events automatically, read by Reports, diagnostics, and an ordinary `check(events, match)` |
| `codex` | ✅ | ✅ | ✅ (`codex exec resume `) | ❌ | ✅ (`http/json` → OTLP) | The built-in parser emits `compaction` events. |
| `bub` | ✅ | ✅ | ✅ (`--session-id` + tape continuation) | ❌ | ✅ (`http/protobuf` → OTLP) | The built-in parser emits `compaction` events. |
All three use `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`), so filesystem assertions and file I/O and command execution—including `t.sandbox.fileChanged()` and diff assertions—work with all three Agents regardless of the table.
The three built-in Sandbox Agents **do not support HITL**. `send` returns only `"completed"` / `"failed"`; it never returns `"waiting"` or emits `input.requested`.
When you need `t.respond()` / `t.requireInputRequest()`: if the system under test is an AI SDK `useChat` backend, use the built-in `uiMessageStreamAgent` below, which natively maps v7 tool approval to HITL. For another system under test, write an Adapter that implements a `waiting` status, `input.requested` event, and resume handoff. [`examples/zh/tier1/pi-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/pi-sdk) and [`examples/zh/tier1/claude-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/claude-sdk) both provide ready-made references for manual HITL.
## Details by Agent
### claude-code
* Connection: spawns `claude --print --dangerously-skip-permissions` in the Sandbox and reads the newest transcript at `~/.claude/projects/**/*.jsonl`.
* Session continuation: when `ctx.session.id` has a value, appends `--resume `; the session ID decoded from the transcript is written back with `ctx.session.capture()`.
* Authentication: `ANTHROPIC_API_KEY`, optionally `ANTHROPIC_BASE_URL`. See `ClaudeCodeConfig` below for options.
* `tracing` uses the claude CLI's native beta OTLP trace spans over `http/protobuf`. Set `CLAUDE_CODE_ENABLE_TELEMETRY` and `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA` in the environment—the explicit beta opt-in—to give the endpoint to the CLI. Trace decoding displays an interaction / llm\_request / tool hierarchy waterfall.
### codex
* Connection: runs `codex exec --json` in the Sandbox, or `codex exec resume --json` for continuation, and uses stdout JSONL as the transcript. The command includes `--dangerously-bypass-approvals-and-sandbox` and `--dangerously-bypass-hook-trust`: nobody can answer Codex's interactive confirmations in a Sandbox, so a Hook installed by a Plugin or `postSetup` can take effect without interactive trust.
* Authentication: `CODEX_API_KEY`, and optionally `CODEX_BASE_URL` for an OpenAI-compatible proxy. See `CodexConfig` below for options.
* `tracing` is configured through `[otel.trace_exporter.otlp-http]` in `~/.codex/config.toml`, over `http/json`.
### bub
* Connection: runs `bub run` with `--session-id` in the Sandbox and reads the tape at `~/.bub/tapes/.jsonl` as the transcript.
* Authentication: `BUB_API_KEY` plus `BUB_API_BASE` for an OpenAI-compatible proxy. See `BubConfig` below for options.
* `tracing` is injected through environment variables such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, over `http/protobuf`.
* Installation uses `uv tool install`, not an npm package. The first install creates a checkpoint cache to speed later Sandboxes.
## Configuration for the three built-in Sandbox Agents
### `ClaudeCodeConfig`
#### `apiKey`
```ts theme={null}
apiKey?: string;
```
The Anthropic API key. When omitted, read `ANTHROPIC_API_KEY` from the environment.
#### `baseUrl`
```ts theme={null}
baseUrl?: string;
```
A custom API base URL, for a proxy or internal endpoint. When omitted, read `ANTHROPIC_BASE_URL` from the environment. When neither exists, use the Anthropic official endpoint, the claude CLI default behavior.
#### `env`
```ts theme={null}
env?: Readonly>;
```
Extra environment variables for the Claude Code process. Values remain only at runtime and do not enter result carry identity. A non-sensitive value that changes behavior must also appear in Experiment Flags or the owning Plugin's identity.
#### `maxTurns`
```ts theme={null}
maxTurns?: number;
```
The maximum number of tool-use turns, passed to `--max-turns`. It limits Eval cost; when omitted, the CLI native default is unlimited.
#### `mcpServers`
```ts theme={null}
mcpServers?: readonly McpServer[];
```
Extra MCP servers, written to user-level `~/.claude.json` during each Sandbox setup. The stdio form writes `command`, optionally with `args` / `env`. The Streamable HTTP form writes `url`, optionally with `headers`, copied verbatim into request headers, as a `{ "type": "http", "url": …, "headers": … }` entry.
#### `skills`
```ts theme={null}
skills?: readonly SkillSpec[];
```
Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to project-level `.claude/skills//`, where the claude CLI discovers them natively.
#### `plugins`
```ts theme={null}
plugins?: readonly ClaudeCodePluginSpec[];
```
Native Claude Code Plugins: first connect a Marketplace, then install the named Plugin from it.
#### `settingsFile`
```ts theme={null}
settingsFile?: string;
```
The path to a complete official-format Claude Code `settings.json` in the local project. It resolves relative to the project root that runs niceeval—the directory containing `niceeval.config.ts`—not inside the Sandbox. It accepts only a relative path within the project root. A path containing `..`, an absolute path, a `~` path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim as the otherwise-empty user-level `~/.claude/settings.json` in the Sandbox. It does not inherit host configuration, concatenate, or reserialize. Reserved keys `model` and `env` cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.
#### `postSetup`
```ts theme={null}
postSetup?: readonly SandboxCommand[];
```
User Hooks run in array order after installation. They reuse SandboxCommand's narrow context and run only after settings are written, MCP attached, Skills and Plugins installed, and the manifest written. They suit process actions that can run only after installed artifacts are ready, such as a Plugin's setup script. A throw counts as an infrastructure error (`attempt errored`). See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
#### `preTeardown`
```ts theme={null}
preTeardown?: readonly SandboxCommand[];
```
The cleanup Hook paired with `postSetup`. It uses `postSetup`'s reverse order and runs before the Agent's own teardown step: a LIFO mirror—`postSetup` runs after Agent installation, while `preTeardown` runs before Agent cleanup. It runs only if execution reached the `postSetup` point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
### `CodexConfig`
#### `apiKey`
```ts theme={null}
apiKey?: string;
```
Proxy / OpenAI API key. Falls back to the `CODEX_API_KEY` env var if omitted.
#### `baseUrl`
```ts theme={null}
baseUrl?: string;
```
OpenAI-compatible proxy base URL (e.g. `https://s2a.example.com/v1`). Falls back to the `CODEX_BASE_URL` env var if omitted.
#### `env`
```ts theme={null}
env?: Readonly>;
```
Extra environment variables injected into every Codex CLI process. The first `codex exec` and continuing `codex exec resume` use the same declaration; lifecycle Hooks started by Codex, dynamic MCP headers, and command subprocesses all inherit it. Values enter only through Sandbox command options: they are not concatenated into shell text or written to the setup manifest, and are all redacted as potential sensitive values from timing, execution, and error evidence. `CODEX_API_KEY` still comes from `apiKey` or the same host environment variable; the Adapter's authentication value overrides a same-named key here.
An environment value does not enter carry identity. A non-sensitive value that changes behavior must also be declared in Experiment Flags or the owning Plugin identity; rotating credentials alone does not invalidate old results. `PATH` is Sandbox-managed and cannot be declared here. Its presence errors at factory construction; use the Sandbox factory's `pathPrepend` instead. See `docs/feature/sandbox/library.md`.
#### `mcpServers`
```ts theme={null}
mcpServers?: readonly McpServer[];
```
Extra MCP servers, appended to `~/.codex/config.toml` during each Sandbox setup. The stdio form (`command` / `args` / `env`) writes a `command` line in `[mcp_servers.]`. The Streamable HTTP form (`url` / `headers`) writes the `url` line and headers in the `[mcp_servers..http_headers]` subtable.
#### `skills`
```ts theme={null}
skills?: readonly SkillSpec[];
```
Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to `.agents/skills//`, and a discovery instruction is written into AGENTS.md. Codex has no native Skill tool like Claude Code, so installing the file alone does not make it read it. See `memory/codex-no-native-skill-tool.md`.
#### `plugins`
```ts theme={null}
plugins?: readonly CodexPluginSpec[];
```
Native Codex Plugins: first connect a Marketplace, then install the named Plugin from it.
#### `configFile`
```ts theme={null}
configFile?: string;
```
The path to a complete official-TOML Codex `config.toml` in the local project. It resolves relative to the project root that runs niceeval—the directory containing `niceeval.config.ts`—not inside the Sandbox. It accepts only a relative path within the project root. A path containing `..`, an absolute path, a `~` path, or a symlink that resolves outside the project root errors during setup. Its original bytes are uploaded verbatim into the otherwise-empty user-level `~/.codex/config.toml` in the Sandbox. It does not inherit host configuration or parse and rewrite it. Reserved keys `model`, `model_provider`, `model_providers`, `model_reasoning_effort`, `mcp_servers`, and `otel` cause a setup error when present. The manifest records only the project-relative path and byte SHA-256, not the contents.
#### `postSetup`
```ts theme={null}
postSetup?: readonly SandboxCommand[];
```
User Hooks run in array order after installation. They reuse SandboxCommand's narrow context and run only after the main configuration is written, MCP attached, Skills and Plugins installed, and the manifest written. They suit process actions that can run only after installed artifacts are ready, such as a Plugin's setup script. A throw counts as an infrastructure error (`attempt errored`). See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
#### `preTeardown`
```ts theme={null}
preTeardown?: readonly SandboxCommand[];
```
The cleanup Hook paired with `postSetup`. It uses `postSetup`'s reverse order and runs before the Agent's own teardown step: a LIFO mirror—`postSetup` runs after Agent installation, while `preTeardown` runs before Agent cleanup. It runs only if execution reached the `postSetup` point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
### `BubConfig`
#### `apiKey`
```ts theme={null}
apiKey?: string;
```
API key for the OpenAI-compatible proxy. Falls back to the `BUB_API_KEY` env var if omitted.
#### `apiBase`
```ts theme={null}
apiBase?: string;
```
Base URL for the OpenAI-compatible proxy. Falls back to the `BUB_API_BASE` env var if omitted.
#### `skills`
```ts theme={null}
skills?: SkillSpec[];
```
Skills installed in the Sandbox: a local directory or file, or a Repo with a pinnable ref and optional enabled subset. They go to `.agents/skills//`, and a discovery instruction is written into AGENTS.md. Bub has no native Skill-loading mechanism.
#### `version`
```ts theme={null}
version?: string;
```
Which Bub version to install, as a PyPI version such as `"0.4.0"`. When omitted, use the default version pinned by NiceEval. It is always a deterministic version and never latest: the system under test's version must be visible from Experiment configuration.
#### `otelPlugin`
```ts theme={null}
otelPlugin?: string;
```
The git dependency for the OTel tape-store Plugin, the source of the timeline. When omitted, use NiceEval's default pin.
The Plugin and Bub must be from the same generation of the tape protocol. The default pin takes types from `bub.tape` and requires Bub ≥ 0.3.10. Earlier Plugin commits validate against republic types and pair with Bub ≤ 0.3.9. A mismatched generation does not fail installation; it rejects every span and leaves the timeline silently empty. When pinning `version` backward, pin the matching Plugin commit in the same change.
#### `pythonPlugins`
```ts theme={null}
pythonPlugins?: PythonPluginSpec[];
```
Extra Python packages installed into the bub tool environment. Each Sandbox setup adds them through `uv tool install … --with `. The normalized package list enters the installation checkpoint key, so two Agent variants with different Plugin sets never reuse the same checkpoint; otherwise the second could silently receive the first variant's environment.
#### `postSetup`
```ts theme={null}
postSetup?: SandboxCommand[];
```
User Hooks run in array order after installation. They reuse SandboxCommand's narrow context and run only after bub, Skills and Python packages are installed, and the manifest is written. A throw counts as an infrastructure error (`attempt errored`). See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
#### `preTeardown`
```ts theme={null}
preTeardown?: SandboxCommand[];
```
The cleanup Hook paired with `postSetup`. It uses `postSetup`'s reverse order and runs before the Agent's own teardown step: a LIFO mirror—`postSetup` runs after Agent installation, while `preTeardown` runs before Agent cleanup. It runs only if execution reached the `postSetup` point. A throw counts as an infrastructure error, and the teardown phase closes it with a teardown-failed diagnostic. See Post-install scripts in `docs/feature/adapters/library/coding-agent-extensions.md`.
## `uiMessageStreamAgent`: built-in non-intrusive Adapter for AI SDK applications, including HITL
`uiMessageStreamAgent`, exported from `niceeval/adapter`, sends and receives non-intrusively against an HTTP endpoint implementing the [UI Message Stream protocol](https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol), the standard SSE for an AI SDK `useChat` backend. It only uses `fetch`; it does not import application code, so it can connect wherever the application is deployed:
```ts theme={null}
import { uiMessageStreamAgent } from "niceeval/adapter";
export default uiMessageStreamAgent({
name: "my-assistant",
url: "https://my-app.example.com/api/chat",
body: (ctx) => ({ model: ctx.model }), // model comparison needs no change when the application supports a per-request model
});
```
Capabilities it provides for you:
* **Send/receive and event stream**: SSE frames are reduced through the framework-agnostic `readUIMessageStream` reducer from the official `ai` package—the same reducer `useChat` uses internally. Tool calls, results, and message text are constructed directly from message parts. **The application does not need OTel.**
* **Session continuation**: the protocol is stateless on the server and the client carries complete history. The factory stores complete `UIMessage[]` history in an Adapter-private typed slot and replays it verbatim each turn. A new session line after `t.newSession()` has not written that slot yet.
* **HITL**: AI SDK v7 tool approval—a tool with `needsApproval: true`—maps natively. When a part stops at `approval-requested`, that Turn has `status: "waiting"` plus `input.requested`. `t.respond("approve" / "deny")` translates to `approval-responded`, rewrites that part in place, and resends `messages` to continue server execution. That is the same protocol behavior as a real frontend's `addToolApprovalResponse()` plus `sendMessage()`. There is no separate approval endpoint.
* **Rejection**: an invocation enters the event stream with `rejected` and, by default, a "do not retry" reason that `denyReason` can override. Without it, models often repeat the exact call.
* **Usage and waterfall**: UI Message Stream frames have no usage, so usage assertions such as `t.maxTokens` have no data by default on this built-in. An application that puts usage in message metadata is adding its own protocol extension. The waterfall is separate: when the application has OTel instrumentation, such as official `@ai-sdk/otel`, send spans to [NiceEval](https://niceeval.com/) according to [OTel integration](/docs/tutorials/connect-otel) and `niceeval view` has the complete waterfall. Spans go only to the waterfall, not assertions.
Install `ai` in the Eval project. It is an optional peer dependency because the protocol reducer comes from it. For a complete runnable example, see [`examples/zh/tier1/ai-sdk-v7`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7).
Complete `uiMessageStreamAgent(options)` parameters (`UiMessageStreamAgentOptions`):
#### `name`
```ts theme={null}
name?: string;
```
The Agent name used for Report and result-aggregation identity. Defaults to `"ui-message-stream"`.
#### `url`
```ts theme={null}
url: string | ((ctx: AgentContext) => string | Promise);
```
The chat endpoint of the application under test. Use its full URL wherever the application is deployed. The function form resolves once per turn.
#### `headers`
```ts theme={null}
headers?: globalThis.Record | ((ctx: AgentContext) => globalThis.Record);
```
Additional request headers, such as authentication. `ctx.telemetry.headers` for `traceparent` always merges in automatically.
#### `body`
```ts theme={null}
body?: (ctx: AgentContext) => globalThis.Record;
```
Fields to merge into the request body in addition to `messages`, such as `(ctx) => ({ model: ctx.model })`. Fields whose value is `undefined` are automatically omitted during serialization.
#### `projectToolCommand`
```ts theme={null}
projectToolCommand?: (tool: Readonly<{ name: string; input: JsonValue }>) => CommandProjection | undefined;
```
Lets the endpoint owner classify every logical tool call as command or not-command. An unknown call returns `undefined` and leaves actions coverage partial. NiceEval does not guess from a name or input.
#### `denyReason`
```ts theme={null}
denyReason?: string;
```
The reason sent with `approval-responded` when approval is denied. The application and SDK make it tool-result text that the model sees. A clear "do not retry" message measurably lowers the chance that a model repeats the same call.
#### `settleMs`
```ts theme={null}
settleMs?: number;
```
Milliseconds to wait after the stream ends before returning, giving application observability exporters such as BatchSpanProcessor time to export.
#### `tracing`
```ts theme={null}
tracing?: AgentTracing;
```
How an application with OTel delivers spans to the endpoint, for the waterfall. The event stream does not depend on it.
#### `spanMapper`
```ts theme={null}
spanMapper?: SpanMapper;
```
How an application with OTel normalizes spans, for the waterfall. The event stream does not depend on it.
## OpenAI-compatible response converters: `turnFromChatCompletion` / `turnFromResponses`
`turnFromChatCompletion(res)` and `turnFromResponses(res)`, exported from `niceeval/adapter`, convert OpenAI's two response shapes—Chat Completions and Responses—into a `Turn` with zero mapping. They are not limited to official OpenAI: any service that declares compatibility with either protocol shape can use them.
```ts theme={null}
import { completeEvidenceCoverage, defineAgent, turnFromChatCompletion } from "niceeval/adapter";
export default defineAgent({
name: "my-openai-compat-agent",
evidenceCoverage: completeEvidenceCoverage,
async send(input) {
const res = await client.chat.completions.create({ messages: [...], tools });
return turnFromChatCompletion(res);
},
});
```
The two shapes differ in how trustworthy a negative assertion such as `notCalledTool` is. Chat Completions does not promise that its response is a complete process—the application might run a whole tool loop server-side and return only the final answer—so a negative assertion means only "not seen," not "did not happen." The Responses protocol guarantees that its `output` array records everything the model decided to do this turn, including every `function_call`, so negative assertions are trustworthy. The `Turn` shapes output by the two converters are the same; this distinction only affects how you interpret negative assertions.
## SDK event-stream converters: `createClaudeSdkEventStream` / `createPiAgentEventStream` / `createCodexThreadEventStream`
Each Agent SDK defines its own streaming protocol, rather than a private application format. `niceeval/adapter` provides converters from native frames to standard events. When writing a non-intrusive Adapter yourself, only provide the application-interface and approval-interface addresses. `driveFrameStream`, described in the next section, reads results frame by frame, so you do not write another `for` loop:
```ts theme={null}
import { sseJsonFrames, createClaudeSdkEventStream, driveFrameStream } from "niceeval/adapter";
const frames = sseJsonFrames(res.body); // standard SSE → JSON for each frame
const stream = createClaudeSdkEventStream(); // SDKMessage → standard events
return driveFrameStream(frames, stream, ctx); // feed every frame to stream.add(), aggregate into a Turn
```
* **`createClaudeSdkEventStream`** (`SDKMessage`): `assistant` text/tool\_use blocks, `user` tool\_result blocks, `system` / `permission_denied` (→ `rejected`), and `result` usage. `markRejected()` records a rejected invocation.
* **`createPiAgentEventStream`** (pi-agent-core `AgentEvent`): `message_end` text/thinking/usage and paired `tool_execution_start/end` tools.
* **`createCodexThreadEventStream`** (Codex SDK `ThreadEvent`): messages from `agent_message` / `reasoning`. Tool items are `command_execution` / `mcp_tool_call` / `file_change` / `web_search`, each mapped to paired tool `operation.started` + `operation.finished`. It includes `turn.completed` usage and error frames. For a waterfall, use native Codex CLI OTLP—`[otel]` in config.toml—to send spans to [NiceEval](https://niceeval.com/), then normalize with official `mapCodexSpans`.
Runnable reference implementations are in [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1). The Claude SDK, pi SDK, and Codex SDK Adapters there each consist of a converter, `driveFrameStream`, and application-interface configuration.
## General assembly helpers: `driveFrameStream` / `deltaStream` + `ctx.session`
Only three pieces of a hand-written send are truly independent: transport (how to send), reduction (raw data → events, handled by the converters above), and orchestration (session continuation plus HITL pause and resume). The third has nothing to do with a particular protocol; it is a pure control-flow pattern. For server history, use `ctx.session.id` / `capture`. For client history and paused HITL state, declare an Adapter-private typed slot with `createSessionSlot()`, then use `ctx.session.get` / `set` / `take` to access it. See `AgentSession` in [Adapter concepts](/docs/explanation/adapter#context-agentcontext). Only two official helpers actually reuse across protocols:
| Helper | What it solves |
| - | - |
| `driveFrameStream(cursor, reducer, ctx, onFrame?)` | Feeds frames to a reducer one at a time, handles transport frames, and detects a HITL pause signal, collecting the loop into one function. On pause, `cursor` stays open. Use `ctx.session.set(slot, cursor)` in `onFrame` to retain it, then `ctx.session.take(slot)` on the response turn to continue reading rather than send another request. |
| `deltaStream(spec)` | A general accumulator for token-by-token or argument-by-argument deltas. Use it for protocols with no complete frame, such as raw streaming OpenAI/Anthropic APIs or a hand-written token-by-token backend. Declare which operation a frame belongs to—text delta, tool-argument delta, or finishing—and it handles concatenation timing. |
```ts theme={null}
const pendingSlot = createSessionSlot("my-adapter/pending");
// On entry to send, first check for retained state from a paused HITL turn.
const pending = ctx.session.take(pendingSlot);
if (pending) {
// …match decisions against input.responses and return approval to the application…
return driveFrameStream(pending.cursor, pending.stream, ctx, onFrame);
}
// Normal turn: session continuation uses ctx.session.id / capture; retain the cursor in a typed slot on pause.
return driveFrameStream(cursor, stream, ctx, (frame, derived) => {
ctx.session.capture(stream.sessionId);
const gated = derived.find((event) =>
event.type === "operation.started" &&
event.operation.kind === "tool" &&
event.operation.name === GATED_TOOL_NAME
);
if (gated?.type === "operation.started") {
ctx.session.set(pendingSlot, { cursor, stream, toolUseId: gated.operationId });
return { pause: { id: gated.operationId, action: GATED_TOOL_NAME, options: [{ id: "approve" }, { id: "deny" }] } };
}
});
```
The three Tier 1 examples—claude-sdk and pi-sdk with `driveFrameStream` plus a typed slot, and codex-sdk with `driveFrameStream` only—have already been rewritten around these helpers. An Adapter is left only with transport and the decision in `onFrame` about whether a frame needs extra handling; it has no hand-written loop or module-level Map.
## `turnFromAiSdk`: AI SDK result → event-stream converter
`turnFromAiSdk`, exported from `niceeval/adapter`, is for your own Adapter, such as direct server construction for an HTTP web Agent. See `examples/zh/ai-sdk/`. It maps AI SDK `generateText` / `streamText` results into `{ events, usage, status }`. It pairs `toolCallId` exactly, preserves order, maps `tool-error` to a failed tool `operation.finished`, and aggregates usage across v4 / v5 / v7 field drift. Spread it directly into a `Turn`:
```ts theme={null}
import { completeEvidenceCoverage, defineAgent, turnFromAiSdk } from "niceeval/adapter";
export default defineAgent({
name: "my-ai-sdk-agent",
evidenceCoverage: completeEvidenceCoverage,
async send(input) {
const result = await generateText({ model, tools, prompt: input.text });
return { ...turnFromAiSdk(result), data: result.text };
},
});
```
The converter supplies `status`: it is `"waiting"` when a tool approval awaits a person, with an `input.requested` event; otherwise it is `"completed"`. Session continuation for multiple-turn resume, returning an HITL decision, and tracing depend on how you write your own `send`.
## How to choose
* For an AI SDK application with a `useChat` backend: use the built-in `uiMessageStreamAgent` for non-intrusive, zero-mapping integration, including HITL.
* For another Agent system, such as an HTTP / gRPC service: integrate non-intrusively and write event mapping. The official converters cover most of it. The five examples under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) all take this route. If the application already has OTel, connect it too for an extra waterfall.
* For a call waterfall—the trace in `niceeval view`: all three `claude-code` / `codex` / `bub` send OTLP themselves. claude-code uses CLI beta telemetry, whose spans have only structure and timing. A self-written Adapter only needs to declare `tracing`; see the OTel guide.
* For human approval or multi-step confirmation (HITL): `uiMessageStreamAgent` provides it natively through AI SDK v7 tool approval. The three Sandbox Agents do not. For another system, write an Adapter and compose it with `driveFrameStream` plus a typed slot; Tier 1 pi-sdk / claude-sdk have ready patterns.
* To run a coding Agent that changes code, inspect diff, and evaluate tool calls: use `claude-code` / `codex` / `bub` for send/receive, event stream, session continuation, workspace, and Sandbox.
* When the backend protocol is your own and no official SDK converter applies, do not write send from zero. Choose a reduction shape: a small mapping for complete frames or `deltaStream` for token deltas. Use `ctx.session.id` / `capture` for server session continuation, a typed slot for client history or HITL state, and compose them into send.
## Related reading
* [Connect your Agent](/docs/tutorials/connect-your-agent) — how to implement each capability and which assertion it supports when writing an Adapter.
* [Sandbox Agent](/docs/tutorials/sandbox-agent) — run a built-in Sandbox Agent and write your own.
* [Adapter Reference](/docs/reference/define-agent) — complete parameters for `defineAgent` / `defineSandboxAgent`.
* [OTel integration](/docs/tutorials/connect-otel) — send application spans to [NiceEval](https://niceeval.com/) to get the call waterfall in `niceeval view`.
# Capabilities and Evidence Coverage Reference
Source: https://niceeval.com/docs/reference/capabilities
How an Agent's construction, runtime behavior, and required six-channel evidenceCoverage declaration together determine what NiceEval assertions can trust.
Capabilities answer two different questions:
1. **What can the Runner construct or call?** An Agent's construction and runtime behavior provide Sandbox access, tracing, session continuation, and HITL.
2. **What conclusions can assertions trust?** Every Agent must use `evidenceCoverage` to declare the completeness of six evidence channels.
There is no separate `capabilities` field on `defineAgent` / `defineSandboxAgent`. That does not mean "nothing needs to be declared": `evidenceCoverage` is required on both definitions so missing evidence never becomes an ambiguous fourth state.
## Construction and runtime capabilities
| Capability | How to obtain it | Effect on evidence |
| - | - | - |
| `t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions | Construct with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`) | Sandbox-result assertions consume the Attempt's agent-attributed diff; they do not correspond to an `evidenceCoverage` channel |
| `tracing` (OTLP receiver → `niceeval view` waterfall) | Configure `tracing` for a CLI-style Agent. Configure a fixed `telemetry` port for a long-running application | Spans go only to the timing waterfall; they cannot fill a gap in behavior evidence |
| Cross-turn continuation and `t.newSession()` isolation | Continue the backend session with `ctx.session.id` / `capture`, or save typed adapter-private history with `ctx.session.get` / `set` | The session connection determines actual behavior; do not infer it from an evidence bit |
| HITL (`t.respond()`) | `send` returns `status: "waiting"` and `input.requested` | The next response is matched by structured request ID |
| Compaction visibility | An official parser or complete manual mapping emits a `compaction` event | Raw events go to reports and diagnostics; they can also be the subject of an ordinary `check(events, match)` |
## The required six-channel declaration
Both `defineAgent` and `defineSandboxAgent` require an `EvidenceCoverage`:
```ts theme={null}
interface EvidenceCoverage {
readonly events: EvidenceCoverageEntry;
readonly actions: EvidenceCoverageEntry;
readonly messages: EvidenceCoverageEntry;
readonly usage: EvidenceCoverageEntry;
readonly status: EvidenceCoverageEntry;
readonly data: EvidenceCoverageEntry;
}
type EvidenceCoverageEntry =
| { readonly status: "complete"; readonly reason?: never }
| {
readonly status: "partial" | "unavailable";
readonly reason: string;
};
```
Use `completeEvidenceCoverage` only when the Adapter really captures every channel completely. An official converter provides its coverage declaration for the protocol it normalizes. A manual mapping must state all six channels honestly:
```ts theme={null}
import type { EvidenceCoverage } from "niceeval/adapter";
const evidenceCoverage = {
events: { status: "partial", reason: "the endpoint emits final events only" },
actions: { status: "unavailable", reason: "the endpoint does not expose tool lifecycles" },
messages: { status: "complete" },
usage: { status: "unavailable", reason: "the endpoint does not return token usage" },
status: { status: "complete" },
data: { status: "complete" },
} satisfies EvidenceCoverage;
```
`Turn.evidenceCoverage` is an optional per-turn **downgrade**. List only channels that are worse for that turn than the Agent default, such as when a stream ends before usage arrives. Omitted channels inherit the Agent declaration. A Turn cannot upgrade the default coverage.
## How coverage changes assertion results
Coverage prevents "not collected" from looking like "confirmed not to have happened":
| Check | Required channel is complete | Required channel is partial / unavailable |
| - | - | - |
| A positive assertion finds matching evidence | `passed` | `passed` — evidence that exists is still valid |
| A positive assertion finds no match | `failed` | `unavailable` — a collection gap cannot prove the Agent did not act |
| A negative assertion such as `notCalledTool()` / `notEvent()` | Evaluated normally | `unavailable` |
| An upper-bound assertion such as `maxTokens()` / `maxCost()` | Evaluated normally | `unavailable` |
An `unavailable` assertion is always persisted with a machine-readable reason. It is never silently discarded or folded into a pass:
* A non-optional unavailable assertion makes the Attempt `errored`.
* An assertion explicitly chained with `.optional()` remains visible as unavailable, but does not affect the Verdict.
Judge follows the same Verdict rule, but has its own unavailable reasons: an unresolved Judge model or API key, a failed Judge call, or a response without an extractable score all produce `unavailable`. Judge is therefore not "always available."
## What happens when a manual mapping misses events
If a manual mapping misses events but declares `events` or `actions` complete, negative assertions can appear to pass on an incomplete picture. That is an incorrect completeness declaration. Until the mapping covers successful, failed, rejected, and concurrent tool lifecycles, declare `partial` or `unavailable` honestly and give a reason.
An official converter guarantees only the surface its protocol contract explicitly covers. OTel spans cannot repair an incomplete event mapping: spans go only to the waterfall, not assertions.
## Related reading
* [Event stream reference](/docs/reference/events) — the standard event stream contract.
* [Adapter concepts](/docs/explanation/adapter) — the full `ctx` / `Turn` contract.
* [defineAgent and defineSandboxAgent](/docs/reference/define-agent) — Agent fields and examples.
* [Assertions](/docs/explanation/assert) — unavailable, `.optional()`, and Verdict propagation.
* [OTel integration](/docs/tutorials/connect-otel) — why spans feed visualization only, not assertions.
* [Built-in Agent capabilities](/docs/reference/builtin-agents) — what each built-in Adapter actually captures.
# CLI
Source: https://niceeval.com/docs/reference/cli
Read run records with query, show, and view.
Run evaluations with `niceeval exp`. There are three ways to read results: scripts and CI use `niceeval query` to get JSON, people and coding agents use `niceeval show` to drill down in a terminal, and `niceeval view` is for review in a browser.
All three read the same run record, `.niceeval/record.sqlite`. At the start of every read, NiceEval fixes a visibility boundary: results already written at that moment are visible, and anything written afterward does not leak into this read. Attempts already written are readable even while their Run is still running.
When you pipe the output of `niceeval exp`, stdout still shows progress. Each line carries the Experiment, the Eval, and the Attempt number. The same Attempt prints at most once per second, and nothing is repeated when the content has not changed. Run directly in a terminal, it shows a live panel. Add `--json` when a script needs structured results.
Pass `--experiment` repeatedly to run several Experiments at once, for example `niceeval exp --experiment baseline --experiment memory --max-concurrency 4`. These Experiments share the concurrency limit and budget within the same run. An ID given twice runs only once. If any ID does not exist, the whole command fails before execution starts. Once you use `--experiment`, positional arguments filter Evals.
## `niceeval query`
```sh theme={null}
niceeval query discover
niceeval query explain [--record ] --request
niceeval query run [--record ] --request
```
`query` uses the versioned JSON request protocol `niceeval.query/v1`. Each fixed read question is called an operation. A successful result and a failure defined by the protocol each write exactly one JSON document to stdout. When arguments are wrong, or the process fails before it can produce a document, the error goes only to stderr.
* `discover` lists the operation IDs the current version supports. It does not read the run record, so it accepts no `--record`, and it does not return the request field structure.
* `explain` validates a complete request and explains which operation it will run and which requests you can send next.
* `run` executes the request and returns that operation's complete result.
First use `discover` to confirm which operations the current version has. Then write the request from the minimal requests on this page or in [Viewing results](/docs/tutorials/viewing-results), and validate it with `explain`. Do not guess fields, and do not build another query language from SQL, JSON paths, or formulas.
Paged results carry a continuation token. The token remembers the operation, the request content, the record file being read, and the visibility boundary of this read. If any of them changes, `query` does not continue reading. It returns a restart correction, which means the caller must start again from `discover` or by resending the request.
## `niceeval show`
```sh theme={null}
niceeval show [--record ]
niceeval show --run ... [--record ]
niceeval show --experiment ... [--record ]
niceeval show @ [--record ]
niceeval show @ --source [--record ]
niceeval show @ --execution [--expand ] [--record ]
niceeval show @ --timing [--record ]
niceeval show @ --usage [--record ]
niceeval show @ --diff [--record ]
```
`show` displays results in a fixed terminal format. With no selection arguments, it shows the default overview. `--run ` shows the given Run and `--experiment ` shows the given Experiment; both can be repeated.
Pass rates and scores are displayed by Eval type:
* Pass/fail Evals show `Verdicts` and `Pass rate`.
* Score-only Evals show `Verdicts` and `Score`, with no pass rate.
* When both are mixed, only the pass/fail rows show a pass rate.
A "complete" score only means the score could be computed; it does not mean the Eval passed. An Eval whose Verdict is `failed` shows as failed even with a complete score, and keeps the score it earned. Only Evals with a `passed` Verdict and a complete score enter the success ranking.
With a `passed` Verdict and a complete score, terminal details show `scored` and the browser shows "Scored". A zero score can also be complete: it says scoring finished, not that the goal was met, and NiceEval has no default passing line. A gate failure still shows as failed. An incomplete (`partial`) score keeps only a lower bound of the known score and does not count as a valid score.
`@` selects an Attempt. A command takes at most one locator, and it cannot be used with `--run` or `--experiment`; `--run` and `--experiment` cannot be used together either.
After selecting an Attempt, add any one of the options below to look at one aspect. The five options are mutually exclusive, and each requires an Attempt locator:
* `--source`: the recorded source information and where each assertion sits in the source.
* `--execution`: an outline of the execution, with a length limit.
* `--timing`: how long each phase took.
* `--usage`: usage totals and every reported usage entry.
* `--diff`: file changes during the run.
`--execution` output shows three things separately: the collection state, whether the reporter declared the data complete, and the paging state. Even when the current page is empty or it is the last page, a `partial` mark from the reporter and the reason for the limit are kept.
* Filter by participant and event type with `--actor ` and `--type `.
* When paging with `--continuation `, pass the same filters.
* Expand one item directly with `--expand `. The ID must come from an `itemId`, `toolOccurrenceId`, or `commandId` in the outline, not a positional number. `--expand` works only with `--execution`, and cannot appear together with filter or paging arguments.
## `niceeval view`
```sh theme={null}
niceeval view [--run ...] [--no-open] [--port ] [--json]
```
`view` opens NiceEval's built-in browser interface on your machine. Without `--run`, it opens the default overview; repeat `--run` to preselect several Runs. It accepts no positional arguments and does not load the project's own pages, themes, components, or renderers.
`--no-open` only starts the server without opening a browser. `--port` sets the port; the server listens only on `127.0.0.1`. Once page assets, local access authorization, and the read endpoint are ready, the command writes an openable local address to stdout once.
With `--json`, stdout switches to NDJSON in the `niceeval.view/v1` format: a `ready` event when ready and a `closed` event on close. Startup or runtime failures are reported through the exit code and stderr.
`view` reads only the current project's `.niceeval/record.sqlite`. When new results are written while it is open, the page prompts you to refresh and switches to the new data only after you confirm. It has no `--record`, SQL, deployment, or sharing options, and cannot specify custom reports, pages, themes, routes, or operations.
## Choose a read path
All three paths read the same results, but each owns its own layout and presentation.
| Goal | Use |
| - | - |
| Give a script or CI stable JSON | `niceeval query discover`, then `explain` / `run` |
| Let a person or coding agent view a summary of the overview, a Run, an Experiment, or an Attempt in a terminal | `niceeval show` |
| Review the overview or a Run in a browser | `niceeval view` |
| Read the same results on another machine | Copy `.niceeval/record.sqlite`, then point to it with `--record` |
## CLI flags
The table below is generated from each command's own option definitions. Flags not in the table are rejected as unknown arguments.
| Flag | Command | Type | Description |
| - | - | - | - |
| `--help` | `niceeval` | boolean | Print the root command index. |
| `--version` | `niceeval` | boolean | Print the installed NiceEval version. |
| `--tag` | `niceeval check`, `niceeval exp`, `niceeval exp list` | string | Select only Evals with this tag. |
| `--help` | `niceeval check`, `niceeval exp`, `niceeval exp list`, `niceeval exp rename`, `niceeval exp --teardown`, `niceeval debug`, `niceeval accept`, `niceeval session` | boolean | Print help for this command. |
| `--experiment` | `niceeval exp` | string\[] | Narrow the current project's results by Experiment selector. |
| `--attempts` | `niceeval exp` | number | Run each selected Eval the given number of times. |
| `--max-concurrency` | `niceeval exp` | number | Limit the number of Attempts executing at the same time. |
| `--max-build-concurrency` | `niceeval exp` | number | Limit the number of Sandbox builds prepared at the same time. |
| `--max-setup-prefix-concurrency` | `niceeval exp` | number | Limit the number of Sandbox setup prefixes prepared at the same time. |
| `--sandbox-setup-cache` | `niceeval exp` | string | Use or bypass the Sandbox preparation cache. |
| `--timeout` | `niceeval exp` | number | Set each Attempt's timeout in milliseconds. |
| `--budget` | `niceeval exp` | number | Set this Invocation's budget in US dollars. |
| `--junit` | `niceeval exp` | string | Write the required JUnit report. |
| `--json` | `niceeval exp` | boolean | Write this command's machine document or event stream. |
| `--dry` | `niceeval exp` | boolean | Plan without creating an Invocation. |
| `--rerun` (alone means `failed`) / `--rerun=failed\|all` / `--rerun failed\|all` | `niceeval exp` | `failed` \| `all` | Request a rerun of failed targets or all targets. |
| `--keep-sandbox` (alone means `failed`) / `--keep-sandbox=failed\|all` | `niceeval exp` | `failed` \| `all` | Keep the Sandbox of failed or all finished Attempts. |
| `--early-exit` / `--no-early-exit` | `niceeval exp` | boolean | Stop the Eval's remaining Attempts after one result passes. |
| `--record` | `niceeval exp`, `niceeval accept` | string | Use the given NiceEval Record root. |
| `--json` | `niceeval exp list` | boolean | Write this command's machine document or event stream. |
| `--run` | `niceeval exp rename` | string\[] | Select one exact Run to adopt into the target Experiment; repeating it is rejected. |
| `--dry` | `niceeval exp rename` | boolean | Preview an explicit Experiment rename. |
| `--json` | `niceeval exp rename` | boolean | Write this command's machine document or event stream. |
| `--teardown` | `niceeval exp --teardown` | boolean | Run explicit Experiment teardown recovery. |
| `--json` | `niceeval exp --teardown` | boolean | Write this command's machine document or event stream. |
| `--recover-shared-state` | `niceeval exp --teardown` | string | Name the shared state key to recover explicitly. |
| `--owner-token` | `niceeval exp --teardown` | string | Provide the exact shared state owner token. |
| `--confirm-owner-terminated` | `niceeval exp --teardown` | boolean | Confirm that the recorded owner has terminated. |
| `--confirm-remote-quiesced` | `niceeval exp --teardown` | boolean | Confirm that the remote shared state is quiescent. |
| `--json` | `niceeval debug` | boolean | Write this command's machine document or event stream. |
| `--run` | `niceeval accept` | string\[] | Adopt one published exact Run; repeating it is rejected. |
| `--dry` | `niceeval accept` | boolean | Complete the whole-Run adoption preflight without publishing a new Run. |
| `--all` | `niceeval session` | boolean | Include status entries for completed Invocations. |
| `--json` | `niceeval session` | boolean | Write this command's machine document or event stream. |
| `--request` | `niceeval query` | string | Read one niceeval.query/v1 request from a file or standard input. |
| `--record` | `niceeval query`, `niceeval show` | string | Read an external canonical SQLite Record. |
| `--help` | `niceeval query` | boolean | Print help for this command. |
| `--run` | `niceeval show` | string\[] | Show one sealed Run; can be repeated. |
| `--experiment` | `niceeval show` | string\[] | Show one exact Experiment; can be repeated. |
| `--all` | `niceeval show` | boolean | Show Attempts hidden in the compact Results view. |
| `--source` | `niceeval show` | boolean | Show an Attempt's captured source facts. |
| `--execution` | `niceeval show` | boolean | Show an Attempt's bounded execution outline. |
| `--actor` | `niceeval show` | string | Filter execution events by exact actor ID. |
| `--type` | `niceeval show` | string | Filter execution events by exact event type. |
| `--continuation` | `niceeval show` | string | Read the next execution page with the same filters. |
| `--timing` | `niceeval show` | boolean | Show an Attempt's captured timing activities. |
| `--usage` | `niceeval show` | boolean | Show an Attempt's captured usage totals. |
| `--diff` | `niceeval show` | boolean | Show an Attempt's captured file changes. |
| `--expand` | `niceeval show` | string | Expand one stable execution identity. |
| `--help` | `niceeval show` | boolean | Print help for this command. |
| `--run` | `niceeval view` | string\[] | Select one sealed Run; can be repeated. |
| `--no-open` | `niceeval view` | boolean | Do not ask the operating system to open a browser. |
| `--port` | `niceeval view` | string | Choose the loopback listening port; 0 picks one automatically. |
| `--json` | `niceeval view` | boolean | Output View lifecycle NDJSON, not Inspection results. |
| `--help` | `niceeval view` | boolean | Print help for this command. |
| `--all` | `niceeval state migrate` | boolean | Migrate all registered first-party Service state modules. |
| `--help` | `niceeval state migrate` | boolean | Print help for this command. |
| `--help` | `niceeval init` | boolean | Print help for this command. |
| `--json` | `niceeval docker profile list`, `niceeval docker profile doctor`, `niceeval docker cache inventory`, `niceeval docker cache gc` | boolean | Write this command's machine document or event stream. |
| `--domain` | `niceeval docker cache inventory`, `niceeval docker cache gc` | string | Select one Docker cache domain managed by NiceEval. |
| `--apply` | `niceeval docker cache gc` | string | Apply a previously issued Docker cache reclamation plan. |
| `--help` | `niceeval docker` | boolean | Print help for this command. |
| `--all` | `niceeval sandbox stop` | boolean | Destroy all retained Sandboxes. |
| `--window` | `niceeval sandbox diff` | string | Select one recorded change window. |
| `--path` | `niceeval sandbox diff` | string | Select one path in the Sandbox diff. |
| `--leave-running` | `niceeval sandbox enter` | boolean | Keep the Sandbox after leaving the shell. |
| `--record` | `niceeval sandbox list`, `niceeval sandbox enter`, `niceeval sandbox history`, `niceeval sandbox diff`, `niceeval sandbox stop`, `niceeval sandbox prune` | string | Use the given NiceEval Record root. |
| `--orphans` | `niceeval sandbox list` | boolean | Check for unregistered Sandbox instances left by terminated runs. |
| `--force` | `niceeval sandbox prune` | boolean | Also clean up unverified orphan candidates. |
| `--development` | `niceeval sandbox` | boolean | Diagnose the isolated Incus development domain instead of reference. |
| `--help` | `niceeval sandbox` | boolean | Print help for this command. |
| `--tag` | `niceeval list` | string | Select only Evals with this tag. |
| `--help` | `niceeval list` | boolean | Print help for this command. |
# Define What You Evaluate: Adapter Reference
Source: https://niceeval.com/docs/reference/define-agent
Context, assertion methods, and resource lifecycle for custom Adapters and Agent integrations.
An Adapter connects the object you evaluate and provides its operations and evidence. Agents, games, and ordinary applications all use NiceEval's Experiments, assertions, and result queries.
A custom object provides native methods through `defineAdapter` and can bind domain checks in `assertions`; several implementations share an interface and assertions through `defineAdapterContract`.
For the complete definition and how to run it, see [Evaluate a custom application](/docs/tutorials/custom-application).
`defineAgent` and `defineSandboxAgent` below are dedicated to conversational integration, using `send` to return the standard event stream.
## `defineAgent`
Use this for direct agent integration:
```ts theme={null}
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
```
An Agent produced by `defineAgent` always has `Agent.kind` set to `"direct"` (an internal discriminator you do not declare). `t` has no capability bits that must be declared to unlock them; filesystem assertions such as `t.sandbox` exist only on Agents built with `defineSandboxAgent` (`kind: "sandbox"`). Everything else is determined by the events `send` actually returns and how `ctx.session` is used; see the [Capabilities reference](/docs/reference/capabilities). For the complete field set, see "Agent and Adapter Context fields" below.
### Direct agent example
```ts theme={null}
export default defineAgent({
name: "echo",
evidenceCoverage: completeEvidenceCoverage,
async send(input) {
return {
status: "completed",
events: [{ type: "message", role: "assistant", text: input.text }],
};
},
});
```
## `input` (`TurnInput`)
The text passed to the current `t.send(...)`.
Files attached to this turn (images and so on). An adapter that does not support multimodal input can ignore them.
Present only on response turns (`t.respond` / `t.respondAll`): structured answers per request, matched by `requestId`.
## `defineSandboxAgent`
Use this for coding agent CLIs. The resulting `Agent.kind` is always `"sandbox"`; filesystem assertions such as `t.sandbox` and `t.sandbox.fileChanged()` are unlocked only on this kind of agent. The complete field set is also in "Agent and Adapter Context fields" below, under `SandboxAgentDef`. The complete methods of `ctx.sandbox` (the `Sandbox` interface) are in "Sandbox interface" further below.
```ts theme={null}
import { completeEvidenceCoverage, defineSandboxAgent } from "niceeval/adapter";
import { command } from "niceeval/sandbox";
export default defineSandboxAgent({
name: "my-cli-agent",
evidenceCoverage: completeEvidenceCoverage,
ensure: {
identity: { agent: "my-cli-agent", version: "1.0.0", revision: "1" },
probe: command("my-agent", ["--version"]),
},
async send(input, ctx) {
await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
return { status: "completed", events: [] };
},
});
```
## Register it
```ts theme={null}
import { defineExperiment } from "niceeval";
import echo from "./agents/echo";
export default defineExperiment({
agent: echo,
attempts: 1,
});
```
## Agent and Adapter Context fields
The construction parameters of `defineAgent` (`DirectAgentDef`) and `defineSandboxAgent` (`SandboxAgentDef`),
plus the `ctx` (`AgentContext`) that both receive in `send(input, ctx)`:
A custom Adapter's `create(ctx)` uses `AdapterCreateContext`, and its cleanup callback uses `AdapterCleanupContext`.
### `AdapterAssertionsFactoryContext`
#### `app`
```ts theme={null}
readonly app: MatchContext;
```
#### `check`
```ts theme={null}
readonly check: AssertionCheck<"polymorphic", Context>;
```
### `AdapterCleanupContext`
#### `timeoutMs`
```ts theme={null}
readonly timeoutMs: number;
```
A fixed millisecond budget for the whole cleanup window; it is not renewed per callback.
#### `deadlineAt`
```ts theme={null}
readonly deadlineAt: number;
```
The Unix millisecond deadline fixed when the cleanup window actually starts.
#### `signal`
```ts theme={null}
readonly signal: AbortSignal;
```
Independent of the execution signal; aborts when the whole Adapter cleanup window reaches its deadline.
### `AdapterCreateContext`
#### `models`
```ts theme={null}
readonly models: ResolvedModelSlots;
```
Frozen application model selections for this Attempt.
#### `recordUsage`
```ts theme={null}
recordUsage(input: import("./adapter-usage.ts").AdapterUsageInput): void;
```
#### `sealUsage`
```ts theme={null}
sealUsage(input: import("./adapter-usage.ts").AdapterUsageSeal): void;
```
Close application usage admission and declare whether the recorded collection is complete.
#### `attach`
```ts theme={null}
attach(input: import("./adapter-attachments.ts").AdapterAttachmentInput): Promise;
```
#### `recordTrace`
```ts theme={null}
recordTrace(
input: import("./adapter-execution-trace.ts").ExecutionTraceInput,
): Promise;
```
#### `evalId`
```ts theme={null}
readonly evalId: string;
```
The public ID of the current Eval.
#### `experimentId`
```ts theme={null}
readonly experimentId: string;
```
The public ID of the current Experiment.
#### `attempt`
```ts theme={null}
readonly attempt: number;
```
The zero-based index of the current Attempt.
#### `signal`
```ts theme={null}
readonly signal: AttemptSignal;
```
The Attempt execution signal; its reason distinguishes timeout from cancelled, and it is not used for the cleanup window.
#### `model`
```ts theme={null}
readonly model?: string;
```
The model selected by the Experiment.
#### `reasoningEffort`
```ts theme={null}
readonly reasoningEffort?: string;
```
The reasoning effort selected by the Experiment.
#### `flags`
```ts theme={null}
readonly flags: Flags;
```
The read-only flags the Experiment passes to the Adapter.
#### `progress`
```ts theme={null}
progress(update: ProgressUpdate): void;
```
Updates the human-readable progress of the current Attempt.
#### `diagnostic`
```ts theme={null}
diagnostic(input: DiagnosticInput): void;
```
Appends a structured diagnostic to the current Attempt.
#### `log`
```ts theme={null}
log(message: string): void;
```
Appends a log line to the current Attempt.
#### `onCleanup`
```ts theme={null}
onCleanup(cleanup: (context: AdapterCleanupContext) => void | Promise): void;
```
Registers release of an Attempt-local resource. Callbacks run in global LIFO order; one failure does not skip the rest.
Zero-argument callbacks can still be passed directly. The context passed in is frozen, and callbacks in the same cleanup window share one signal.
### `DirectAgentDef`
#### `name`
```ts theme={null}
name: string;
```
The agent's display name/identifier, copied verbatim into `Agent.name`. It is not a registry lookup key; it is used only for display, result attribution, and the dedupe fingerprint.
#### `evidenceCoverage`
```ts theme={null}
evidenceCoverage: EvidenceCoverage;
```
This Adapter's normal evidence coverage declaration; use `completeEvidenceCoverage` for complete collection.
#### `setup`
```ts theme={null}
setup?: DirectAgentSetup;
```
Once per attempt. A Direct Agent does not receive a Sandbox; commonly used for one-time preparation such as opening connections and authentication.
#### `tracing`
```ts theme={null}
tracing?: Omit;
```
OTLP export configuration: how the system under test sends traces to the endpoint (env-based injection).
#### `spanMapper`
```ts theme={null}
spanMapper?: SpanMapper;
```
A thin mapper from native spans → canonical; omitted uses the generic heuristic. Affects only the waterfall.
#### `send`
```ts theme={null}
send(input: TurnInput, ctx: AgentSendContext): Promise;
```
Once per turn: sends one turn's prompt directly to a function, SDK, or service endpoint, and parses the response into events. The author callback keeps the Promise ABI.
#### `classifySendFailure`
```ts theme={null}
classifySendFailure?: SendFailureClassifier;
```
Optional send execution failure classifier: see `Agent.classifySendFailure`.
#### `teardown`
```ts theme={null}
teardown?: DirectAgentTeardown;
```
Cleanup before the run ends, executed if and only if this attempt reached the `setup` point (a `setup` throw does not exempt it),
run once in finally.
### `SandboxAgentDef`
#### `name`
```ts theme={null}
name: string;
```
The agent's display name/identifier, copied verbatim into `Agent.name`. It is not a registry lookup key; it is used only for display, result attribution, and the dedupe fingerprint.
#### `evidenceCoverage`
```ts theme={null}
evidenceCoverage: EvidenceCoverage;
```
This Adapter's normal evidence coverage declaration; use `completeEvidenceCoverage` for complete collection.
#### `sandbox`
```ts theme={null}
sandbox?: SandboxLayer<"command-only">;
```
The Adapter's own Sandbox actions; an Agent cannot choose or contribute a template.
#### `ensure`
```ts theme={null}
ensure: AgentEnsure | readonly AgentEnsure[];
```
A single entry or an array is normalized into Agent layers in declaration order.
#### `installers`
```ts theme={null}
installers?: readonly AgentInstaller[];
```
Paired installation layers; omitted means this adapter provides only the probe protocol, and on a miss the Runner
names the exact missing identity at `agent.ensure`. The factory normalizes an omission into an empty array.
#### `setup`
```ts theme={null}
setup?: AgentSetup;
```
Once per Attempt (not once per send): writes config.toml / authentication configuration (model/base/auth and other
runtime configuration that does not change within this Attempt). Probing, installing, and rechecking the CLI belong to `agent.ensure`. The runner calls it once after the Sandbox
is ready (after layer prepare/ensure/baseline) and before the first send; it returns no value.
#### `tracing`
```ts theme={null}
tracing?: AgentTracing;
```
OTLP export configuration: how to make the CLI in the Sandbox send traces to the endpoint (env / config file), split out of setup.
#### `spanMapper`
```ts theme={null}
spanMapper?: SpanMapper;
```
A thin mapper from native spans → canonical; omitted uses the generic heuristic. Affects only the waterfall.
#### `send`
```ts theme={null}
send(input: TurnInput, ctx: SandboxAgentSendContext): Promise;
```
Once per turn: runs the prompt (fresh / resume) and parses it into events. The author callback keeps the Promise ABI.
#### `classifySendFailure`
```ts theme={null}
classifySendFailure?: SendFailureClassifier;
```
Optional send execution failure classifier: see `Agent.classifySendFailure`.
#### `teardown`
```ts theme={null}
teardown?: AgentTeardown;
```
Cleanup before the Sandbox is destroyed, executed if and only if this attempt reached the `setup` point (a `setup` throw does not exempt it),
run once in finally.
### `AgentContext`
#### `signal`
```ts theme={null}
readonly signal: AbortSignal;
```
Soft cancellation signal: combines the attempt timeout, run-level interruption (user Ctrl+C), and the Eval's own interruption request
(see src/runner/attempt.ts). An adapter can optionally check it (or pass it straight to `fetch`) to exit
gracefully early, but it is not the only hard boundary: even if the adapter ignores it entirely, the runner uses
`Effect.timeoutTo` as a backstop to force teardown (stopping the Sandbox container).
#### `model`
```ts theme={null}
readonly model?: string;
```
The model name used by this attempt, passed through from the Experiment's `model` field. Sandbox agents usually write it into configuration in setup; direct agents usually use it in send to select the model.
#### `reasoningEffort`
```ts theme={null}
readonly reasoningEffort?: string;
```
The model's reasoning effort; owned the same way as model: the Experiment decides it, and when omitted the agent's native default is not overridden.
#### `flags`
```ts theme={null}
readonly flags: ExperimentFlags;
```
The Experiment's `flags` field passed through verbatim; its content and structure are entirely defined by the Experiment author
(such as `{ webResearch: true }`, `{ systemPrompt: "..." }`). The adapter reads its fields by its own convention;
the framework itself neither interprets nor validates the content. The name deliberately avoids the CLI-parsed
`flag` (run-level --timeout/--budget and so on); the two are unrelated concepts.
#### `experimentId`
```ts theme={null}
readonly experimentId?: string;
```
The Experiment id derived from the path (the same source as result attribution `runWho` / `AgentRun.experimentId`); undefined when not running
through an Experiment (such as outside the CLI, constructing `AgentRun` directly). Typical uses:
a SandboxLayer command isolating cross-attempt infrastructure content per Experiment, or an adapter switching authentication / routing per Experiment.
It is a different dimension from `flags` (the concrete
values of Experiment conditions): this is only a stable identifier of "which Experiment is running" and carries no condition content.
#### `evalId`
```ts theme={null}
readonly evalId?: string;
```
The eval ID of the current Attempt. The NiceEval Runner always fills it from the Eval identity after discovery;
third parties constructing AgentContext directly can omit it. An Adapter can use it to locate read-only host assets in the same directory as the task,
but must not use it to bypass the Sandbox's hidden-criteria isolation.
#### `evalGroup`
```ts theme={null}
readonly evalGroup?: {
readonly id: string;
readonly definitionHash: string;
};
```
The eval group of the current Attempt; omitted for ungrouped Evals.
#### `attempt`
```ts theme={null}
readonly attempt?: AttemptRef;
```
The current Attempt reference filled in by the Runner; third parties constructing the context directly can omit it.
#### `session`
```ts theme={null}
readonly session: AgentSession;
```
#### `telemetry`
```ts theme={null}
readonly telemetry?: Telemetry;
```
Present only when OTel integration is configured (the agent's `tracing` block / the config's telemetry exists):
this run's OTLP traces receiver information (endpoint + env for env-based export).
How to hand it to the CLI is declared by the agent's `tracing` block: env-based spreads ctx.telemetry.env
into send; file-based writes configuration in tracing.configure. A remote HTTP integration's send
only needs to spread headers into the request headers (a new traceparent per turn); the endpoint is fixed by startup configuration
(defineConfig(\{ telemetry: \{ port } })) and is not passed from here.
#### `progress`
```ts theme={null}
progress(update: ProgressUpdate): void;
```
Scoped feedback: reports what is happening right now (turn / tool / install progress). Short-lived state: the Human profile
updates the active line, `agent`/`ci` do not print each one, and it does not enter the final result; do not call it for every token/delta.
The runner attributes it by the lifecycle phase the callback is currently in (agent.setup / agent.run / agent.teardown);
the caller cannot impersonate another phase (see docs/feature/experiments/library.md).
#### `diagnostic`
```ts theme={null}
diagnostic(input: DiagnosticInput): void;
```
Scoped feedback: reports problems that should remain after the run ends (protocol downgrade, incomplete data, cleanup issues).
A permanent event that lands in the attempt's diagnostics and each profile's permanent output; deduplicated by `dedupeKey`.
Even with level "error" it does not change Turn.status / verdict; throw when you cannot continue.
#### `log`
```ts theme={null}
log(msg: string): void;
```
An explicit timeout breadcrumb. It also updates the Human active line, and the most recent entries are merged into the result's error information
on a timeout failure; do not use it to carry user messages, tool input, or other text meant only for short-lived display.
`ctx.session` (`AgentSession`) is the state slot for one session line: every `send` on the same session line gets the same `ctx.session`, and a new session line (the Eval's first turn / after `t.newSession()`) gets a fresh one. Accessors:
* `id?: string` / `capture(id): void` — for session continuation when the server keeps history: `id` is the session id this line has recorded (`undefined` on a new line), and `capture` records the returned id (it takes effect only if none has been recorded yet).
* `createSessionSlot(name)` — creates a typed slot at Adapter module scope. Slots are isolated by symbol identity.
* `get(slot)` / `set(slot, value)` — store and read client-side history or Adapter-private state.
* `take(slot)` — reads and deletes the HITL pause state, consumed once.
For the complete contract, see [Adapter concepts](/docs/explanation/adapter#context-agentcontext).
## Sandbox interface
`ctx.sandbox` (`Sandbox`) of a Sandbox agent is the handle to the current isolated environment. `Sandbox` extends `SandboxOperations` and `SandboxTransferOperations`; all members are listed below by declaring interface. `CommandOptions` holds the options for `runCommand` / `runShell`:
### `SandboxOperations`
#### `workdir`
```ts theme={null}
readonly workdir: string;
```
The absolute path of the project/workspace root inside the Sandbox (the default cwd for agent commands, and where the git baseline is committed). Relative paths in every method resolve against it, and omitting `cwd`/`targetDir` also lands here.
#### `runCommand`
```ts theme={null}
runCommand(cmd: string, args?: readonly string[], opts?: CommandOptions): Promise;
```
Runs a single command; `args` are passed as separate argv entries and are not interpreted by a shell (no `&&`, pipes, or glob expansion).
Prefer it when you only want to run one executable, arguments come from external input, or you are worried about injection.
#### `runShell`
```ts theme={null}
runShell(script: string, opts?: CommandOptions): Promise;
```
Runs a whole script interpreted by a shell (bash), supporting `&&`, pipes, `$()`, redirection, and so on.
Use it when you need to chain several commands or branch on conditions.
#### `runCommandOrThrow`
```ts theme={null}
runCommandOrThrow(
cmd: string,
args?: readonly string[],
opts?: CommandOptions,
): Promise;
```
Runs a single command like `runCommand`, but throws `SandboxCommandExitError` on a non-zero exit. The error message includes
a bounded, cleaned, and redacted stderr tail; it falls back to stdout when stderr is empty. The full output is kept in the exception's
`result` field. A successful result's `exitCode` is fixed to `0` in its type.
#### `runShellOrThrow`
```ts theme={null}
runShellOrThrow(script: string, opts?: CommandOptions): Promise;
```
Runs a shell script like `runShell`, but throws `SandboxCommandExitError` on a non-zero exit. The error summary,
full output, and successful result semantics are the same as `runCommandOrThrow`.
#### `readText`
```ts theme={null}
readText(path: string): Promise;
```
Reads the text content (UTF-8) of a file inside the Sandbox. Throws when the file does not exist.
#### `writeText`
```ts theme={null}
writeText(path: string, content: string): Promise;
```
Writes a UTF-8 text file inside the Sandbox; missing parent directories are created automatically.
#### `readBytes`
```ts theme={null}
readBytes(path: string): Promise;
```
Reads a file's exact bytes inside the Sandbox; the public contract is not bound to Node Buffer.
#### `writeBytes`
```ts theme={null}
writeBytes(path: string, content: Uint8Array): Promise;
```
Writes a file's exact bytes inside the Sandbox; missing parent directories are created automatically.
#### `pathExists`
```ts theme={null}
pathExists(path: string): Promise;
```
Checks whether a file or directory path exists inside the Sandbox.
### `SandboxTransferOperations`
#### `upload`
```ts theme={null}
upload(content: SandboxContent, targetPath: string): Promise;
```
Transfers registered immutable content without exposing its host path.
#### `uploadFile`
```ts theme={null}
uploadFile(source: string | URL, targetPath: string): Promise;
```
#### `uploadDirectory`
```ts theme={null}
uploadDirectory(
sourceDir: string | URL,
targetDir?: string,
options?: { readonly ignore?: readonly string[] },
): Promise;
```
#### `downloadFile`
```ts theme={null}
downloadFile(sourcePath: string, target: string | URL): Promise;
```
#### `downloadDirectory`
```ts theme={null}
downloadDirectory(
sourceDir: string,
targetDir: string | URL,
options?: { readonly ignore?: readonly string[] },
): Promise;
```
### `Sandbox`
#### `stop`
```ts theme={null}
stop(): Promise;
```
Destroys the compute resources the Sandbox occupies (container/microVM). The Sandbox cannot be used after the call; whether repeated calls are safe varies by provider, so do not rely on it.
#### `sandboxId`
```ts theme={null}
readonly sandboxId: string;
```
This Sandbox's stable identifier (each provider's native ID, such as a Docker container ID prefix); used to correlate session state of the same Sandbox across calls, and for log display.
#### `otlpHost`
```ts theme={null}
readonly otlpHost: string | null;
```
The placement capability of the OTLP receiver.
* `string`: the host receiver is reachable from inside the Sandbox through this hostname.
* `null`: the provider does not promise a connection back to the host; the runner tries to start an attempt-scope receiver inside the Sandbox.
This does not guarantee tracing succeeds; when the image lacks the runtime the receiver needs, only a supplemental diagnostic is recorded.
`defineConfig({ telemetry: { host } })` can override it explicitly when the author already provides a controlled tunnel.
#### `appendLog`
```ts theme={null}
appendLog?(line: string): Promise;
```
Optional: writes a line into the container's "main log" (which PID1 tails), so `docker logs` /
the Logs tab of the Docker UI shows the agent's turn-by-turn activity live. Implemented by the docker provider; others can omit it.
### `CommandOptions`
#### `sensitiveValues`
```ts theme={null}
readonly sensitiveValues?: readonly string[];
```
Sensitive plaintext this command is known to handle (such as API keys, tokens, HTTP header values). The Runner still hands the
original values to the provider for execution, but replaces these values exactly before any timing / commands / execution / error evidence is persisted;
this array itself is never persisted and does not enter the fingerprint. Empty strings are ignored.
This is explicit provenance, not a secret scanner: unregistered free text cannot be reliably identified; if the caller encodes or splits a value
first, also register the encoded form that will actually appear in the command/output.
#### `env`
```ts theme={null}
readonly env?: Readonly>;
```
Adds/overrides environment variables for this command (layered on the Sandbox default environment without clearing defaults). `PATH` is a Sandbox
managed variable; each provider keeps the `PATH` it computed and does not guarantee it can be overridden here; to extend PATH, use
the Sandbox factory's `pathPrepend` (see docs/feature/sandbox/library.md "PATH: managed variable and
pathPrepend").
#### `cwd`
```ts theme={null}
readonly cwd?: string;
```
This command's working directory; omitted lands on `Sandbox.workdir`. Relative paths resolve against the workdir, and absolute paths are used as is.
#### `stream`
```ts theme={null}
readonly stream?: boolean;
```
Also sends this command's output into the Sandbox's "native log stream" (so `docker logs` / the Logs tab of the Docker UI
shows it live). Turn it on for agent commands (codex exec / bub run / claude) to see the agent's raw output in the container
log. Each provider implements it (docker: tee into the file PID1 tails;
unsupported providers ignore it); how the log surfaces is the provider's business, and the adapter only declares intent.
#### `onStdout`
```ts theme={null}
readonly onStdout?: (chunk: string) => void | Promise;
```
Called once for each stdout chunk of the command. The callback is only for short-lived feedback while running; the full stdout still appears verbatim
in the returned `CommandResult`. When the provider does not support real streaming, it calls it at least once with the full
stdout after the command ends; it must not drop it silently.
#### `onStderr`
```ts theme={null}
readonly onStderr?: (chunk: string) => void | Promise;
```
The stderr counterpart of `onStdout`; the full stderr is still kept in `CommandResult`.
#### `user`
```ts theme={null}
readonly user?: string;
```
Overrides the execution identity of this command; omitted = the Sandbox default identity (the identity the environment declares itself: the Docker image `USER`,
the Compose service `user:`, the E2B template default user; see
docs/feature/sandbox/library.md "Execution identity").
The semantics are consistent across providers, and each provider maps them to its own native mechanism (docker: `exec --user`; E2B:
`{ user }`; Vercel: accepts only `"root"`, mapped to `{ sudo: true }`, other values error).
Providers that run as root throughout treat it as a no-op; providers that cannot switch identity at all may not support it (and throw), but **the semantics of omission and
explicit values stay consistent** and do not vary by provider.
#### `timeoutMs`
```ts theme={null}
readonly timeoutMs?: number;
```
This command's own limit (milliseconds). **Omitting it is the norm**: when omitted, the limit = the remaining attempt deadline
(see docs/feature/sandbox/architecture.md "Deadline ownership"); the provider layer has no independent default.
Passing a shorter value explicitly is an intentional declaration and takes effect as usual; on hitting it, the attribution is recorded as "explicit command timeout".
#### `signal`
```ts theme={null}
readonly signal?: AbortSignal;
```
Cancels this managed command tree. The provider must confirm the command tree has terminated before the Promise settles; when it cannot terminate
precisely, it should retire the whole Sandbox rather than only closing the transport and leaving processes in the background.
## ctx and t
`ctx` is the run context the adapter side sees. `t` is the test context the Eval author sees. Both use the same run data, but have different responsibilities.
# defineConfig: Project Defaults
Source: https://niceeval.com/docs/reference/define-config
defineConfig reference: Judge Runtime, Reporters, concurrency, timeout, and Sandbox defaults.
`defineConfig` is the default export from `niceeval.config.ts` at the project root and holds only project-level defaults. Agent, model, Flags, Attempts, and Experiment budgets belong in `defineExperiment` files under `experiments/`.
```ts theme={null}
import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";
export default defineConfig({
judgeRuntime: OpenAIProvider({ model: "gpt-5.4-mini", maxOutputTokens: 512 }),
maxConcurrency: 4,
timeoutMs: 300_000,
});
```
## Config fields
#### `name`
```ts theme={null}
name?: LocalizedText;
```
The project name, shown in the hero (``) at the top of `niceeval view`; it falls back to a generic title when omitted.
It can be a string, or text in multiple locales (such as `{ en: "...", "zh-CN": "..." }`) that switches with the view language.
#### `workspace`
```ts theme={null}
workspace?: string;
```
The workspace root uploaded into a Sandbox; it defaults to the project root. An Eval's Sandbox view starts from here.
#### `judgeRuntime`
```ts theme={null}
judgeRuntime?: JudgeProvider;
```
The project-level default Judge Provider; an Eval or Experiment can replace the model or the whole Provider.
#### `reporters`
```ts theme={null}
reporters?: Reporter[];
```
The project-level default Reporter list (for example, persisting or uploading results); `EvalDef.reporters` merges with it.
#### `maxConcurrency`
```ts theme={null}
maxConcurrency?: number;
```
The project-level default concurrency limit; the CLI flag or Experiment setting with the same name has higher priority (there is no environment-variable layer).
#### `maxBuildConcurrency`
```ts theme={null}
maxBuildConcurrency?: number;
```
Run-level concurrency for preparing Sandbox images; independent of Attempt concurrency, with a safe default of 2 when omitted.
#### `maxSetupPrefixConcurrency`
```ts theme={null}
maxSetupPrefixConcurrency?: number;
```
Run-level concurrency for preparing the PreparedArtifact prefix DAG; independent of build and Attempt concurrency.
#### `sandboxCache`
```ts theme={null}
sandboxCache?: SandboxCacheConfig;
```
The project-level SetupPrefix cache default; an Experiment or an explicit CLI flag can override it.
#### `timeoutMs`
```ts theme={null}
timeoutMs?: number;
```
The project-level default timeout for one Attempt (milliseconds); the CLI flag, Experiment, or `EvalDef` setting with the same name has higher priority.
#### `telemetry`
```ts theme={null}
telemetry?: { host?: string; port?: number };
```
OTLP receiver configuration, the only entry point inside a niceeval project (it does not read `NICEEVAL_OTLP_*` environment variables).
`port` pins the receiving port (fixed-port mode: a long-running service points `OTEL_EXPORTER_OTLP_ENDPOINT` once at
`http://localhost:/v1/traces`, and never changes it no matter how many Eval runs follow). When omitted, each run dynamically assigns
a temporary port (handed to the Adapter through `ctx.telemetry`). The tradeoff: with a fixed port, only one niceeval process can run on the machine at a time,
and an error occurs if another process holds that port; pick a free port and write it back here.
`host` is the receiver hostname reported to the Adapter (not the listening address, which is always `0.0.0.0`); it defaults to
`"127.0.0.1"`. Override it here only when you already provide a controlled tunnel or reachable route; a Docker Sandbox
puts the receiver inside the Sandbox by default and does not rely on an implicit host gateway.
#### `pricing`
```ts theme={null}
pricing?: globalThis.Record;
```
User overrides or additions on top of the built-in price table (`o11y/prices.json`), looked up by model (see Observability
· Usage and Cost). A key can be an exact model name or a `provider/*` wildcard (to override self-hosted or gateway discounts in bulk per Provider);
exact keys take priority over wildcards. The Runner uses it to compute the budget estimate for `maxCost`; the Adapter physical-call
collector seals a matching explicit fixed profile as a call-bound estimate proof only when reported cost is absent.
Inspection reads only sealed proofs; it does not take mutable config as an operation input, and does not backfill historical calls by endpoint.
> Cost boundary: this `pricing` only provides the Runner's Config/runtime price-table input. Whether or not `Usage.costUSD` already exists, the Runner independently produces `estimatedCostUSD` from the `model`, the reported token usage, and this table. The Experiment budget and `t.maxCost()` use that estimate; `Usage.costUSD` does not drive it. Reading results uses fixed Inspection operations, which do not turn runtime price configuration into another input.
[NiceEval](https://niceeval.com/) keeps environment preparation in ordinary code: files an Eval needs go in `test(t)`, and an Agent's own preparation goes in the Adapter's setup. `Config` deliberately has no `sandbox` field and does not auto-detect a Provider. Declare a template-bearing `SandboxLayer` on the Eval or Experiment so link planning settles its owner and pairing result before creating any resources.
# defineEval: Declare, Configure, and Run NiceEval Evals
Source: https://niceeval.com/docs/reference/define-eval
defineEval reference: options, the test context t, the Turn return value, Sandbox helpers, and array and keyed-record test set exports.
`defineEval` is the main entry point for writing Evals. Call it once per Eval file, pass a description and `test(t)`, and default-export the result.
For how to organize business criteria, evidence, and points, see [NiceEval Taste](/docs/tutorials/niceeval-taste).
```ts theme={null}
import { defineEval } from "niceeval";
export default defineEval({
description: "Brooklyn weather query",
async test(t) {
const turn = await t.send("What's the weather like in Brooklyn today?");
turn.succeeded().label("Turn completed");
},
});
```
Do not provide `id` or `name`. NiceEval generates the Eval ID from the file path.
## `defineEval` options
#### `description`
```ts theme={null}
description?: string;
```
A one-sentence description shown in `niceeval list` and the view; purely informational, it does not affect scheduling or scoring.
#### `tags`
```ts theme={null}
tags?: string[];
```
Tags for CLI `--tag` filtering and view categorization; a selection dimension independent of id-prefix filtering.
#### `sandbox`
```ts theme={null}
sandbox?: SandboxLayer;
```
The Sandbox declaration layer this Eval contributes. Omitting it is equivalent to an empty command-only layer and provides no implicit template.
In each actual Eval × Experiment pair, exactly one side must provide a template-bearing layer.
#### `plugins`
```ts theme={null}
plugins?: readonly PluginInstance<"eval">[];
```
Explicit, immutable Eval Plugin occurrences; there is no directory inheritance.
#### `judge`
```ts theme={null}
judge?: JudgeSelection;
```
A model string replaces only the model of the project's Judge Provider; a Provider instance replaces it entirely. The Experiment setting takes priority.
#### `reporters`
```ts theme={null}
reporters?: Reporter[];
```
Overrides / appends to the project-level Config.reporters, only for this Eval.
#### `timeoutMs`
```ts theme={null}
timeoutMs?: number;
```
Overrides the project-level / CLI per-attempt timeout (milliseconds), only for this Eval.
#### `metadata`
```ts theme={null}
metadata?: globalThis.Record;
```
Arbitrary extra metadata, saved as Attempt Provenance; it does not take part in scheduling or scoring and is for custom reporters to consume.
#### `diff`
```ts theme={null}
diff?: { include?: string[]; ignore?: string[] };
```
Adjusts the attribution exclusion list for the agent diff (Sandbox only; see docs/feature/eval/README.md): both arrays are
gitignore-style globs (relative to the workdir). By default .git/node\_modules/build output/package-manager caches are excluded;
`ignore` adds exclusions on top of the default list; `include` has the highest priority and explicitly adds matching paths back.
The composition rule is fixed as "default ∪ ignore, then punched through by include", and the list is frozen at the ledger anchor.
#### `test`
```ts theme={null}
test(t: TestContext): EvalTestReturn;
```
## Test context: `t`
`t` (`TestContext`) is the context an Eval author receives in `test(t)`. Each call to a check method registers one assertion.
* `t.check(subject, match)` checks a value you pass explicitly against a match condition.
* `t.check(contextualMatch)` takes no value; NiceEval hands the current read-only ctx to the match condition to read.
* `t.judge(material, match)` gives material to a Judge for scoring, through the same registration path as `t.check`.
`judge` on the root, a Session, and a Turn all require material passed explicitly. NiceEval ships four ready-made judges, which also build a match condition and hand it to `check`. `closeQA(selector, question)` first selects a set of material with the selector, then has the Judge answer an acceptance question. Agent scenarios also have `closeQA(question)`, which reads the current scope's full history by default; you can also pass an EventMatch or ToolMatch to look at only part of the facts.
How an assertion affects the result by default depends on the Eval type:
* In a pass/fail Eval, Boolean assertions decide the Attempt's Verdict by default; in a scored Eval, Boolean assertions only record by default and do not affect the result.
* Scoring assertions (measurements) do not fail the Eval by default. In both Eval types, after adding `.gate(minimum)`, a score below the minimum counts as a failure. In a scored Eval, the same assertion can have both `.gate()` and `.score()`.
* `.orStop()` stops later steps when the assertion fails. A Boolean assertion uses its own result; a measurement writes `.orStop(minimum)`, or a parameterless `.orStop()` after `.gate(minimum)` to reuse the gate's minimum.
Compute ordinary formulas directly in TypeScript, then register them with `t.score(ratio, { weight: 50 }).label("Completion time")`. `ratio` is a number between 0 and 1, and `weight` is a non-negative finite number. The result keeps the ratio, the corresponding points, and the actual score. When the numeric material it depends on is unavailable, the item's score is recorded as unknown.
The `ScoreTestContext` of `defineScoreEval` also provides `t.score(n)`, which registers a score item directly; existing assertions count toward the score with `.score(n)`. In both places `n` must be a finite number not less than zero, and 0 is also an explicit score. For the complete rules of `calledTool` and `notCalledTool`, see [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md). All members are:
#### `evaluationKind`
```ts theme={null}
readonly evaluationKind: Kind;
```
#### `send`
```ts theme={null}
send(input: string | { readonly text: string; readonly files?: readonly InputFile[] }): Promise>;
```
#### `sendFile`
```ts theme={null}
sendFile(path: string, text?: string): Promise>;
```
#### `requireInputRequest`
```ts theme={null}
requireInputRequest(filter?: InputRequestFilter): InputRequest;
```
#### `respond`
```ts theme={null}
respond(...responses: readonly (string | InputAnswer)[]): Promise>;
```
#### `respondAll`
```ts theme={null}
respondAll(optionId: string): Promise>;
```
#### `reply`
```ts theme={null}
readonly reply: string;
```
#### `sessionId`
```ts theme={null}
readonly sessionId: string | undefined;
```
#### `events`
```ts theme={null}
readonly events: readonly StreamEvent[];
```
#### `eventOccurrences`
```ts theme={null}
readonly eventOccurrences: ManagedEventOccurrences<"attempt">;
```
#### `newSession`
```ts theme={null}
newSession(): SessionHandle;
```
#### `signal`
```ts theme={null}
readonly signal: AbortSignal;
```
#### `model`
```ts theme={null}
readonly model?: string;
```
#### `reasoningEffort`
```ts theme={null}
readonly reasoningEffort?: string;
```
#### `flags`
```ts theme={null}
readonly flags: ExperimentFlags;
```
#### `progress`
```ts theme={null}
progress(update: import("../types.ts").ProgressUpdate): void;
```
#### `diagnostic`
```ts theme={null}
diagnostic(input: import("../types.ts").DiagnosticInput): void;
```
#### `log`
```ts theme={null}
log(message: string): void;
```
#### `skip`
```ts theme={null}
skip(reason: string): never;
```
#### `group`
```ts theme={null}
group(
title: string,
body: () => Value | PromiseLike,
): Promise>;
```
#### `check`
```ts theme={null}
check: import("../assertions/api.ts").AssertionCheck>;
```
#### `judge`
```ts theme={null}
judge: JudgeFunction;
```
#### `toolCalls`
```ts theme={null}
readonly toolCalls: ManagedToolCalls<"attempt">;
```
#### `sandbox`
```ts theme={null}
readonly sandbox: Sandbox;
```
#### `o11y`
```ts theme={null}
readonly o11y: import("../o11y/types.ts").O11ySummary;
```
#### `usage`
```ts theme={null}
readonly usage: EvalUsage;
```
#### `succeeded`
```ts theme={null}
succeeded(): BooleanAssertionHandle;
```
#### `usedNoTools`
```ts theme={null}
usedNoTools(): BooleanAssertionHandle;
```
#### `maxToolCalls`
```ts theme={null}
maxToolCalls(max: number): BooleanAssertionHandle;
```
#### `noFailedActions`
```ts theme={null}
noFailedActions(): BooleanAssertionHandle;
```
#### `event`
```ts theme={null}
event(match: EventMatch | EventOccurrenceMatch): BooleanAssertionHandle;
```
#### `notEvent`
```ts theme={null}
notEvent(match: EventMatch): BooleanAssertionHandle;
```
#### `elapsedMs`
```ts theme={null}
readonly elapsedMs: import("../assertions/match.ts").NumericMaterial;
```
#### `maxTokens`
```ts theme={null}
maxTokens(max: number): BooleanAssertionHandle;
```
#### `maxCost`
```ts theme={null}
maxCost(usd: number | string): BooleanAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(selector: ToolMatch | EventMatch, question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(selector: MaterialMatch, question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
## Judge scoring
First declare a rubric with `defineJudge`, then pass it directly to `t.check` or `t.judge`:
```ts theme={null}
const factualSummary = defineJudge({
name: "factual-summary",
rubric: "Judge, against source, whether summary faithfully keeps the facts without inventing any.",
});
const turn = await t.send("Summarize the requirements.");
t.judge({ source: expected, summary: turn.message }, factualSummary)
.gate(0.8)
.label("Factually consistent");
```
Judge material can be a size-bounded JSON value or an image produced by `judgeImage()`; a snapshot is saved at registration.
Which model the Judge uses is decided in order: first the project config's `judgeRuntime`, then the Eval's `judge`, and finally the Experiment's `judgeRuntime`. When a later layer gives a complete Provider, it replaces the service and execution configuration entirely; when it gives only a model string, it replaces just the model of the selected Provider. An Eval's `judge` only decides which model or Provider this Eval uses; it does not restrict which match conditions can be used.
Both pass/fail and scored Evals can call `.gate(minimum)`. Scored Evals can also call `.score(n)`, and the item keeps its score when the gate fails.
### Image material
Import `judgeImage` from `niceeval/judge`; it copies the original image data immediately. The limits are:
* Only PNG and JPEG formats are supported.
* Each image is at most 4 MiB and 16,777,216 pixels.
* Each assertion and each model request can reference images at most 4 times, totaling no more than 8 MiB. Referencing the same image again also counts.
* Each Attempt saves at most 32 MiB of images.
NiceEval checks only the file signature, header, and dimensions; it does not fully decode or transcode images.
`OpenAIProvider`, `VercelProvider`, and `OpenRouterProvider` do not send images by default. Set `supportsImages: true` explicitly and choose a model that supports both images and tool calls. `TypesafeProvider` does not support images. When images are unsupported, the result is `judge-capability-unavailable`; images are never swapped for a text description before scoring.
#### `judgeImage`
```ts theme={null}
export function judgeImage(input: JudgeImageInput): JudgeImage { ... }
```
Capture PNG or JPEG bytes for an existing Judge call without sending a request.
### `JudgeImageInput`
#### `body`
```ts theme={null}
readonly body: Uint8Array;
```
Original encoded image bytes; copied synchronously, at most 4 MiB.
#### `mediaType`
```ts theme={null}
readonly mediaType: "image/png" | "image/jpeg";
```
Must match the encoded image. Paths, URLs and SVG are not accepted.
### Ready-made judges and custom Matches
`defineJudge` and the ready-made judges all return a `ScoreMatch`, which you can pass directly to `t.check`. `t.judge` is just shorthand for the same path, so registration, snapshot, budget, scoring, audit, and result storage each happen only once.
| Judge | Input | Score meaning |
| - | - | - |
| `factuality()` | `{ input, output, expected }` | 1 when consistent, 0.5 when incomplete, 0 when contradictory |
| `faithfulness()` | `{ input, output, context }` | The share of facts in the output supported by the context |
| `instructionFollowing()` | `{ instructions, output }` | The share of explicit requirements met |
| `pairwisePreference()` | `{ instructions, output, reference }` | 1 when the candidate is better, 0.5 for a tie, 0 when the reference is better |
| `t.closeQA(selector, question)` | A MaterialMatch that reads material from ctx and judges item by item | 1 when acceptance is met, 0 when not, unavailable when evidence is insufficient |
For example, using an `instructionFollowing()` instance directly:
```ts theme={null}
t.check(
{ instructions: ["State the meeting place", "Do not invent a specific time"], output: post.content },
followsInstructions,
).gate(0.7).score(25);
```
`faithfulness` first extracts facts from the output, judges them one by one, and finally computes the ratio in code. When extraction is incomplete there is no valid score. TypeSafe does not support `extract`, so on `TypesafeProvider` the result is `unavailable`; it is never switched to estimating one overall score.
`pairwisePreference` compares once in a fixed order and suits a comparison between one candidate and one reference.
`closeQA` gives all selected material to the Judge at once in original order, and the Judge answers the acceptance question based only on that material.
Agent scenarios can use `turn.closeQA(question)`, `session.closeQA(question)`, and `t.closeQA(question)`. Passing an EventMatch looks only at the matching events; passing a ToolMatch looks at the complete information of the matching tool calls.
The state of the material affects the result: when the selected material is complete but empty, the score is 0 and the model is not called; when the material is incomplete, undecidable, or over the limit, the result is `unavailable`, and it is never truncated before scoring.
When you need to combine several model steps, write your own scoring logic with `score(value, context)` in `defineScoreMatch`. `context.llm` provides `score`, `classify`, `extract`, and `batchClassify`, all returning an Effect.
They share the call budget, timeout settings, and audit records with the ready-made judges.
* Static configuration validation sends no network requests. No request is sent when the Provider, model, or key is missing either.
* Chat-style services use forced-function requests, and TypeSafe uses the `/systemone` endpoint; both validate responses strictly.
* Network transport errors or timeouts make the result unavailable; HTTP 400 or protocol incompatibility makes it errored.
For the complete rules, see [Custom Matches](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/custom-assertions.md).
## The `Turn` return type
`t.send(...)` returns a `TurnHandle`. It contains common fields derived from the event stream, plus a full set of assertions scoped to this turn only.
`calledTool` and `notCalledTool` are also assertions on the Turn; for their signatures and matching rules, see [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md).
#### `input`
```ts theme={null}
readonly input: string;
```
#### `events`
```ts theme={null}
readonly events: readonly StreamEvent[];
```
#### `toolCalls`
```ts theme={null}
readonly toolCalls: ManagedToolCalls<"turn">;
```
#### `eventOccurrences`
```ts theme={null}
readonly eventOccurrences: ManagedEventOccurrences<"turn">;
```
#### `status`
```ts theme={null}
readonly status: "completed" | "failed" | "waiting";
```
#### `message`
```ts theme={null}
readonly message: string;
```
#### `data`
```ts theme={null}
readonly data?: JsonValue;
```
#### `usage`
```ts theme={null}
readonly usage: EvalUsage;
```
#### `check`
```ts theme={null}
check: import("../assertions/api.ts").AssertionCheck>;
```
#### `judge`
```ts theme={null}
judge: JudgeFunction;
```
#### `succeeded`
```ts theme={null}
succeeded(): BooleanAssertionHandle;
```
#### `toolOrder`
```ts theme={null}
toolOrder(matches: readonly [ToolMatch, ToolMatch, ...ToolMatch[]]): BooleanAssertionHandle;
```
#### `usedNoTools`
```ts theme={null}
usedNoTools(): BooleanAssertionHandle;
```
#### `maxToolCalls`
```ts theme={null}
maxToolCalls(max: number): BooleanAssertionHandle;
```
#### `noFailedActions`
```ts theme={null}
noFailedActions(): BooleanAssertionHandle;
```
#### `event`
```ts theme={null}
event(match: EventMatch | EventOccurrenceMatch): BooleanAssertionHandle;
```
#### `notEvent`
```ts theme={null}
notEvent(match: EventMatch): BooleanAssertionHandle;
```
#### `eventOrder`
```ts theme={null}
eventOrder(matches: readonly [EventMatch, EventMatch, ...EventMatch[]]): BooleanAssertionHandle;
```
#### `elapsedMs`
```ts theme={null}
readonly elapsedMs: import("../assertions/match.ts").NumericMaterial;
```
#### `maxTokens`
```ts theme={null}
maxTokens(max: number): BooleanAssertionHandle;
```
#### `maxCost`
```ts theme={null}
maxCost(usd: number | string): BooleanAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(selector: ToolMatch | EventMatch, question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
#### `closeQA`
```ts theme={null}
closeQA(selector: MaterialMatch, question: string, options?: JudgePresetOptions): MeasurementAssertionHandle;
```
## Test set exports
```ts theme={null}
export default rows.map((row) =>
defineEval({
description: row.task,
async test(t) {
await t.send(row.prompt);
},
}),
);
```
When you export an array, each item gets a stable ID: `file/0000`, `file/0001`, and so on.
When you already have stable business keys, you can also default-export a `Record`. For example, the key `15193` in `swelancer.eval.ts` produces `swelancer/15193`. A key must be a non-empty single path segment; it cannot be `.` or `..`, and cannot contain `/`, `\\`, or control characters. Evals are ordered by key in lexicographic order.
### Reading material from ctx, and official usage accounting
`defineMaterialMatch({ name, read, match, capture? })` defines a set of material read from ctx; import it from `niceeval` or `niceeval/expect`. The Adapter's `create` provides the application's own ctx. When registering in `check` or `closeQA`, NiceEval passes the current read-only ctx to `read`.
`read` returns `{ state: "complete", items: [{ id, value }] }`, or a partial or unavailable state with a `reason`. Each item's `id` must be unique within the set, at most 128 UTF-8 bytes.
`capture` controls how much material is saved: by default at most 256 items and 48 KiB, adjustable up to 16384 items and 4 MiB.
`check(materialMatch)` returns a Boolean assertion that by default passes when at least one item satisfies the condition. `closeQA(materialMatch, question, options?)` returns a Judge-scored assertion: `question` cannot be empty and is at most 8 KiB; `options` is of type `JudgePresetOptions`. All selected material is given to the Judge at once in original order. When the material is complete but empty, the score is 0 and the model is not called.
In Agent scenarios, `read` reads the tool calls (`toolCalls`) or events (`eventOccurrences`) NiceEval has organized from `AgentMatchContext`, then matches them with a ToolMatch or EventMatch respectively.
`defineContextMatch({ name, read, match, capture? })` reads a single small value from ctx. `read` returns `{ state: "available", value }` or `{ state: "unavailable", reason }`. Paired with a BooleanMatch it yields a Boolean assertion; paired with a ScoreMatch it yields a scoring assertion. When the value is unavailable, the scoring logic is not called.
`t.usage` reads the usage officially recorded by NiceEval, fixed at the moment it is read. Failed and retried calls, cache buckets, and the basis for cost calculation are all kept.
* For an ordinary Adapter, the usage basis is every recorded call (`recorded-calls`).
* For an Agent, the usage basis is the numbers reported on each send (`reported-sends`), viewable per Turn, Session, or Attempt.
* Fields such as `usage.totalTokens` are NumericMaterial. Unknown values stay unknown and are never filled in as 0.
`t.maxTokens(max)` limits total tokens. `t.maxCost(usd)` limits by USD cost: it prefers the actual charged amount, and a charge of 0 also counts; only when there is no actual charge does it use an estimate computed from prices you configured explicitly.
`t.elapsedMs` is the milliseconds elapsed from the start of the Attempt to the moment it is read, and can be passed to `t.check(..., atMost(limit))`. The business moment of "first completion" still has to be judged with the business's own match condition.
# defineExperiment: Experiment Configuration
Source: https://niceeval.com/docs/reference/define-experiment
Reference for defineExperiment fields, types, and default behavior.
An Experiment decides what you run against, including the Agent under test, the model, and the run conditions. It is kept separate from Evals, so the same set of Evals can compare different configurations. For the complete approach, see [Write an Experiment](/docs/tutorials/write-experiment).
```ts theme={null}
// experiments/local.ts
import { defineExperiment } from "niceeval";
import { webAgent } from "../agents/web-agent.ts";
export default defineExperiment({
agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
});
```
## Experiment fields
#### `models`
```ts theme={null}
models?: Readonly>;
```
Selects a model for each named model slot of a custom Adapter; cannot be combined with the single-model shorthand.
#### `description`
```ts theme={null}
description?: string;
```
A one-sentence description shown in the view and CLI; purely informational, it does not affect scheduling or scoring.
#### `agent`
```ts theme={null}
agent?: Agent;
```
Shorthand for a conversational Adapter; exactly one of `agent` and `adapter` must be provided.
#### `adapter`
```ts theme={null}
adapter?: Adapter;
```
The Adapter implementation this Experiment uses.
#### `model`
```ts theme={null}
model?: string;
```
A single model (decided by the Experiment when the agent leaves it empty); omitted = use the agent's native default. To compare models, write several Experiment files instead of an array.
#### `reasoningEffort`
```ts theme={null}
reasoningEffort?: string;
```
The model's reasoning effort (such as "low"/"medium"/"high"; valid values depend on the specific model/adapter); omitted = use the agent's native default. Passed to the adapter and eval through ctx.reasoningEffort.
#### `judgeRuntime`
```ts theme={null}
judgeRuntime?: JudgeSelection;
```
This Experiment's Judge execution configuration. It overrides only the model / endpoint / credential selector / call budget;
the rubric is owned by the Match, and the material and consumption threshold come from the Assertion. Each field resolves in the order
Experiment → Eval → Config → built-in default.
#### `flags`
```ts theme={null}
flags?: globalThis.Record;
```
Experiment conditions (the feature flags in an A/B), declared by the Experiment file; must be a flat scalar object
(strings, finite numbers, or booleans only, validated when defineExperiment parses it). Passed to the adapter through ctx.flags,
exposed to the eval as t.flags, and copied verbatim into the result snapshot's ExperimentRunInfo.flags.
#### `labels`
```ts theme={null}
labels?: globalThis.Record;
```
Report grouping labels: the Experiment's coordinates on each comparison axis (such as `{ line: "codex", memory: "mempal" }`).
Values are string | number (validated at parse time). The line between labels and `flags` is "does it change what happens in the attempt":
labels are not passed to ctx / t (the agent and eval cannot see them) and do not take part in the comparability configuration (changing them does not invalidate existing results);
they are only projected verbatim into the snapshot's `ExperimentRunInfo.labels` for report dimensions (`label()` / `numericLabel()`)
to group by. The default report recognizes the `line` key: when any Experiment in the group declares it, scatter points are grouped by line and connected.
See docs/feature/experiments/library.md "labels".
#### `attempts`
```ts theme={null}
attempts?: number;
```
How many times to repeat the same eval (each result counts as one attempt); defaults to 1 when omitted or when CLI `--attempts` overrides it.
#### `earlyExit`
```ts theme={null}
earlyExit?: boolean;
```
Within a repetition (attempts > 1), whether to skip the remaining repeats after one attempt passes; defaults to false when omitted (`attempts` run in full to measure the complete pass rate).
Turn it on explicitly to save cost when you only want to know whether it can pass.
#### `evals`
```ts theme={null}
evals?: "*" | readonly string[] | ((e: EvalDescriptor) => boolean);
```
Which evals this Experiment covers: `"*"` for all, a string array matched by id prefix, or a custom predicate (which receives each discovered and fanned-out
read-only `EvalDescriptor`, with no path / execution fields exposed); omitted is equivalent to `"*"`. The predicate is evaluated once per
candidate eval of this invocation, and the result becomes the in-memory `selectedEvalIds` plan, not a filter called repeatedly at runtime
(see docs/feature/eval/library.md "EvalDescriptor" and docs/feature/experiments/library.md
"evals: iterate discovery results, choose your own").
#### `timeoutMs`
```ts theme={null}
timeoutMs?: number;
```
Overrides the project-level / CLI per-attempt timeout (milliseconds), only for this Experiment.
#### `sandbox`
```ts theme={null}
sandbox?: SandboxLayer;
```
The Sandbox declaration layer this Experiment contributes. It is linked pairwise with the same field of each selected Eval;
in each pair, exactly one side provides a template-bearing layer.
#### `sandboxCache`
```ts theme={null}
sandboxCache?: SandboxCacheConfig;
```
The Host's execution cache policy; unlike the Sandbox declaration layer, this field does not take part in identity computation.
#### `plugins`
```ts theme={null}
plugins?: readonly PluginInstance<"experiment">[];
```
The list of Plugin instances this Experiment declares explicitly, normalized by `defineExperiment()`.
#### `sandboxReuse`
```ts theme={null}
sandboxReuse?: boolean;
```
Reuse Sandboxes within the same Run; such runs are isolated from history carry in both directions.
#### `sharedState`
```ts theme={null}
sharedState?: SharedStateConfig;
```
Declares shared external state this Experiment needs exclusively. Invocations with the same key in the same project
Coordination domain are mutually exclusive from before Experiment/Sandbox setup until Sandbox teardown,
provider finalizers, and Experiment teardown have all finished; waiters do not create a Sandbox first,
and continue their own plan after acquiring the lease. It is not a state store, a transaction, or a cross-machine lock.
#### `budget`
```ts theme={null}
budget?: number;
```
This Experiment's estimated spend limit (USD). The scheduler accumulates the `estimatedCostUSD` of finished attempts; that value is
computed from the model, token usage, and the runtime/config pricing table, independent of the
observed `usage.costUSD` reported by the Provider / Adapter, which does not drive the budget. Once the estimate reaches the limit, the Experiment's remaining
attempts that have not started are skipped and `run:budgetExceeded` is reported once (attempts already in flight still finish).
#### `maxConcurrency`
```ts theme={null}
maxConcurrency?: number;
```
This Experiment's own concurrency limit within this Invocation: the scheduler throttles only this Experiment's attempts; other Experiments in the batch are unaffected
and still run under the global concurrency (CLI / env / config / Sandbox default). Use it to serialize Experiments with shared state
(such as memory accumulated across evals: `maxConcurrency: 1` guarantees attempts run one by one in eval order),
or to slow down an Experiment that hits provider limits. A slot shares the attempt's lifetime: it is held from before Sandbox creation until
teardown and Sandbox destruction finish, and any wait along the way (including turn retry backoff) keeps holding it.
`maxConcurrency: 1` is therefore a strict critical section that the next attempt of the same Experiment cannot enter early.
It does not span Invocations; for shared external state across Invocations, declare `sharedState`.
#### `classifyFailure`
```ts theme={null}
classifyFailure?: AttemptFailureClassifier;
```
This Experiment's failure classifier: it recognizes your own shared-infrastructure causes of death that surface in third-party error shapes (such as connections refused by your own tunnel host),
and returns `undefined` for "unknown, hand off to the rest of the chain". Failures in any per-attempt phase of this Experiment
are all routed to it; on the send failure chain it runs before the adapter's `classifySendFailure`, because filtering by your own coordinates
is more specific than the protocol's generic shape, so the scope axis wins when both claim the failure. The classifier must be fast, pure, and must not throw (a throw falls back to
`undefined` and is swallowed); it declares only the decision axis and the `reason` word, and retry and gating policy belong to the executor.
See docs/feature/error-classification/library.md "Experiment / eval authors: declaring the blast radius of a cause of death".
#### `setup`
```ts theme={null}
setup?: ExperimentHook;
```
The setup side of the Experiment-level lifecycle hook pair: at most once per run, on the host side, for host resources that are "one per Experiment, shared by all attempts"
(tunnels, mock servers, license leases). Triggered by the first attempt of this Experiment that passes dispatch admission
(memoized: concurrent attempts wait for the same result; not run when all results are carried in).
setup returns no value; write products to module-level variables, and `teardown` and agent / sandbox hooks in the same file read them from the closure;
the runner does not mediate values. If setup throws, every Attempt of this Experiment gets an `errored` Verdict
(code `"experiment-setup-failed"`, phase `"experiment.setup"`); other Experiments in the batch are unaffected.
The function body does not enter the fingerprint; after changing hook logic, use `--rerun all` to explicitly rerun everything.
See docs/feature/experiments/architecture.md "Experiment-level lifecycle".
#### `teardown`
```ts theme={null}
teardown?: ExperimentHook;
```
The teardown side of the Experiment-level lifecycle hook pair: runs after all of this Experiment's attempts finish (also when the run is interrupted),
if and only if execution reached the setup point. A setup throw does not exempt it (a half-initialized scene also needs cleanup; teardown should guard closure variables
that may be unassigned); having no setup declared does not prevent it from firing; it is skipped when no attempt is dispatched.
A throw or exceeding the 30s cleanup limit records only a run-level diagnostic (`experiment-teardown-failed`) and does not change the Verdict.
# Standard Event Stream Reference
Source: https://niceeval.com/docs/reference/events
The ten StreamEvent variants: each one's fields, when it is emitted, and which assertions read it. Producing this stream is an Adapter's main job.
An Adapter's `send` returns a `Turn`, and its `events: StreamEvent[]` is this turn's standard event stream; event assertions read their facts from it. `turn.calledTool(...)`, `turn.toolOrder(...)`, `turn.event(...)`, and `turn.succeeded()` all judge from this turn's events and the tool calls within them, and register an assertion directly when called. Translate what the agent did in this turn truthfully into the stream, and every event assertion works.
## Turn: the return value of send
```ts theme={null}
interface Turn {
events: StreamEvent[]; // this turn's events, in their actual order
data?: JsonValue; // structured output, which t.check(...) can compare
status: "completed" | "failed" | "waiting"; // waiting = paused for human input (HITL)
usage?: Usage; // → maxTokens / maxCost / cost reports; full fields below
}
```
`data` holds this turn's structured output. Fill it only when the application's answer itself is a structured object, such as extraction, classification, or form filling, so an Eval author can compare it with `t.check(turn.data, ...)`. An application that returns only text leaves `data` unset. Do not copy the raw response body or `message` text into it, and do not serialize structured output into `events`. Fill `usage` when you have it and leave it empty when you do not; do not estimate a number to fill it.
The full fields of `usage` (the `Usage` type):
#### `inputTokens`
```ts theme={null}
inputTokens?: number;
```
Input tokens that missed the cache and are billed at full price; mutually exclusive with the two cache buckets.
#### `outputTokens`
```ts theme={null}
outputTokens?: number;
```
The number of output (completion) tokens.
#### `cacheReadTokens`
```ts theme={null}
cacheReadTokens?: number;
```
Input tokens served from a prompt-cache hit; a separate billing bucket, not included in inputTokens (omitted means the agent does not report it).
#### `cacheCreationTokens`
```ts theme={null}
cacheCreationTokens?: number;
```
Input tokens written to the prompt cache; a separate billing bucket, not included in inputTokens (omitted means the agent does not report it).
#### `reasoningTokens`
```ts theme={null}
reasoningTokens?: number;
```
The number of reasoning (thinking) tokens, a detail already included in outputTokens and listed separately for display; present only when the protocol actually provides it.
#### `requests`
```ts theme={null}
requests?: number;
```
The number of model requests that actually occurred. Omit it when the protocol does not provide a request count; never write 1 to fill it.
#### `costUSD`
```ts theme={null}
costUSD?: number;
```
The actual USD cost measured by the gateway/adapter (it can only be brought back explicitly through `Turn.usage.costUSD`, and is never derived from
token usage or OTel spans). It and the top-level `estimatedCostUSD` (a price-table estimate) are two
independent facts under a one-way field contract: this field stores only the observed value; `estimatedCostUSD` always equals
the estimate from `estimateCost(model, usage, pricing)`, computed independently even when an observed value exists.
Neither overrides or backs up the other; an observed value never replaces or triggers the estimate.
> How cost is calculated: `Usage.costUSD` records only the USD cost actually returned by a Provider or Adapter; it is never derived from token counts, a model catalog, or a local price table. Whether or not that value exists, the Runner separately computes an estimated cost, `estimatedCostUSD`, from the `model`, the reported token usage, and the configured price table.
>
> The Experiment budget and `t.maxCost()` look at this estimate, not `Usage.costUSD`. Costs in reports are computed only from a `PricingProfile` you specify explicitly and the recorded usage; they do not read the Runner's estimate either.
## StreamEvent variants at a glance
`StreamEvent` has ten variants, listed field by field below. For which assertions read them and what to watch for when using them, see the "Event table" and "Event details" sections that follow.
#### `message`
```ts theme={null}
{ type: "message"; role: "assistant"; text: string; loc?: SourceLoc }
```
#### `message`
```ts theme={null}
{ type: "message"; role: "user"; text: string; loc?: SourceLoc; sourceOrder?: number }
```
#### `operation.started`
```ts theme={null}
{
type: "operation.started";
operationId: string;
operation:
| {
kind: "tool";
name: string;
input: JsonValue;
tool?: ToolName;
/**
* The Adapter's protocol-level command / not-command classification. Temporarily optional for
* third-party Adapters that have not migrated; when absent, core must not fabricate it from the name, input, or shell text.
*/
command?: CommandProjection;
}
| { kind: "subagent"; name: string; remoteUrl?: string };
}
```
#### `operation.finished`
```ts theme={null}
{
type: "operation.finished";
operationId: string;
kind: "tool";
output?: JsonValue;
status: "completed" | "failed" | "rejected";
}
```
#### `operation.finished`
```ts theme={null}
{
type: "operation.finished";
operationId: string;
kind: "subagent";
output?: JsonValue;
status: "completed" | "failed";
}
```
#### `skill.loaded`
```ts theme={null}
{ type: "skill.loaded"; skill: string; operationId?: string }
```
#### `input.requested`
```ts theme={null}
{ type: "input.requested"; request: InputRequest }
```
#### `thinking`
```ts theme={null}
{ type: "thinking"; text: string }
```
#### `context.injected`
```ts theme={null}
{ type: "context.injected"; text: string; source?: string }
```
#### `compaction`
```ts theme={null}
{ type: "compaction"; reason?: string }
```
#### `error`
```ts theme={null}
{ type: "error"; message: string }
```
## Event table
| Event | What it says | Assertion / API that reads it |
| - | - | - |
| `message` | The agent (or user) said something | `turn.message`, `turn.event(eventMatch("message", ...))`, Judge material |
| `operation.started` (`kind: "tool"`) | Starts a tool call | `turn.calledTool(...)`, `turn.toolOrder(...)`, `turn.event(...)` |
| `operation.finished` (`kind: "tool"`) | The result of that tool call | `ToolMatch` with a status condition |
| `operation.started` (`kind: "subagent"`) | Delegates to a subagent | Kept in raw events and reports; there is no subagent-specific assertion |
| `operation.finished` (`kind: "subagent"`) | The subagent returns | Kept in raw events and reports; a failure counts toward `noFailedActions()` |
| `input.requested` | Stops to wait for human input (HITL) | `t.check(turn.status, equals("waiting"))`, `t.requireInputRequest()` |
| `thinking` | Thinking text | Shown in the view; check it with `t.check(turn.events, ...)` when needed |
| `compaction` | Context was compacted | Shown in the view; check it with `t.check(turn.events, ...)` when needed |
| `error` | This turn failed | Shown in the view; check it with `t.check(turn.events, ...)` when needed |
Event assertions have three entry points: `event(match)`, `notEvent(match)`, and `eventOrder(matches)`, all of which take a match condition built with `eventMatch(...)`. `eventMatch` supports three events: `message`, tool `operation.started`, and tool `operation.finished`. To require a number of occurrences, use the match condition's own methods such as `.atLeast()` and `.exactly()`. For other events, treat `turn.events` as an ordinary value and check it with `check(turn.events, valueMatch)`.
## Event details
### `message` — what was said
```ts theme={null}
{ type: "message", role: "assistant" | "user", text: string }
```
Each segment of assistant text produces one `message` with `role: "assistant"`. A tool result is not an assistant message; do not wrap tool output in `message`, or `t.reply` will read the wrong content. NiceEval records user-input `message` events automatically, so the Adapter does not need to emit them.
### Tool `operation.started` / `operation.finished` — which tool ran and what happened
```ts theme={null}
{ type: "operation.started", operationId: string,
operation: { kind: "tool", name: string, input: JsonValue,
tool?: ToolName, command?: CommandProjection } }
{ type: "operation.finished", operationId: string, kind: "tool",
output?: JsonValue, status: "completed" | "failed" | "rejected" }
```
* Every tool `operation.started` needs a tool `operation.finished` with the same `operationId`; concurrent calls are matched up by this ID. Use the ID already in the agent's return when there is one, such as AI SDK's `toolCallId` or Anthropic's `tool_use.id`; generate one by order only when there is none.
* Set `status` to what actually happened: `"failed"` when the tool execution failed, `"rejected"` when a person rejected it. Eval authors use `ToolMatch` status conditions to tell the two apart, so do not mix them up.
* Set `name` to the tool's original name.
* With only started and no finished yet, the call's status is `pending` and its output is recorded as unavailable. A finished event without `output` also means the output is unavailable. Do not write missing output as empty JSON, and do not let it look like an ordinary mismatch.
#### Pass tool events to `ToolMatch`
The same `shell` tool might first run `git status` and then `pnpm test`. The tool name only says which kind of tool it is. To tell the specific commands apart, Eval authors match on the normalized command information (command projection):
```ts theme={null}
import { commandMatch } from "niceeval/expect";
turn.calledTool(commandMatch("pnpm", { argsStart: ["test"] }));
```
`ToolMatch` compares one tool call at a time. For the complete filter conditions, and the JSON, path, output, and counting rules, see [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md).
When the native protocol directly provides a command's structured argv, generate the command information with the constructor exported from `niceeval/adapter`:
```ts theme={null}
import { commandProjection } from "niceeval/adapter";
events.push({
type: "operation.started",
operationId: call.id,
operation: {
kind: "tool",
name: call.name,
input: call.input,
command: commandProjection({
state: "available",
executable: call.executable,
args: call.args,
}),
},
});
```
`commandProjection()` keeps the original arguments the Adapter confirmed and normalizes them into the `logical-command/v1` format by NiceEval's shared rules. For example, `pnpm exec niceeval view` is matched exactly by `commandMatch("niceeval", { argsStart: ["view"] })`.
Mark the original command available (`state: "available"`) only when the native protocol itself provides argv, or when one command can be split out unambiguously by the protocol's grammar.
When the protocol provides only shell source, or the content is truncated or redacted, use `opaqueCommandProjection(reason)`. When you can confirm the call is not a command, use `notCommandProjection()`. When you cannot tell whether it is a command, downgrade actions coverage; do not guess from the tool name, input, or shell text.
### Subagent `operation.started` / `operation.finished` — who was delegated to
```ts theme={null}
{ type: "operation.started", operationId: string,
operation: { kind: "subagent", name: string, remoteUrl?: string } }
{ type: "operation.finished", operationId: string, kind: "subagent",
output?: JsonValue, status: "completed" | "failed" }
```
When the system under test delegates a task to a subagent and waits for it to return, emit this pair; the `operationId` pairing rule is the same as above. They stay in the raw event stream and reports. To check them, use an ordinary value match on `turn.events`; the tool matching in `eventMatch(...)` does not treat a subagent as a tool call.
### `input.requested` — stop and wait for a person (HITL)
```ts theme={null}
{ type: "input.requested", request: {
id?: string,
action?: string, // the action where it paused (such as a tool name)
input?: JsonValue, // the action's input
prompt?: string, // the question for the person
options?: { id: string, label?: string }[], // choices (approve / deny…)
} }
```
When the agent stops to wait for a person, each question awaiting an answer produces one `input.requested`, and the Turn's `status` is `"waiting"`. `t.requireInputRequest(filter)` matches `request` field by field, so fill in as many fields as you can, or the Eval cannot select it. For how to wire it up, see the [HITL section of the connection tutorial](/docs/tutorials/connect-your-agent).
### `thinking` / `compaction` / `error`
```ts theme={null}
{ type: "thinking", text: string }
{ type: "compaction", reason?: string } // context compaction; emitted when the Adapter parses it
{ type: "error", message: string }
```
Emit them when they exist; do not make them up when they do not. `compaction` mostly comes from coding agent CLIs, which compact automatically when the context fills up. These events are mainly for reports and diagnostics; when you need to assert on them, pass `turn.events` to `check` as an ordinary value.
## Three rules for mapping events
1. **Keep the real order**: event order is the order things happened. `toolOrder` and `eventOrder` match different calls from front to back, so the wrong order gives the wrong assertion result. `eventOrder` can only order events that `eventMatch` supports.
2. **Pair with `operationId`**: every started needs a finished with the same `kind` and ID. Only with complete pairs can NiceEval attribute status and input to the same tool call.
3. **Declare completeness honestly**: official converters declare event coverage (evidence coverage) according to what the Adapter can actually do. When the relevant input or action coverage is incomplete, a negative assertion (such as "did not call a tool") results in `unavailable`; "not seen" is never treated as "did not happen." Manual mappings must also declare coverage honestly; see the [Capabilities reference](/docs/reference/capabilities).
## A complete mapping example
When the agent's return includes step records, the mapping is a simple loop:
```ts theme={null}
import type { StreamEvent } from "niceeval";
function toStreamEvents(body: MyAgentResponse): StreamEvent[] {
const events: StreamEvent[] = [];
for (const step of body.steps) {
if (step.type === "tool_call") {
events.push({
type: "operation.started",
operationId: step.id,
operation: { kind: "tool", name: step.tool, input: step.args },
});
events.push({
type: "operation.finished",
operationId: step.id,
kind: "tool",
output: step.result,
status: step.error ? "failed" : "completed",
});
}
if (step.type === "text") events.push({ type: "message", role: "assistant", text: step.text });
}
return events;
}
```
## Related reading
* [Connect your agent](/docs/tutorials/connect-your-agent) — the tutorial for getting it working from scratch.
* [Capabilities](/docs/reference/capabilities) — what declaring "the event stream is complete" means.
* [Authoring Evals](/docs/tutorials/authoring) — the full set of assertions that read this stream.
# niceeval/expect Match Reference
Source: https://niceeval.com/docs/reference/expect
Pure Match factories from niceeval/expect: compose text, structure, and conditions, then register an Assertion with t.check.
`niceeval/expect` provides only pure Match factories. A Match compares one candidate value; it does not record a result or decide a Verdict or score. `t.check(value, match)` reads `value` when called and registers an Assertion directly; the returned handle only configures that same entry.
```ts theme={null}
import { and, excludes, includes, pattern } from "niceeval/expect";
```
## How to use it
```ts theme={null}
t.check(t.reply, includes("confirmed"));
t.check(t.reply, and(pattern(/order #\d+/i), excludes("cancelled")));
const config = await t.check(rawConfig, pattern(/runtime/)).orStop();
```
| Entry point | Purpose |
| - | - |
| `t.check(value, match)` | Takes exactly two arguments and registers an Assertion when called; a failure does not stop later code |
| `await handle.orStop()` | An async barrier on the same handle; stops the current continuation on mismatch or below |
| `handle.score(n)` / `t.score(n)` | Lets this item contribute a score in a scored Eval, or registers a contribution directly |
## Matchers
The signature and description for every matcher below are generated from the `niceeval/expect` source and stay in sync with the current implementation.
#### `and`
```ts theme={null}
export function and(
first: BooleanMatch,
...rest: readonly BooleanMatch[]
): BooleanMatch { ... }
```
#### `or`
```ts theme={null}
export function or(
first: BooleanMatch,
...rest: readonly BooleanMatch[]
): BooleanMatch { ... }
```
#### `not`
```ts theme={null}
export function not(match: BooleanMatch): BooleanMatch { ... }
```
#### `includes`
```ts theme={null}
export function includes(text: string, options?: TextMatchOptions): BooleanMatch { ... }
```
#### `excludes`
```ts theme={null}
export function excludes(text: string, options?: TextMatchOptions): BooleanMatch { ... }
```
#### `pattern`
```ts theme={null}
export function pattern(expression: RegExp, options?: TextMatchOptions): BooleanMatch { ... }
```
#### `similarity`
```ts theme={null}
export function similarity(expected: string): ScoreMatch { ... }
```
Normalized Levenshtein similarity; it is a continuous score and carries no default threshold or Verdict policy.
#### `includesUrl`
```ts theme={null}
export function includesUrl(min = 1): BooleanMatch { ... }
```
A plain-text matcher that requires at least min distinct http(s) URLs.
#### `hasSections`
```ts theme={null}
export function hasSections(min = 2): BooleanMatch { ... }
```
A plain-text matcher that requires at least min Markdown headings.
#### `isDefined`
```ts theme={null}
export function isDefined(label?: string): BooleanMatch> { ... }
```
#### `isTrue`
```ts theme={null}
export function isTrue(label?: string): BooleanMatch { ... }
```
A refinement matcher for value === true.
#### `isFalse`
```ts theme={null}
export function isFalse(label?: string): BooleanMatch { ... }
```
A refinement matcher for value === false.
#### `commandSucceeded`
```ts theme={null}
export function commandSucceeded(): BooleanMatch { ... }
```
An exitCode === 0 matcher for CommandResult-like candidates; it does not read stdout / stderr.
#### `equals`
```ts theme={null}
export function equals(expected: T): BooleanMatch { ... }
```
#### `matches`
```ts theme={null}
export function matches(
schema: S,
): BooleanMatch, "value"> { ... }
```
#### `satisfies`
```ts theme={null}
export function satisfies(
label: string,
predicate: (value: T) => boolean | Promise,
): BooleanMatch | BooleanMatch { ... }
```
#### `defineValueMatch`
```ts theme={null}
export function defineValueMatch(spec: {
readonly name: string;
readonly evaluate: (value: T) => boolean | { readonly state: "unavailable"; readonly reason: string } | Promise;
}): BooleanMatch | BooleanMatch { ... }
```
#### `defineScoreMatch`
```ts theme={null}
export function defineScoreMatch(spec: ManagedScoreMatchOptions | {
readonly name: string;
readonly score: (value: T) => number | Promise;
}): ScoreMatch { ... }
```
#### `eventMatch`
```ts theme={null}
export function eventMatch(
type: K,
options?: EventOptionsByType[K],
): EventMatch> { ... }
```
#### `mapValue`
```ts theme={null}
export function mapValue(
label: string, project: (value: T) => U, match: Match, "value"> & { readonly kind: "boolean" },
): BooleanMatch { ... }
```
Project a single value without replacing the original subject's refinement.
#### `mapEach`
```ts theme={null}
export function mapEach>(
project: (value: T) => U,
aggregate: Match>, "value"> & { readonly kind: "boolean" },
): BooleanMatch, Subject> { ... }
```
Project each complete item, retaining its material identity and order.
#### `filterWhere`
```ts theme={null}
export function filterWhere>(
selector: BooleanMatch, aggregate: BooleanMatch, CollectionValue>,
): BooleanMatch { ... }
```
Select only definite matches; an unknown selector makes the aggregate unavailable.
#### `countWhere`
```ts theme={null}
export function countWhere>(
item: BooleanMatch>, count: Match & { readonly kind: "boolean" },
): BooleanMatch { ... }
```
Compare an exact match count only after every item in a complete collection is known.
#### `defineMaterialMatch`
```ts theme={null}
export function defineMaterialMatch(definition: unknown): ContextualMatch { ... }
```
#### `defineContextMatch`
```ts theme={null}
export function defineContextMatch(definition: unknown): ContextualMatch { ... }
```
### Match tool calls
`ToolMatch` compares one `LogicalToolOccurrence`, and expresses the occurrence count with methods such as `.exactly(n)` and `.atLeast(n)`. `calledTool` accepts a name, an unquantified `ToolMatch`, or a positive occurrence Match; `notCalledTool` uses an unquantified selector to express zero matches.
```ts theme={null}
import {
commandMatch,
jsonMatch,
referencesAnyPath,
toolMatch,
} from "niceeval/expect";
turn.calledTool(
toolMatch("get_weather", {
input: jsonMatch({ city: "Taipei", unit: "celsius" }),
status: "completed",
}).exactly(1),
).label("Query weather precisely");
turn.calledTool(
toolMatch("read_file", {
input: referencesAnyPath([".env", "secrets/**"]),
}),
).label("Read a restricted path");
turn.notCalledTool(commandMatch("rm", { argsStart: ["-rf"] }));
```
Use `jsonMatch` for ordinary JSON, `referencesAnyPath` for paths, and `commandMatch` for command tokens. When output is absent, material is incomplete, or HITL has not finished, the Match becomes unavailable; it does not pretend to be an ordinary mismatch.
For complete signatures, material states, and three-valued counting rules, see [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md).
### Common usage examples
```ts theme={null}
t.check(t.reply, includes("Paris"));
t.check(t.reply, pattern(/order #\d+/i));
t.check(turn.data, equals({ intent: "refund" }));
t.check(turn.data, matches(OrderSchema));
t.check(turn.data, satisfies("returns an array", Array.isArray));
t.check(t.reply, similarity("The answer explains the refund window")).gate(0.8);
```
## Composition and three states
```ts theme={null}
t.check(t.reply, and(includes("required"), excludes("forbidden")));
t.check(t.reply, or(includes("approved"), includes("accepted")));
```
Matches within the same `and` / `or` must belong to the same domain. The internal result of a Boolean Match is `matched`, `mismatched`, or `unavailable`. `or` returns `unavailable` only when no child matches and at least one child's evidence is incomplete; it does not treat unknown as negative. A Match itself holds no Verdict, scoring, or control flow; `.gate(minimum)`, `.score(n)`, and `.orStop(minimum)` for a continuous measurement all belong to the registered AssertionHandle.
The following lists stable fields on Match values themselves. Brands and evaluators are module-private implementation details, not author API.
### `Match`
#### `domain`
```ts theme={null}
readonly domain: D;
```
#### `name`
```ts theme={null}
readonly name: string;
```
### `BooleanMatch`
#### `kind`
```ts theme={null}
readonly kind: "boolean";
```
#### `[matchRefinementBrand]`
```ts theme={null}
readonly [matchRefinementBrand]: () => R;
```
### `ScoreMatch`
#### `kind`
```ts theme={null}
readonly kind: "score";
```
## Ordering conditions in a collection
`inOrder` combines 2–64 ordinary value Matches into a collection Match, which you pass to `t.check` or to an application's collection assertion method.
It accepts a read-only array or a `MaterialCollection` with a completeness state, and looks for a strictly increasing subsequence by original position; the items need not be adjacent.
```ts theme={null}
import { equals, inOrder, not, or } from "niceeval/expect";
t.check([1, 0, 2], inOrder([equals(1), equals(2)])).gate();
t.check([2, 1], not(inOrder([equals(1), equals(2)]))).gate();
t.check([3, 4], or(
inOrder([equals(1), equals(2)]),
inOrder([equals(3), equals(4)]),
)).gate();
```
The same item cannot satisfy two steps, and each step can use `and` to constrain the same fact. The application is responsible for providing the formal commit order; a request being sent does not mean the operation was adopted or finished executing.
When you only need both parties to have spoken, check existence separately.
An incomplete collection returns undecidable. A complete collection passes when there is known ordering evidence, fails when no ordering is possible, and is undecidable when it can only hold by depending on unknown items.
Diagnostics keep the original positions, material IDs, and step judgments. Ordering combinations of ToolMatch and EventMatch keep their own Agent scope and cannot be mixed with ordinary value steps.
## Custom matchers
```ts theme={null}
import { defineValueMatch } from "niceeval/expect";
const validEmail = () =>
defineValueMatch({
name: "valid email",
evaluate(value) {
return value.includes("@");
},
});
t.check(t.reply, validEmail());
```
Use `defineValueMatch` for a custom Boolean Match and `defineScoreMatch` for a continuous score. Both implement only a single-candidate comparison; reading files, scanning events, deciding coverage, and changing a Verdict remain the responsibility of Assertion registration entry points and the Runner.
# Official Adapter Overview
Source: https://niceeval.com/docs/reference/official-adapters
NiceEval's built-in Sandbox and non-Sandbox Adapters, their authentication, how to install MCP servers, Skills, and plugins for Sandbox Adapters, and how to use an Agent's official configuration file.
[NiceEval](https://niceeval.com/) ships several official Adapters—factory functions exported from `niceeval/adapter`—in two families based on whether the system under test needs an isolated workspace. **Sandbox Adapters** (`claude-code` / `codex` / `bub`) run a coding-Agent CLI in a Docker or cloud Sandbox and can install MCP servers, Skills, and Python plugins. **Non-Sandbox Adapters** connect non-intrusively to an HTTP service that is already running, or save event-stream mapping work when you write an Adapter yourself. This page is organized by family and individual Adapter, and focuses on each Adapter's configuration. For choosing one and getting the first Eval working, see [Connect your Agent](/docs/tutorials/connect-your-agent).
## Sandbox Adapters
All three built-in Sandbox Agents use `defineSandboxAgent`, authenticate through environment variables that factory options can override, and support installing extensions during Sandbox `setup`. For running a built-in Sandbox Agent, directory structure, and writing a custom Sandbox Adapter, see [Sandbox Agent](/docs/tutorials/sandbox-agent). This page covers only what each Adapter can install and how to write its options.
On this page, `settingsFile` and `configFile` are both resolved relative to the NiceEval project root. The project root is the current working directory when `niceeval` runs—the directory containing `niceeval.config.ts`—not the directory containing the Eval or Experiment file. For example, when an Experiment is at `experiments/web/no-search.ts` and the configuration is at `configs/codex/no-web.toml`, still write `configFile: "configs/codex/no-web.toml"`.
### claude-code
* **Authentication**: `ANTHROPIC_API_KEY`, which the `apiKey` factory option can override; optionally `ANTHROPIC_BASE_URL`, through the `baseUrl` factory option.
* **Install an MCP server**: use `mcpServers`. During `setup`, it writes to the top-level `mcpServers` field in the user-level `~/.claude.json` in the Sandbox. The two shapes are distinguished by fields. A local stdio process uses `command`, optionally with `args` / `env`. A remote Streamable HTTP endpoint uses `url`, optionally with `headers`, which are written verbatim to request headers and commonly contain `Authorization`; it is written as a `{ "type": "http", "url": …, "headers": … }` entry. The `url` must be reachable inside the Sandbox. If the service runs on your machine, first expose it as a public address through a tunnel such as cloudflared or tailscale.
* **Install a Skill**: use `skills: SkillSpec[]`. A local Skill is `{ kind: "local", path }`, a file or directory read from the project root. A Repo Skill is `{ kind: "repo", source, ref, skills }`, which can pin a commit or tag and enable only part of a multi-Skill repository. They are installed at project-level `.claude/skills//` in the Sandbox, where the claude CLI discovers them natively. The Adapter normalizes a native `Skill` invocation into a `skill.loaded` raw event instead of also recording it as a tool call. The public assert-first API currently has no dedicated Skill assertion, and it cannot be disguised as `calledTool("Skill", ...)`.
* **Install a native Plugin**: use `plugins: ClaudeCodePluginSpec[]`. Each item declares a Marketplace connection (`name` / `source` / optional `ref`) and the Plugin name inside it. This type belongs only to claude-code and cannot be passed to codex. An Experiment that reuses a Sandbox needs no extra residual cleanup: before every Attempt, the named Marketplace registration and installed Plugin are reinstalled from the declared `source` / `ref`.
* **Official configuration file**: `settingsFile` is a local project path on the machine running NiceEval, not a path inside the Sandbox. It points to a complete Claude Code `settings.json`. The path is relative to the project root and accepts only an ordinary relative path or a `./` prefix. `..`, an absolute path, `~`, and a symlink that resolves outside the project root all cause an error.
The Adapter reads the file locally, uploads it, and replaces the otherwise-empty user-level `~/.claude/settings.json` in the Sandbox verbatim. It does not inherit host configuration, deep-merge, or reserialize. `model` and `env` belong to the Experiment and Adapter. If they appear in the file, setup errors and names the conflicting key.
Keep secrets in environment variables, not the configuration file.
* **Post-install scripts**: `postSetup: SandboxCommand[]` runs your commands in array order inside the Sandbox after settings are written and MCP servers, Skills, and Plugins are all installed. A typical use is a Plugin's own setup script, such as one that registers a Hook in global configuration. This kind of script must wait until its installed files are ready. Its paired cleanup, `preTeardown: SandboxCommand[]`, runs in reverse order before the Agent's own teardown step, and runs only if execution has reached the point where `postSetup` ran. A command that throws is an infrastructure error—the Attempt is recorded as `errored`—not the Agent failing the task.
* **Tracing**: the claude CLI's beta native telemetry (`CLAUDE_CODE_ENHANCED_TELEMETRY_BETA`). Spans carry only structure and timing. For details, see [OTel integration](/docs/tutorials/connect-otel).
For example, use `configs/claude-code/no-web.json` to disable built-in web search:
```json theme={null}
{
"$schema": "https://json.schemastore.org/claude-code-settings.json",
"permissions": { "deny": ["WebSearch", "WebFetch"] }
}
```
```ts theme={null}
import { defineExperiment } from "niceeval";
import { claudeCodeAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";
export default defineExperiment({
agent: claudeCodeAgent({
mcpServers: [
{ name: "browser", command: "npx", args: ["-y", "@anthropic/mcp-browser"] },
],
skills: [
{ kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2", skills: ["effect"] },
{ kind: "local", path: "skills/repository-guide.md" },
],
plugins: [
{
marketplace: { name: "acme", source: "acme/claude-code-plugins", ref: "v1.3.0" },
name: "safe-shell",
},
],
settingsFile: "configs/claude-code/no-web.json",
}),
model: "claude-sonnet-4-6",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});
```
### codex
* **Authentication**: `CODEX_API_KEY`, which the `apiKey` factory option can override. Optionally, use `CODEX_BASE_URL` through the `baseUrl` factory option for an OpenAI-compatible proxy.
* **Agent process environment**: `env` is injected into every `codex exec` and `codex exec resume`. Session lifecycle Hooks started by Codex, dynamic MCP request headers, and command subprocesses inherit it. Values are not concatenated into shell text or written to the installation manifest, and potential sensitive values are redacted from timing, execution, and error evidence. `CODEX_API_KEY` still comes from `apiKey` or the host environment and overrides the same key in `env`. `PATH` is managed by the Sandbox and cannot be declared through `env`; its presence errors at `codexAgent()` call time. Use the Sandbox `pathPrepend` instead; see [Sandbox Providers](/docs/tutorials/sandbox-providers).
* **Install an MCP server**: use `mcpServers`. During `setup`, it appends a `[mcp_servers.]` section to `~/.codex/config.toml`. The two shapes are distinguished by fields. A local stdio process uses `command`, optionally with `args` / `env`. A remote Streamable HTTP endpoint uses `url`, optionally with `headers`, written as a `[mcp_servers..http_headers]` subtable. The `url` must be reachable inside the Sandbox. If the service runs on your machine, first expose it as a public address through a tunnel such as cloudflared or tailscale.
Use plural `mcp_servers`: the codex CLI silently ignores singular `[mcp_server.x]`, so MCP never attaches and no error appears. Check with `codex mcp list`.
* **Install a Skill**: use `skills: SkillSpec[]`, the same type as claude-code. They are installed in `.agents/skills//`, and a discovery instruction is written to AGENTS.md. Codex has no native Skill tool like claude-code, so installing a file alone does not make it read it. To assert whether it was used, check whether it actually ran the shell command that reads the file; there is no direct tool call to assert.
* **Install a native Plugin**: use `plugins: CodexPluginSpec[]`. It declares the Marketplace connection (`name` / `source` / optional `ref` / optional `sparse`) and the Plugin name within it. `sparse` is an array of paths such as `[".agents", "plugins/repo-map"]`; each adds `--sparse ` to `codex plugin marketplace add`, so a large repository fetches only the paths a Plugin needs without changing the installed result. This type belongs only to codex and cannot be passed to claude-code. An Experiment that reuses a Sandbox needs no extra residual cleanup: before every Attempt, the named Marketplace registration and installed Plugin are reinstalled from the declared `source` / `ref`.
* **Official configuration file**: `configFile` is a local project path on the machine running NiceEval, not a path inside the Sandbox. It points to a complete Codex `config.toml`. The path is relative to the project root and accepts only an ordinary relative path or a `./` prefix. `..`, an absolute path, `~`, and a symlink that resolves outside the project root all cause an error.
The Adapter reads the file locally, uploads it, and replaces the otherwise-empty user-level `~/.codex/config.toml` in the Sandbox verbatim. It does not inherit host configuration, concatenate, deep-merge, or parse and rewrite. `model`, `model_provider`, `model_providers`, `model_reasoning_effort`, `mcp_servers`, and `otel` belong to the Experiment and Adapter. If they appear in the file, setup errors and names the conflicting key.
Keep secrets in environment variables, not the configuration file.
* **Post-install scripts**: `postSetup: SandboxCommand[]` has the same semantics as claude-code. It runs your commands in order in the Sandbox after all installation steps finish, so it is suited to a Plugin's own setup script. Its paired `preTeardown: SandboxCommand[]` cleans up in reverse order before Agent teardown, and runs only if execution has reached the point where `postSetup` ran. A Hook registered by a script into Codex global configuration does not need interactive trust confirmation: at runtime, `codex exec` has already bypassed the Hook trust gate, so the Hook takes effect directly.
* **Tracing**: built in and configured through the `[otel.trace_exporter.otlp-http]` section of `config.toml`, using `http/json`.
For example, use `configs/codex/no-web.toml` to disable built-in web search:
```toml theme={null}
#:schema https://developers.openai.com/codex/config-schema.json
web_search = "disabled"
```
```ts theme={null}
import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";
const memorySpace = "memorybench-nowledge";
export default defineExperiment({
agent: codexAgent({
env: { NMEM_SPACE: memorySpace },
mcpServers: [
{ name: "browser", command: "npx", args: ["-y", "@anthropic/mcp-browser"] },
{ name: "team-memory", url: "https://mem.example.com/mcp/", headers: { Authorization: `Bearer ${process.env.MEM_API_KEY}` } },
],
skills: [{ kind: "repo", source: "Effect-TS/skills", ref: "8f3c1a2", skills: ["effect"] }],
plugins: [
{
marketplace: { name: "acme", source: "acme/codex-plugins", ref: "8f3c1a2" },
name: "repo-map",
},
],
configFile: "configs/codex/no-web.toml",
}),
flags: { memorySpace },
model: "gpt-5.4",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});
```
### bub
* **Authentication**: `BUB_API_KEY` plus `BUB_API_BASE` for an OpenAI-compatible proxy. The `apiKey` / `apiBase` factory options can override them.
* **Install a Skill**: use `skills: SkillSpec[]`, the same type as the other two Adapters. They are installed in `.agents/skills//`, with a discovery instruction in AGENTS.md.
* **Install a Plugin**: use `pythonPlugins: PythonPluginSpec[]`. `{ package }` can be a PyPI package, version specifier, or git URL. During `setup`, it becomes `uv tool install … --with `. This type belongs only to bub. The package set enters the installation checkpoint key, so two variants with different Plugins never reuse the same installation cache.
* **Prebuilt Bub**: NiceEval's E2B recipe includes Bub, the OTel Plugin, and the Python Plugin set in an installation fingerprint. The Adapter reuses only an environment with an exactly matching fingerprint. Simply having `bub` on `PATH` is not proof of compatibility. For the build entry point, see [Prebuilt Sandbox environments · Build on the official baselines](/docs/tutorials/prebuilt-sandbox#continue-building-from-the-official-baseline).
* bub has no `mcpServers`. MCP belongs only to Adapters that support it, and this configuration has no such field.
* **Post-install scripts**: `postSetup: SandboxCommand[]` has the same semantics as the other two Adapters. It runs your commands in order in the Sandbox after all installation steps finish. Its paired `preTeardown: SandboxCommand[]` cleans up in reverse order before Agent teardown, and runs only if execution has reached the point where `postSetup` ran.
* **Installation method**: `uv tool install`, a PyPI package rather than an npm package. The first installation creates a checkpoint cache that speeds later Sandboxes.
* **Tracing**: built in, injected through environment variables, using `http/protobuf`.
```ts theme={null}
import { defineExperiment } from "niceeval";
import { bubAgent } from "niceeval/adapter";
import { dockerSandbox } from "niceeval/sandbox";
export default defineExperiment({
agent: bubAgent({
skills: [{ kind: "local", path: "skills/repository-guide.md" }],
pythonPlugins: [{ package: "bub-plugin-memory==1.3.0" }],
}),
model: "gpt-5.4",
sandbox: dockerSandbox({ source: { type: "image", image: "node:24-slim" } }),
});
```
### Compare the three Sandbox Adapters
| | claude-code | codex | bub |
| - | - | - | - |
| Authentication environment variable | `ANTHROPIC_API_KEY` | `CODEX_API_KEY` + optional `CODEX_BASE_URL` | `BUB_API_KEY` + `BUB_API_BASE` |
| Additional Agent process environment | ❌ | ✅ `env` | ❌ |
| MCP server | ✅ `mcpServers` (stdio + HTTP) | ✅ `mcpServers` (stdio + HTTP) | ❌ |
| Skill | ✅ `skills: SkillSpec[]` (native discovery) | ✅ `skills: SkillSpec[]` (+ discovery instruction) | ✅ `skills: SkillSpec[]` (+ discovery instruction) |
| Native Plugin | ✅ `plugins: ClaudeCodePluginSpec[]` | ✅ `plugins: CodexPluginSpec[]` | ❌ |
| Official configuration file | ✅ `settingsFile` (complete settings.json) | ✅ `configFile` (complete config.toml) | ❌ |
| Python Plugin | ❌ | ❌ | ✅ `pythonPlugins: PythonPluginSpec[]` |
| Post-install scripts | ✅ `postSetup` / `preTeardown` | ✅ `postSetup` / `preTeardown` | ✅ `postSetup` / `preTeardown` |
| Tracing | ✅ (beta, structure and timing only) | ✅ | ✅ |
| Installation method | npm global package (the image needs Node) | Native package with its runtime included (unpack and use; the image needs no Node) | `uv tool install` (PyPI) |
What was installed can be verified. At the end of `setup`, the Adapter hands an installation manifest to the Runner, which stores it as the Attempt Artifact `agent-setup.json`. Read it in the library with `attempt.agentSetup()`. The manifest never reaches Sandbox disk, so the Agent cannot see it. It records only sources, refs, Skill and Plugin names, resolved versions, and an official configuration file's project-relative path and SHA-256. It does not preserve configuration contents, API keys, or environment-variable values. MCP servers likewise record only non-sensitive fields: stdio records `name` / `command` / `args`, not `env`; HTTP records `name` / `url`, not `headers`.
## Non-Sandbox Adapters
When the system under test does not need an isolated workspace—an already-deployed HTTP service or a home-grown Agent loop—use a non-Sandbox Adapter for a non-intrusive connection. Docker is not required.
* **`uiMessageStreamAgent`**: built in, connects non-intrusively to an AI SDK `useChat` backend HTTP endpoint with zero mapping, including HITL.
```ts theme={null}
import { uiMessageStreamAgent } from "niceeval/adapter";
export default uiMessageStreamAgent({
name: "my-assistant",
url: "http://localhost:3000/api/chat",
body: (ctx) => ({ model: ctx.model }),
});
```
* **SDK event-stream converters** (`createClaudeSdkEventStream` / `createPiAgentEventStream` / `createCodexThreadEventStream`): for hand-written SDK Adapters. NiceEval already provides the mapping from native frames to standard events. You only need to add the code that requests the application interface, then use `driveFrameStream` to read results frame by frame.
* **`turnFromAiSdk`**: converts AI SDK `generateText` / `streamText` results into the standard event stream when you write your own HTTP web-Agent Adapter.
For the full parameters, capability tables, and example code for these Adapters, see [Built-in Agent Capabilities Reference](/docs/reference/builtin-agents).
## How to choose
* For a coding Agent that must change code and run commands on a real filesystem: `claude-code` / `codex` / `bub`. Choose configuration in the Sandbox Adapter sections above.
* For an AI SDK application with a `useChat` backend: `uiMessageStreamAgent`, with zero mapping and HITL included.
* For another deployed Agent system over HTTP / gRPC: use non-Sandbox integration and write an Adapter. The official SDK converters cover most mapping work.
## Related reading
* [Connect your Agent](/docs/tutorials/connect-your-agent) — the full view: configuring an Experiment and writing an Eval.
* [Sandbox Agent](/docs/tutorials/sandbox-agent) — running built-in Sandbox Agents and writing your own.
* [Built-in Agent Capabilities Reference](/docs/reference/builtin-agents) — a capability-by-capability inventory of every Adapter and full example code for non-Sandbox Adapters.
* [OTel integration](/docs/tutorials/connect-otel) — send spans to [NiceEval](https://niceeval.com/) to get the call waterfall in `niceeval view`.
* [Adapter Reference](/docs/reference/define-agent) — complete parameters for `defineAgent` / `defineSandboxAgent`.
# Keep Sandbox state to debug problems
Source: https://niceeval.com/docs/troubleshooting/debug-sandbox
Use --keep-sandbox to keep a failed Attempt's Sandbox as an environment you can wake at any time, enter it with niceeval sandbox enter to investigate by hand, and inspect and clean up with sandbox list / stop.
By default, a Sandbox is destroyed after each Attempt ends. Most problems can be located first with `niceeval show @` or `niceeval view` by looking at the verdict, assertions, file changes, commands, and conversation. For the full workflow, see the [Debugging guide](/docs/troubleshooting/debugging).
Some problems, though, can only be seen in a live environment:
* **The environment does not start**: installing dependencies fails during setup, or the agent CLI cannot start. The agent has not started running yet, so the conversation record is empty. The fastest fix is to enter the Sandbox and rerun the install command by hand.
* **The change is outside `git diff`**: which packages were installed globally, what configuration was written under `$HOME`, and what `PATH` actually is. The saved file changes do not include these.
* **Rerunning is too slow**: a cold start plus installation takes minutes. When you want to test guesses one by one, keeping the environment is much faster than rerunning every time.
## Keep the state while running
Add `--keep-sandbox` to `niceeval exp`:
```bash theme={null}
pnpm exec niceeval exp local onboarding/tool-first --keep-sandbox # Same as --keep-sandbox=failed
pnpm exec niceeval exp local onboarding/tool-first --keep-sandbox=all # Keep passing ones too
```
`--keep-sandbox` has two settings:
* `failed` (the default) keeps Attempts whose verdict is `failed` or `errored`, including ones interrupted by a timeout.
* `all` keeps passing ones too. Use it when tuning a setup Hook or checking the real state of the environment when it passes, so you do not have to make an eval case fail on purpose.
Without this flag, everything is destroyed. A run with `--keep-sandbox` always really executes and never reuses earlier results.
When the run ends, the summary lists which Sandboxes were kept and how to enter them:
```text theme={null}
Kept sandboxes (1)
@0123456789ABCDEFGHJKMNPQRS onboarding/tool-first #1 errored docker · a3f9c2d1
enter: niceeval sandbox enter a3f9c2d1
Stop them with: niceeval sandbox stop --all
```
Each line gives three things: the Attempt locator, the Sandbox instance id, and the command to enter the environment. To see the results saved for this Attempt, use `niceeval show @` or `niceeval view --run `.
Kept Sandboxes do not keep running and consuming resources: Docker containers stop on disk, E2B microVMs pause billing, and Vercel saves the file system.
## Enter the environment
```bash theme={null}
pnpm exec niceeval sandbox enter a3f9c2d1
```
`niceeval sandbox enter` wakes the environment first, then opens a shell in the working directory. Inside is the environment as it was when this Attempt finished, so you can run commands, browse files, and reproduce the failure by hand.
After you exit the shell, the environment goes back to sleep automatically. To keep it running, add `--leave-running`.
## Inspect and clean up
The project keeps a record of kept Sandboxes. Manage them with `niceeval sandbox`:
```bash theme={null}
pnpm exec niceeval sandbox list # List kept Sandboxes and their state
pnpm exec niceeval sandbox enter a3f9c2d1 # Wake and enter; goes back to sleep on exit
pnpm exec niceeval sandbox stop a3f9c2d1 # Destroy the given Sandbox (the id can be a unique prefix)
pnpm exec niceeval sandbox stop --all # Destroy all
```
These management commands do not read `niceeval.config.ts`. Even when the configuration file cannot run for now, you can still inspect or destroy environments. The commands read `.env` to prepare Provider credentials; when `.env` cannot be read, fix it first and retry.
The `sandbox` commands have no `--json` output. `niceeval sandbox list --json` reports `Unknown option '--json'` directly and does not go on to print human-readable text.
`stop` can be run repeatedly. A Sandbox that is already gone (deleted by hand, or expired in the cloud) is not an error; the command just removes the record and says so. If the Provider fails to destroy it, the command keeps the record and returns an error so you can retry later, and it does not hide a resource that is still alive from the list.
There is also a reminder if you forget to clean up: at the start of the next run, if Sandboxes from earlier are still kept, the terminal prints one line about it.
## Provider differences
* **Docker**: the container stops on disk, uses no memory, survives a Docker restart, and starts automatically when you enter it. Containers never disappear on their own, so Docker is the only Provider that needs active cleanup. Besides `niceeval sandbox stop`, you can also check directly with `docker ps -a -f label=niceeval.keep-candidate=true`.
* **E2B**: the microVM is paused, and files and memory are saved together. Billing stops while it is paused, it is kept indefinitely, and it resumes automatically when you enter it.
* **Vercel Sandbox**: the microVM is stopped. The file system is saved and can be resumed later, but memory state is not kept, so processes must restart after waking. After the Provider's retention period passes, `niceeval sandbox list` marks it `expired`.
* **Custom Providers**: Providers created with `defineSandbox` do not support keeping environments. The later `sandbox stop` does not load user configuration, so it cannot safely recover the destroy function you wrote in a new process.
## Boundaries
A kept Sandbox is only for investigation. It cannot continue running, and it cannot be rescored. Verdicts, assertions, and file changes follow the saved results. Nothing you change in the environment is written back to the results. For how to view results, see [View run results](/docs/tutorials/viewing-results).
Sandboxes left behind by a forced kill are not registered and do not appear in `sandbox list`. For how to reclaim them, see [Recover after a forced kill](/docs/troubleshooting/recover-after-kill#reclaim-sandboxes-left-by-a-forced-kill).
# Debug failed eval results
Source: https://niceeval.com/docs/troubleshooting/debugging
When an eval fails or errors, start from a specific Run or Attempt and use show to drill into errors, conversations, timing, and changes, to find whether the problem is in the application, the Adapter, the environment, or the assertion.
A batch of evals has finished, and a few are `failed` or `errored`. You need to find the cause: the application answered wrong, the Adapter mapping missed something, the environment did not start, or an assertion is too strict.
Start from a specific Run or Attempt. Use `niceeval show` in the terminal and `niceeval view` in the browser; scripts and CI use `query` to read JSON.
## Find the problem Attempt from a Run
First look at the run as a whole. The Run ID is in the `NEXT` panel when `exp` ends:
```sh theme={null}
pnpm exec niceeval show --run
```
The output lists the Run's time, each eval case's verdict, scores, result coverage, usage, and limits. Each Attempt has a locator starting with `@` in front of it. Copy the one with the problem:
```sh theme={null}
pnpm exec niceeval show @
```
This screen first shows the verdict and an assertion summary. For `errored`, it shows the phase that failed and the full error text. Looking here first is usually enough.
## Choose what to look at by question
When the summary is not enough, add one flag for the direction you suspect:
```sh theme={null}
pnpm exec niceeval show @ --source
pnpm exec niceeval show @ --execution
pnpm exec niceeval show @ --timing
pnpm exec niceeval show @ --usage
pnpm exec niceeval show @ --diff
```
| You want to know | Flag |
| - | - |
| The eval source used in this run and each assertion's result | `--source` |
| What the Agent said and which tools it called | `--execution` |
| Which phase took the time, and whether it got stuck in setup | `--timing` |
| How many tokens, requests, and how much cost this run used | `--usage` |
| Which files the Agent changed | `--diff` |
`--execution` shows a condensed execution outline. To expand one item, copy the ID next to it:
```sh theme={null}
pnpm exec niceeval show @ --execution --expand
```
## Find out why an assertion got this score
When the problem comes down to one assertion, use `query` to get structured details. It returns that assertion's contributions, measurements, and the Judge's rationale:
```text theme={null}
query discover
→ attempt.get(locator)
→ choose entryId from assertions.entries
→ attempt.assertion.detail(locator, entryId)
```
Do not work backward from the formatted text of `show` to assertion fields. For a complete request example, see [View run results](/docs/tutorials/viewing-results#see-why-an-assertion-got-its-score).
## Cannot open results
| Error | Next step |
| - | - |
| The project's results file cannot be found | Confirm you are running in the project directory, and check that `.niceeval/record.sqlite` exists. |
| The file named by `--record` is rejected | Confirm it is a complete single file copied after the run ended, and that no other process was writing during the copy. Do not delete WAL files that have not been merged. External files go through integrity and data consistency validation. |
| Unsupported format version or invalid database | Keep the original file and check which NiceEval version wrote it. Supported historical formats are upgraded automatically; unknown formats or damaged data are not patched, and NiceEval does not read only part of them. |
| A Run is still `active`, but its original process has ended | First check with `pnpm exec niceeval run show `, then run `pnpm exec niceeval run recover --yes`. |
Published results cannot be edited by hand. When you need different results, run again to create a new Run. `niceeval run recover` only ends a Run whose original process is proven to have ended, and keeps every published Attempt. For the full steps, see [Recover after a forced kill](/docs/troubleshooting/recover-after-kill).
## Read historical Records
The format version here is an internal identifier of the results file and is unrelated to NiceEval's npm package version. Do not try to upgrade by editing the version field in the database.
`show`, `query`, `run list`, `run show`, and `view` can read historical results files in the 0.15, 0.16, and 0.17 formats directly. When reading, NiceEval upgrades them automatically in a private copy. The original file, Runs, Attempts, and references stay unchanged, and you do not need to run an eval first.
When reading with `--record`, the file must be a complete single file that has already been handed over, and no other process can modify it while it is read. The command refuses to read when the file still has unmerged WAL, a run is still writing to it, or NiceEval cannot confirm that the writing process has ended. Do not delete the files next to it to get around the check.
When you run write commands such as `exp` normally in a project, NiceEval first backs up and validates, then upgrades supported old formats automatically. Reading a project in the current format never runs an upgrade or recovery.
## Results are incomplete or cannot be compared
Scripts or CI first run `pnpm --silent exec niceeval query discover` to find the right query, then use `explain` to check the request, and finally use `run` to execute it. The result shows which results are missing, what issues exist, which evidence was used, and how to fix it.
`query` only outputs JSON and is not meant for people to read directly. To explain a score item by item, use `attempt.get` and `attempt.assertion.detail` instead of looking only at the summary score.
When you need a browser interface, run `pnpm exec niceeval view --run --no-open`. Do not parse the View page, and do not query the tables in SQLite directly.
To compare two Runs, use `runs.compare`. Its `exact` and `paired` modes refuse outright when evals or pairs cannot be aligned, instead of quietly shrinking the denominator.
## Read batched errors
In a run, the first time a known kind of error appears, the terminal immediately shows the full cause, the responsible party, the source location, and the next step. When the run ends, a summary lists the total count, the affected objects, and the source locations. Migration guides are shown in full only once per `guideId`, not repeated for each object.
Errors are grouped only when their `code`, `owner`, fix target, and `guideId` are all identical. The summary keeps at most 32 groups, and each group shows at most 32 affected objects and 32 source locations. Counts beyond that are stated separately, and every original error is still kept on its own. Unknown errors without a stable source are shown separately and are never merged just because their message or stack looks similar.
Source locations come from known call sites or source maps, and show unknown when none is found. NiceEval does not scan the whole project to report errors, and does not require the application to enable extra AST scanning.
## Old project state blocks a run
If the project data predates the formats that support automatic upgrade, or an error says old directories such as `locks/` or `sessions/` block writes, follow these steps to keep the history. The 0.15, 0.16, and 0.17 formats mentioned above are upgraded automatically and do not need this process.
1. Stop any NiceEval process still running in the original project. Do not decide whether a lock file can be deleted based on its modification time.
2. Keep the original project and the complete `.niceeval` directory. To view history, use the NiceEval version that wrote the data, in the original project. If you are not sure which version that is, keep everything as is and check first.
3. Copy the configuration, eval cases, experiments, and other project files to a new directory, without `.niceeval`, and install the current version of NiceEval in the new directory.
4. Run the original experiment in the new directory, for example `pnpm exec niceeval exp --rerun all`. New results go into the new directory's results file, and the original project's history stays as is.
This process does not bring old results into the new results file. Do not delete individual lock files or data files to get around the check. `niceeval state migrate --all` manages user-level service state and cannot convert a project's results file.
## View shows old results
When `niceeval view` opens, it reads the results published at that moment and stays on them. When the page says new results are available, confirm the refresh to see the latest content. If the refresh fails, the page keeps the last results it could read.
For the normal workflow, see [View run results](/docs/tutorials/viewing-results). For command details, see [CLI](/docs/reference/cli).
# Recover after a forced kill
Source: https://niceeval.com/docs/troubleshooting/recover-after-kill
After kill -9, a CI timeout, or a power loss interrupts the process, keep the results that already finished, end the Run that lost its process, and reclaim leftover Sandboxes.
When `niceeval exp` is interrupted directly by `kill -9`, a CI time limit, or a power loss, the process has no chance to clean up. Attempts that already finished are still saved in the project's `.niceeval/record.sqlite` and can be read as usual; unfinished ones never leave half-written results.
When you press Ctrl+C or send SIGTERM normally, NiceEval cleans up by itself: it saves the finished Attempts and marks the remaining planned Attempts as interrupted. This page only covers cases where the process had no chance to clean up.
## Look at the saved results first
List the Runs this run created. If you got the `invocationId` from `--json` output or the CI log, use it to narrow the list:
```bash theme={null}
pnpm exec niceeval run list --invocation
pnpm exec niceeval run show
pnpm exec niceeval show --run
```
You can also look in the browser: `pnpm exec niceeval view --run `.
A Run that was force-killed is still `active`. It shows the Attempts that already finished, and the ones that did not run show as `pending`. Do not decide it can be taken over based on how long it has run, an expired PID, or a stopped heartbeat.
## End the Run that lost its process
Run the recovery only after confirming the original process has ended:
```bash theme={null}
pnpm exec niceeval run recover --yes
```
`run recover` first verifies that the original process really has ended. Once verified, it marks the Run `interrupted` and records why each unfinished Attempt is missing. It does not delete the Run or the finished Attempts, and the original process cannot write again even if it lingers.
When the evidence is insufficient, the command refuses to recover. In that case, first confirm that the original process, the CI job, and any remote workers have all stopped, then retry. Do not bypass this check because "it has waited long enough".
## Rerun the unfinished part
When you need complete results, run the original Experiment again. The new run creates a new Run; Attempts that finished last time and meet the reuse conditions are reused, and the rest run again. Before running, you can add `--dry` to see which will be reused and which will rerun. For the rules, see [Rerun only the evals affected by a change](/docs/tutorials/rerun-and-cache).
## Reclaim Sandboxes left by a forced kill
Sandboxes that were running at the time of a forced kill are not registered, so `niceeval sandbox list` does not show them. First list these leftover instances read-only:
```bash theme={null}
pnpm exec niceeval sandbox list --orphans
```
After confirming, reclaim them:
```bash theme={null}
pnpm exec niceeval sandbox prune
```
* `orphan` means the process that created it on the same host is proven to have ended, so it can be destroyed safely.
* `unverified` means it came from another host or cannot be confirmed, and it is not deleted by default. Use `pnpm exec niceeval sandbox prune --force` only after confirming nobody is using it.
* Compose companion containers and networks are listed and destroyed as a group.
* Vercel Sandbox leftovers cannot be looked up; they are reclaimed when the Provider's retention period ends.
* Environments kept with `--keep-sandbox` are not affected by `prune`. Keep managing them with `niceeval sandbox stop`; see [Keep Sandbox state to debug problems](/docs/troubleshooting/debug-sandbox).
## Recover a shared-state lease
For an Experiment that declares `sharedState.key`, only one run can hold the same key at a time. When the original run did not release it normally, new runs keep waiting until you recover it explicitly.
First run the check command without a token to see the current holder's information:
```bash theme={null}
pnpm exec niceeval exp --teardown \
--recover-shared-state
```
After confirming that the original run has ended, and that the remote services and checkpoints it operated on have stopped changing, copy the token the command shows and confirm explicitly:
```bash theme={null}
pnpm exec niceeval exp --teardown \
--recover-shared-state \
--owner-token \
--confirm-owner-terminated \
--confirm-remote-quiesced
```
Recovery reruns `teardown` only for this Experiment. When the token is wrong, the token has expired, or teardown fails, the lease is not released, and later holders' leases are not affected. After fixing teardown, try again with the token currently shown.
# Confirm Old Results Still Hold and Skip a Rerun
Source: https://niceeval.com/docs/tutorials/accept-results
When configuration changes and NiceEval will not reuse old results, but you have confirmed they are unaffected, accept them explicitly with niceeval accept and save a paid rerun.
You moved your company's base image from an old registry to a new one. The image content did not change by a single byte; only the image address in the Experiment changed. NiceEval sees that the configuration changed, so it does not reuse the last results automatically, and `--dry` shows that the whole batch must run again. You know the change does not affect any task, but rerunning the whole batch costs real money.
In this case, use `niceeval accept` to accept those old results explicitly. NiceEval does not let them through on your word alone. It verifies on its own that the change really does not affect the results. Only two kinds of change pass that check:
* The Experiment was renamed and nothing else changed. You can also rename directly with `niceeval exp rename`; see the [CLI reference](/docs/reference/cli).
* The image reference of the official `dockerSandbox` changed address, but both the old and new references spell out the same `@sha256:` digest and a complete platform. Tag-only images do not qualify, because the content a tag points to may have changed.
Any change to the model, flags, Judge, task, assertions, or scores is rejected with `accept-ineligible`. Those changes can really change the result, so you should rerun.
## Find acceptable old results in the run plan
First preview the Experiment you are about to run:
```sh theme={null}
pnpm exec niceeval exp checkout --dry
```
Old results that cannot be reused automatically because of a configuration change show their full Attempt locator, along with an `accept` command you can copy directly. The locators you put in the command are the entire scope of the acceptance. NiceEval does not accept other results from the same Run or the same Experiment along the way.
If all the old results come from the same Run and all of them can be accepted, `--dry` suggests `niceeval accept --run --dry` instead, so you can preview the whole Run at once.
## Check the old result before accepting it
Open the old Attempt in the terminal with its full locator:
```sh theme={null}
pnpm exec niceeval show @1K1P0VJAPVJ12
```
Check the verdict, assertions, usage, duration, and evidence, and confirm this is the result you want to keep. When you need to see the whole Run it belonged to, open that Run:
```sh theme={null}
pnpm exec niceeval show --run
pnpm exec niceeval view --run
```
## Accept the results you need
```sh theme={null}
pnpm exec niceeval accept @1K1P0VJAPVJ12
```
When the same Experiment has several results to accept, put all the locators in one command:
```sh theme={null}
pnpm exec niceeval accept @1K1P0VJAPVJ12 @1K1P0VJAPVJ13
```
To see what would happen without writing anything, add `--dry`.
NiceEval checks every input first. If any locator is invalid, ineligible, or conflicts with another input, the command writes no results at all. It never accepts half of them.
On success, NiceEval creates a new Run that references the original Attempts in the matching positions and records that you accepted them explicitly. The original Attempt, its Run, and its locator stay unchanged, and the result is not copied.
## Check the new Run
The success output shows the new Run ID. Open it to confirm the results:
```sh theme={null}
pnpm exec niceeval show --run
```
The denominator of this Run is still every Attempt the current Experiment plans to run. Positions you accepted show as `accepted`, and positions you did not accept show as `not-dispatched`. If you accepted only some of them, the report shows honestly that the results are incomplete.
Each `accept` creates a new Run. It does not modify or merge an earlier acceptance. To get a batch of results into the same Run, put them in the same command.
## Common failures
| Error | Next step |
| - | - |
| `malformed-locator` | Copy the full uppercase locator from the `--dry` or `show` output. Do not use an old UUID or a shortened prefix. |
| `locator-not-found` | Make sure you run the command in the same project that produced the locator, and copy it exactly from the `--dry` or `show` output. |
| `accept-ineligible` | Follow the hint to check the verdict, timeout, configuration change, or duration record. When you cannot prove the old result still holds, run it again. |
| `duplicate-accept-member` | Keep only one locator per position and remove duplicate or conflicting inputs. |
To choose between automatic reuse and a forced rerun, see [Rerun Only the Evals Affected by a Change](/docs/tutorials/rerun-and-cache).
# Let a Coding Agent Iterate on Results
Source: https://niceeval.com/docs/tutorials/agent-feedback-loop
Let a Coding Agent run evals, read results, decide whether to change the application or the assertions, and verify again, without you copying terminal output back and forth.
You ask a Coding Agent to fix a regression. It changes the code, you run the evals, you copy the failing output to it, it changes the code again, and you run again. After a few rounds, most of the time goes into moving terminal output around.
A better approach is to let the Agent run the loop itself:
1. Run the evals.
2. Get this run's Run ID from the receipt printed when the run ends.
3. Use the Run ID to view the results and find the failing runs.
4. Decide whether to change the application, the Adapter, or the eval.
5. After the change, run the same scope again and confirm the fix.
This page explains which command to use at each step and a few boundaries the Agent must respect.
## Have the Agent read the docs for the installed version
Commands and APIs the Agent writes from training memory may not match the NiceEval version you installed. Have it read the docs that ship with the npm package first.
NiceEval ships its Chinese docs in the npm package's `docs-site/zh/` directory and provides an `INDEX.md` at the package root for Coding Agents. The Agent reads `node_modules/niceeval/INDEX.md` first, then follows the index to the pages the current task needs.
`pnpm exec niceeval init` initializes the configuration and writes a managed guidance section into the project's `AGENTS.md`. If the project only has `CLAUDE.md`, it writes into `CLAUDE.md`; if neither exists, it creates `AGENTS.md`. After upgrading NiceEval, run `init` again to refresh that section.
The guidance separates two things:
* When writing code, the Agent can of course read project source such as `evals/` and `agents/`.
* When it needs to prove what happened in a run, it uses `niceeval show` in the terminal, `niceeval query` for stable JSON, and `niceeval view` for browser review. It does not scan raw files under `.niceeval/`, and it does not reconstruct a historical run from current source.
When these commands cannot show the needed information, have the Agent report "the CLI cannot show this" instead of working around it by reading files.
You can write the Agent's starting task like this:
```text theme={null}
Read node_modules/niceeval/INDEX.md first.
Then read the pages in the index that are relevant to the current task.
Run the Experiment with the command, and pick the Run to inspect from the final receipt.
```
## Run and save the receipt
Have the Agent run with `--json` and save the output:
```sh theme={null}
pnpm exec niceeval exp checkout --json | tee .niceeval-invocation.ndjson
```
Each line of `--json` describes the progress and diagnostics of this run. Exactly one final line is the receipt:
```json theme={null}
{
"type": "receipt",
"receipt": {
"invocationId": "01J8ZK3M6P4T7V9X2C5N8QW0RY",
"createdRunIds": ["01J9ZK3M6P4T7V9X2C5N8QW0RY"],
"publicationCutoff": "{...}",
"startedAt": "2026-08-09T10:00:00.000Z",
"completedAt": "2026-08-09T10:01:00.000Z",
"completion": "completed"
},
"summary": { "...": "..." }
}
```
The Agent trusts only this last line. `createdRunIds` lists every Run created this time; use it to view results afterwards. Do not parse `publicationCutoff`; pass it to later commands as is.
`completion` being `completed` only means the run ended. It does not mean the task is done, and the evals may not have passed. The receipt is also not a long-term result format; read results through `createdRunIds`.
## View results by Run ID
After the run ends, the Agent first opens the Run from the receipt in the terminal:
```sh theme={null}
pnpm exec niceeval show --run 01J9ZK3M6P4T7V9X2C5N8QW0RY
```
The output lists the Run's denominator, each eval's verdict, scores, result coverage, and the locator of each Attempt. To look at an Attempt in detail, copy its locator:
```sh theme={null}
pnpm exec niceeval show @
pnpm exec niceeval show @ --source
pnpm exec niceeval show @ --execution
```
`show @` shows the Attempt's verdict, assertion summary, whether each part is complete, and known limitations. `--source` shows the eval source used in this run and the result of each assertion. `--execution` shows a condensed execution trace; to expand one item, copy the ID next to it and run `--execution --expand `.
When a script, CI, or Agent needs to process results reliably, first find the available queries, then send a complete request:
```sh theme={null}
pnpm --silent exec niceeval query discover
pnpm --silent exec niceeval query explain --request request.json
pnpm --silent exec niceeval query run --request request.json
```
`query` returns JSON in a fixed format. When a person wants to look in the browser, use `pnpm exec niceeval view --run `; `view` does not accept locators.
## Decide the next step from the status
| Status | Next step |
| - | - |
| Attempt `failed` | Compare the outcomes the task allows, the actual artifacts, the evidence, and the assertion conditions, then decide whether to change the system under test, the Adapter, or the eval. |
| `errored` | Check the error text, Adapter, Sandbox, or credentials. |
| `not-recorded` | An Attempt planned for this run has no result. Check why it did not run. |
| `core-invalid` | Read the issue stated in the output. Do not treat it as missing collection. |
| `partial` | The result is incomplete. Read the denominator and the issue notes to see which Attempts were not counted. |
| `record-schema-unsupported` or old project state blocks the run | Keep the original project and its history, and follow the [old project recovery guide](/docs/troubleshooting/debugging#old-project-state-blocks-a-run) to rerun in a copy of the project. |
| Query result is `unsupported` or `failed` | This only affects views that depend on that input. Read the stated issue. |
When handling `failed`, do not change an artifact that already meets the task just to pass an overly narrow Match. If the artifact meets the task but the assertion rejects an outcome the task allows, the eval misjudged it and the eval should change. For the full attribution process, see the [debugging guide](/docs/troubleshooting/debugging).
## Run again after a change
Have the Agent verify one hypothesis at a time: change the program or the eval, run the same scope, then read the new receipt and Run.
By default, Attempts unaffected by the change that already have a conclusion from last time are reused. Add `--dry` before running to see which ones will be reused, which will rerun, and why. When every Attempt must really execute, add `--rerun all`:
```sh theme={null}
pnpm exec niceeval exp checkout --rerun all --json
```
For the rules on reusing and accepting old results, see [Rerun Only the Evals Affected by a Change](/docs/tutorials/rerun-and-cache).
A finished run only means this run handed off its results. It does not mean the user's task is done.
* When the task asks for a fix and you have authorization, fix what made the run error first, then reverify only the affected scope.
* When the task asks only for analysis, only do attribution. Do not change code or rerun.
* When a business failure is what the task expects, keep the result and report it honestly. Do not change the application or the eval just to turn it green.
## Saved results cannot be changed
Saved results are never modified. If the Agent wants a different result, it changes the program or the eval, runs again, and views the result through the new receipt's `createdRunIds`.
When you need to hand results to someone else, copy the project's [`.niceeval/record.sqlite`](/docs/explanation/record). Do not use terminal output or NDJSON as a sharing format.
# Onboard a Project from Scratch with a Coding Agent
Source: https://niceeval.com/docs/tutorials/agent-onboarding
The full onboarding flow for a Coding Agent: explore the project, confirm the path with the user, configure the Judge, write the Adapter, Experiment, and evals, and get the first experiment running.
This page is the execution guide for a Coding Agent integrating NiceEval into a project. Prerequisites: the `niceeval` dependency is installed in the project, `niceeval init` has run, and you reached this page from the bundled `INDEX.md`. Talk to the user in the user's language. Every API, field, and CLI behavior follows the bundled docs. Do not make them up from training memory.
Before writing evals, use [NiceEval Taste](/docs/tutorials/niceeval-taste) to decide the business criteria, sufficient evidence, and weights. At delivery, a reader should be able to explain the pass conditions and scoring by reading only the eval body.
## Step 1: Explore the project, then confirm with the user
This step decides the whole path that follows. **Read the code and find out first, list your findings for the user to check, and only ask about what you could not find.** Do not open with a list of questions, and do not assume without exploring. Find out:
1. **What kind of Agent this is**: read the README, `package.json` dependencies, routes, and agent loop code to determine what it is built with (AI SDK, LangGraph, OpenAI Agents SDK, Claude Agent SDK, a custom loop...) and what its core use case is (customer support? SQL? coding tasks?).
2. **How the frontend talks to the Agent**: HTTP, or gRPC / WebSocket? Is the protocol standard or custom: a standard protocol like AI SDK UI Message Stream or OpenAI Responses / Chat Completions, a passthrough of the SDK's native event stream, or user-defined JSON/SSE frames? This directly decides whether the Adapter is built in (zero mapping) or a handwritten `send` (you write the event mapping).
3. **How the real target runs**: confirm the build or install command, start command, readiness check, port, dependent services, credential sources, and how to stop it. The Adapter only connects to the target. It does not start a substitute implementation inside `send` or the eval. Credentials go only to the run boundary that needs them, never into eval source, the workspace, or the run handoff.
4. **Whether the backend already uses OTel**: search for OTel SDK initialization, AI SDK telemetry, or instrumentation such as LangSmith / OpenLLMetry / OpenInference. If it exists, Tier 2 costs almost nothing.
5. **Whether the user already has A/B tests or feature flags**: if the application already has variant switches, the Experiment's `flags` can pass straight through to them (a ready-made Tier 3 entry point).
6. **What the Judge uses**: semantic evaluation (`t.judge(material, definition)`) needs **a Judge model separate from the Agent under test**, over an OpenAI-compatible protocol. Ask which service the user has a key for and which Judge model they want. A missing key never passes silently: Judge assertions are recorded as `unavailable`. Without Judge configuration, write exact Matches first.
7. **Whether the Agent itself must go into a Sandbox**: if the system under test is a coding agent CLI, or a Skill/Plugin/Hook/MCP server written for a coding agent (which edits files and runs commands in an isolated workspace), it must use a Sandbox. It cannot be called with `send` directly like an HTTP service.
The default suggestion is `dockerSandbox({ source: { type: "image", image: "node:24-slim" } })`. If the task or existing configuration already specifies a Provider, keep it. Only ask when the Provider, Docker availability, or remote execution needs would still change the plan. The Provider is declared on the eval or the Experiment; there is no CLI flag, project-level default, or auto-detection. See [Choose a Sandbox Provider](/docs/tutorials/sandbox-providers).
If the task has not specified an integration tier, **introduce the integration tiers** to the user and give a recommendation (see [Integration tiers](/docs/explanation/tier)):
* **Tier 1 (send only)**: no change to the application. The full assertion set (text, Judge, multi-turn, tools, HITL) is available at this tier.
* **Tier 2 (send + OTel)**: the application also sends a copy of its OTel spans to NiceEval, and you view the call waterfall in `niceeval view`. With existing instrumentation (found in point 3), it takes zero changes.
* **Tier 3 (invasive changes + flags)**: expose internal application variants as `flags` for feature A/B testing. Existing A/B switches (found in point 4) are a ready-made entry point.
**The default recommendation is to get Tier 1 running first, then move to Tier 2.** Especially when point 3 found existing OTel, tell the user clearly that "moving to Tier 2 just sends one more copy of the spans; the cost is close to zero." Bring up Tier 3 only when the user explicitly wants to compare variants.
With explicit authorization, continue with the shape you found. Only confirm with the user when a missing decision would change the target, run boundary, credential boundary, or budget. Do not start writing the Adapter without reading the matching page:
| System under test | Read |
| - | - |
| An application built on the Vercel AI SDK (a `useChat` backend) | [Built-in Agent reference](/docs/reference/builtin-agents): the built-in `uiMessageStreamAgent` connects without changes and without handwritten event mapping |
| A coding agent CLI (claude-code / codex / bub, tasks that edit files) | [Evaluate an Agent in a Sandbox](/docs/tutorials/sandbox-agent): configure `sandbox`, default to `dockerSandbox({ source: { type: "image", image: "node:24-slim" } })`, and confirm the Provider with the user first |
| A Skill, Plugin, Hook, or MCP server written for Claude Code / Codex | [Evaluate Coding Agent extensions](/docs/examples/coding-agent-extensions): also runs in a Sandbox; confirm the Provider the same way |
| Other custom agent loops, LangGraph, OpenAI Agents SDK, deployed Agents | Start with [Connect Your Agent](/docs/tutorials/connect-your-agent); the full tutorial for a handwritten `send` is [Write send](/docs/tutorials/write-send) |
| Pure functions with no separate service | First read the "why not call it directly" section of [Connect Your Agent](/docs/tutorials/connect-your-agent), confirm with the user that this edge case is really what they want, then continue |
## Step 2: Configure the Judge
Configure the Judge service you asked about in step 1. In `niceeval.config.ts`, choose the protocol explicitly with a named Provider:
```ts theme={null}
import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";
export default defineConfig({
judgeRuntime: OpenAIProvider({
model: "gpt-5.4-mini", // Required: there is no built-in default model
// When using an OpenAI-compatible gateway, add these two explicitly:
// baseUrl: "https://api.deepseek.com/v1",
// apiKeyEnv: "DEEPSEEK_API_KEY",
}),
});
```
**When connecting a compatible gateway, `baseUrl` must be set explicitly.** If you set only the key and not `baseUrl`, NiceEval calls the official endpoint, the gateway credential is sent to OpenAI, and you get back "Incorrect API key provided". It looks like an expired key, but the endpoint is wrong.
Points to raise with the user:
* **A misconfiguration never pretends to pass.** When the model or key cannot be read, or the request times out, the Judge assertion is recorded as `unavailable` with the reason. When the gateway rejects the request or returns the wrong shape, it is recorded as `errored`. Results never show a score that could not be obtained as a failure or a zero.
* **Verify once after configuring.** Declare a rubric and anchors with `defineJudge`, then call `t.judge(material, definition).gate(0.8)`. After the run, check the score with `pnpm exec niceeval show --run `. See [Configure and Verify the Judge](/docs/tutorials/verify-judge) for the steps.
* **Without a user present (autonomous onboarding), do not just skip the Judge.** First check whether the environment already has a key for the chosen Provider. If it does, verify it. If not, write exact Matches first and state at the end that the Judge is not configured.
* The Judge model must be **separate from the Agent under test**, so a model does not grade its own output. Execution configuration is evaluated from the Experiment and the project config's `judgeRuntime`; for definitions and materials, see [Judge](/docs/explanation/judge).
## Step 3: Write the Adapter, Experiment, and evals
After reading the docs for the direction chosen in step 1, write these in order:
1. **Adapter** (`agents/*.ts` or the directory the user's project uses): implement `send` with `defineAgent`. Configuration goes through factory parameters; do not hard-code it or read `process.env`. The contract is in [Adapter](/docs/explanation/adapter), API signatures in the [Adapter reference](/docs/reference/define-agent), and event mapping in the [events reference](/docs/reference/events). Two common pitfalls: **choose the endpoint or mode that enters the system's core capability, not the one that is easiest to get running.** For example, if the platform under test has both a "plain LLM chat" mode and a "connect to the database and execute" mode, connecting the former only evaluates an underlying model proxy, not the product itself. **Declare `evidenceCoverage` honestly according to what you actually map.** If you only map the final text, do not use `completeEvidenceCoverage`. The declaration affects assertion completeness, and over-claiming is worse than being conservative.
2. **Experiment** (`experiments/*.ts`): reference the Adapter above and declare `model`, `flags`, `attempts`, and so on. For a model comparison, write two experiment files, each with its own `model`. `evals: (eval) => boolean` decides which evals each Experiment runs. The file path decides the Experiment ID, and also which Experiments one command can run as a batch.
3. **Evals** (`evals/*.eval.ts`): **first find out what this application does, then write an eval that follows its real function.** Read its README, routes, tool definitions, or system prompt, find its core use case (ask a real support question of a support bot, give a SQL agent a real query task), and use that for the first eval's input and assertions. Two kinds of input do not qualify: placeholder input unrelated to the application, such as "hello", and **meta questions to the system about itself**, such as "what are you / what can you do". Those are not tasks users give it. Still start from the smallest form: one input, `t.succeeded()` plus one content assertion on the expected answer. But the minimal form is only scaffolding to get things wired up, not the delivery standard. Before finishing, it must also meet two conditions:
* **Assertions must turn red when the system makes things up.** Do not assert on words already in the input (ask "what is X" and then assert the answer contains "X", and a system that repeats the question passes). Assert on substantive content unique to the expected answer: specific facts, structure (`hasSections()`), real links (`includesUrl()`), or a semantic verdict with `t.judge(material, definition)`.
* **At least one negative case.** Give the system an input it should not be able to answer (a nonexistent table, a topic retrieval cannot find), and assert that it clearly says it cannot find or do it, instead of inventing a plausible result. For an agent connected to real data or a retrieval source, this is the failure mode most worth testing first.
See [Write Evals](/docs/tutorials/authoring) for how to write them, [Eval Kinds and Assertions](/docs/tutorials/evaluation-kinds) for assertions and eval kinds, and the [defineEval reference](/docs/reference/define-eval) for signatures.
For how parameters flow from the Experiment to the Adapter, and how to split static configuration from per-turn dynamic values (factory parameters vs `ctx`), see [Connect Your Agent](/docs/tutorials/connect-your-agent).
Two architectural rules must not be broken when writing the Adapter:
* **Do not make an in-process direct call.** Even if the agent runtime and the evals live in the same codebase, the Adapter must use HTTP (or the matching transport). Do not replace `fetch` with a direct `import` of the function under test. See "why not call it directly" in [Connect Your Agent](/docs/tutorials/connect-your-agent).
* **The eval side does not manage the process under test.** Do not spawn the application or open another port. The user starts the application the way they normally do (`pnpm dev` and so on). When the Adapter cannot connect, report a clear error such as "start the application first". Do not start the service yourself.
## Step 4: Run and verify
First use `--dry` to confirm which evals run and how many times each, then run for real. The examples use pnpm; if the user's project uses another package manager, switch to its `exec`:
```sh theme={null}
pnpm exec niceeval exp models --dry # Confirm which evals each model configuration runs
pnpm exec niceeval exp models # Run both model configurations by path
pnpm exec niceeval show # View results in the terminal
```
When a person wants to compare results in the browser, run `pnpm exec niceeval view`.
Follow the [Coding Agent Feedback Loop](/docs/tutorials/agent-feedback-loop) to run, view, change, and rerun. When a machine needs to read the results, first run `pnpm --silent exec niceeval query discover`, then send a complete request. For ways to view results, see [View Run Results](/docs/tutorials/viewing-results).
Sort by final status first, then decide whether to change anything:
* `errored` means the run itself did not complete and produced no result that can be scored. Check the target service, endpoint, credentials, dependencies, timeouts, and the Adapter's error mapping, fix them, and run again.
* `failed` means the run completed but the business assertions were not met. First look at the assertions and evidence of the same Attempt. When the system under test really answered wrong, there is no need to turn it green.
* After a scored eval completes, still look at the score and each assertion. A completed or passed overview does not mean a full score.
Within existing authorization and budget, proactively fix what caused `errored`, and reverify only the affected scope. Read-only requests do not change or rerun anything. When real external conditions are missing, such as the target service, port, credentials, database, or model service, report the blocker clearly. Do not use a temporary stand-in to pretend the real target ran.
To look at an Attempt's details in the terminal, first copy its locator from the current overview, then use the same locator to view each part:
```sh theme={null}
pnpm exec niceeval show
pnpm exec niceeval show @
pnpm exec niceeval show @ --execution
pnpm exec niceeval show @ --source
```
Do not guess a locator from old output, and do not combine another run's source with this run's execution as evidence. Results can only prove what was recorded. `partial`, `unavailable`, and `not-recorded` mean the information is incomplete; they do not mean zero or failure.
## Step 5: Wrap up and tell the user what you did
Before summarizing, go through the wrap-up self-check. **If any item is not met, go back to step 3 and fill it in. Do not gloss over it in the summary.**
* [ ] The eval input is a core use case of the system under test, not a meta question about itself or a placeholder greeting
* [ ] Every content assertion turns red when the system repeats the question or makes things up (the asserted words are not already in the input)
* [ ] There is at least one negative case (an input it should not be able to answer, asserting that it clearly says it cannot)
* [ ] With a key, the Judge is configured and a Judge score was seen in `niceeval show` or `niceeval view`. Without a key, the summary says so
* [ ] The `model` / `flags` declared in the Experiment are actually consumed by the Adapter (no dead configuration nobody reads, and no invented model values the system does not have)
* [ ] The real target was started the project's own way and confirmed ready; credentials went only to the run boundary that needs them and were not written into eval source, the workspace, or the handoff
* [ ] At least one Attempt completed without being `errored`; business failures are kept honestly, and the same locator was used to check that scores, assertions, and evidence are complete
After it runs, summarize first, then talk about next steps. The summary should cover: which system under test was connected, where the Adapter, Experiment, and evals live, how to run `niceeval exp