> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Connect your Agent
> Write an Adapter, configure an Experiment, and run the first eval; then pass configuration and parameters from the Experiment to the Adapter and the app under test.
Connecting a subject under test to [NiceEval](https://niceeval.com/) requires an **Adapter**. `defineAgent` wraps a `send` function: it receives the input, drives the Agent, and returns the turn's result.
This tutorial first gets a minimal integration running with an Adapter, an Experiment, and an eval, then explains how parameters flow from the Experiment to the Adapter and on to the app under test. Event streams, multi-turn, HITL, and tracing are optional capabilities, with the corresponding tutorials listed at the end.
## Choose the integration path for your subject under test
Apps built with the Vercel AI SDK can connect to an existing HTTP interface using the built-in adapter.
Evaluating a standalone Agent like Claude Code / Codex / bub: use the built-in Sandbox Agent.
Your own Agent needs to [write Send](/docs/tutorials/write-send). If your app already has OTel instrumentation, you can [connect OTel](/docs/tutorials/connect-otel).
## Minimal integration example
Assume the project already has `npx niceeval init` run, so `niceeval.config.ts` and the `evals/` directory exist. Three files, one job each: **the Adapter connects to the system under test, the Experiment pins down the run configuration, and the eval defines the interaction and assertions.**
**1. Write the Adapter.** The minimal integration only fills `status` and `events`: put the Agent's reply into a `message` event.
```ts theme={null}
// agents/my-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
export default defineAgent({
name: "my-agent",
evidenceCoverage: completeEvidenceCoverage,
async send(input, ctx) {
// Example URL: swap in your own agent's real endpoint (HTTP, CLI, or SDK all work,
// as long as send can get the reply text)
const r = await fetch("http://localhost:3000/chat", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ message: input.text, model: ctx.model }), // ← experiment.model reaches here via ctx.model
signal: ctx.signal,
});
const body = await r.json();
return {
status: r.ok ? "completed" : "failed",
events: [{ type: "message", role: "assistant", text: body.reply }],
};
},
});
```
The minimal example starts with a hardcoded URL. When you need to pass the URL in per environment, use [how parameters flow in](#experiment-flags) below. The full `send` contract (every field of `TurnInput` / `AgentContext` / `Turn`) is in [Adapter](/docs/explanation/adapter).
Even when the Agent runtime and the evals live in the same codebase, still call the interface the way a frontend user would, and do not replace `fetch` with an in-process function call, because:
* **An in-process call is not the path your users take.** The HTTP layer, serialization, middleware, and streaming are all bypassed; a passing eval does not mean production behavior is correct.
* **An Adapter that calls in-process cannot be reused across deployment environments.** An HTTP Adapter connects to local, staging, or production just by swapping `baseUrl` (see the two Experiment files below); an in-process call is tied to the current codebase.
**2. Experiment**
```ts theme={null}
// experiments/my-agent.ts
import { defineExperiment } from "niceeval";
import myAgent from "../agents/my-agent.ts";
export default defineExperiment({
description: "my-agent baseline",
agent: myAgent,
model: "gpt-4o",
attempts: 1,
});
```
**3. Eval**
```ts theme={null}
// evals/refund-policy.eval.ts
import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";
export default defineEval({
description: "Refund policy Q&A",
async test(t) {
const turn = await t.send("What is your refund policy?");
await turn.succeeded().stopOnFailure();
t.check(t.reply, includes("30 days"));
t.judge.autoevals.closedQA("Does the answer explain the refund window?").atLeast(0.7);
},
});
```
```bash theme={null}
npx niceeval exp my-agent # run every eval under this experiment
npx niceeval exp my-agent refund # only run those whose ID starts with refund
npx niceeval view --experiment my-agent # inspect results in the local viewer
```
**What a successful run looks like**: the terminal shows a live dashboard where the completed and queued counts update in place; failures, errors, and warnings stay in the output. When the run finishes, it prints the summary and receipt. You can use a receipt Run ID with `npx niceeval view --run ` to inspect each eval's per-turn inputs, events, and assertion details.
If it doesn't run, triage by where the error shows up, into three buckets:
* **`fetch` throws directly** (connection refused, etc.): the app is not running, or the URL in `send` is wrong — first send the same request to that endpoint with `curl` to confirm.
* **`t.succeeded()` fails and the turn's verdict is failed**: the request went out, but the Turn the app returned is `failed`. Map the protocol's failures onto `Turn.status` or a standard `error` event; keep any extra bounded context you need with `ctx.diagnostic(...)`, not by printing the full response body.
* **Only content assertions fail**: the integration itself already works — compare the actual value of `t.reply` in `view`, then adjust the assertion or the app.
Once these steps are done, text assertions and Judge assessments both work. Tool, multi-turn, and approval-flow assertions require adding the optional capabilities listed at the end.
## Experiment flags
Configuration belongs to exactly two channels; keep them separate and the integration stays untangled:
1. **Static configuration goes through the Adapter factory.** Environment-level configuration such as URL, auth, and protocol details is written as factory parameters in the Experiment file. The `agent` field of `defineExperiment` receives an **already-configured instance**.
2. **Per-turn dynamic values go through `ctx`.** The `model` and `flags` declared by the experiment are handed to `send` verbatim via `ctx` on every turn; the Adapter does not interpret their meaning, it only forwards them to the app with the request.
Turn step 1's `my-agent` from an instance with a hardcoded URL into a factory that receives configuration. Change the default export to a function that returns `defineAgent(...)`, and have `send` read the factory parameters. The `model` and `flags` declared by the Experiment arrive via `ctx` on every turn, and `send` forwards them with the request:
```ts theme={null}
// agents/my-agent.ts — factory: static configuration goes into the closure
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";
export function myAgent(options: { baseUrl: string }): Agent {
return defineAgent({
name: "my-agent",
evidenceCoverage: completeEvidenceCoverage,
async send(input, ctx) {
const r = await fetch(`${options.baseUrl}/chat`, { // ← factory option: where to connect
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
message: input.text,
model: ctx.model, // ← experiment.model: forward it if the app's interface accepts a model choice
flags: ctx.flags, // ← experiment.flags: hand it to the app as-is; the app switches on it
}),
signal: ctx.signal, // ← runner timeout and cancellation: attach to every request
});
const body = await r.json();
return {
status: r.ok ? "completed" : "failed",
events: [{ type: "message", role: "assistant", text: body.reply }],
};
},
});
}
```
Auth headers, protocol switches, and the like are static configuration too — add them to `options` the same way. The experiment side changes in two small places: a named import of the factory, and the `agent` field goes from referencing an instance to calling the factory:
```ts theme={null}
// experiments/my-agent.ts — where parameters are declared
import { defineExperiment } from "niceeval";
import { myAgent } from "../agents/my-agent.ts";
export default defineExperiment({
description: "my-agent baseline",
agent: myAgent({ baseUrl: "http://localhost:3000" }), // ← static configuration: passed at instantiation
model: "gpt-5.4", // ← dynamic value: reaches send via ctx.model
flags: { promptVariant: "concise" }, // ← dynamic value: reaches send via ctx.flags
});
```
The `ctx` fields a turn may use, and how to consume them:
| `ctx` field | Source | How the Adapter uses it |
| ------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `signal` | The runner (timeout and cancellation) | Attach it to every outgoing request |
| `model` | The experiment's `model` | Forward it with the request if the app's interface accepts a model choice; ignore it otherwise |
| `flags` | The experiment's `flags` | Forward as-is (request body or header both work); the app switches variants on it |
| `telemetry` | Present when OTel integration is configured | Touch only `headers`: a fresh W3C `traceparent` each turn, spread into the request headers. The receiving endpoint is the same on every run — pin it in `defineConfig` and point the app at it on startup; it is not passed from send — see [OTel Integration](/docs/tutorials/connect-otel) |
| `session` | The runner (one per session line) | Session continuation and HITL held state live here: `id` / `capture()`, plus typed-slot `get()` / `set()` / `take()` — see [Write Send](/docs/tutorials/write-send) |
| `progress(update)` | The runner (bound to the current `agent.run`) | Report the short-lived status of a turn or tool; the human dashboard may show it, but it is not saved with the results |
| `diagnostic(input)` | The runner (bound to the current `agent.run`) | Save warnings/errors such as protocol degradation or an incomplete response; inspect them through a planned Attempt page |
### Progress, diagnostics, and fatal errors in the Adapter
```ts theme={null}
async send(input, ctx) {
ctx.progress({ message: "Waiting for the upstream model" });
const response = await callAgent(input, { signal: ctx.signal });
if (response.eventsIncomplete) {
ctx.diagnostic({
code: "incomplete-event-stream",
level: "warning",
message: "The upstream response is missing tool result events",
data: { requestId: response.requestId },
dedupeKey: `incomplete-event-stream:${response.requestId}`,
});
}
return toTurn(response);
}
```
`progress` is short-lived status that later updates overwrite; `diagnostic` is a bounded record you can still review after the run ends. Neither can specify a phase or output stream, and neither automatically changes `Turn.status` or the Attempt verdict. Infrastructure errors — a failed connection, parsing that cannot continue — should throw an exception; a normal failure of the Agent under test is expressed with `Turn.status: "failed"`.
The terminal shows only a one-layer error summary and Attempt identity. The full code, message, cause, stack, and diagnostics live in Attempt-owned channels; inspect them with `niceeval show --run --page attempt-`. An OTel trace only adds call relationships and timing — it is not a prerequisite for recording errors.
To evaluate local and production separately, create two Experiment files and pass in different factory parameters:
```ts theme={null}
// experiments/local.ts
export default defineExperiment({
agent: myAgent({ baseUrl: "http://localhost:3000" }),
model: "gpt-5.4",
});
// experiments/prod.ts
export default defineExperiment({
agent: myAgent({ baseUrl: "https://api.example.com" }),
model: "gpt-5.4",
});
```
```bash theme={null}
npx niceeval exp local
npx niceeval exp prod
```
Do not put URLs into CLI positional arguments — the positional arguments after the experiment name are only for filtering eval IDs. For the full experiment fields (`attempts`, `budget`, concurrency, `sandbox`), see [Write Experiments](/docs/tutorials/write-experiment).
## Add optional capabilities
Once the minimal integration is done, extend the Adapter as needed. Existing evals don't need to change:
| Target capability | Adapter change | Tutorial |
| ------------------------------------------------------------------ | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| Tool assertions (`calledTool` / `toolOrder` / negative assertions) | Map the app's response into the standard event stream | [Write Send](/docs/tutorials/write-send), [Events reference](/docs/reference/events) |
| Multi-turn conversations, `t.newSession()` isolation | Connect `ctx.session`: a typed slot or `id` + `capture()` | [Write Send](/docs/tutorials/write-send) |
| Approval flows (HITL, human-in-the-loop) | Return `waiting` + `input.requested` to pause the turn, resume on the answer turn | [HITL](/docs/explanation/hitl) |
| The call waterfall in `niceeval view` | The app sends OTel spans to NiceEval (does not affect assertions) | [OTel Integration](/docs/tutorials/connect-otel) |
| Feature A/B comparison | The app exposes variants as configuration switchable via `flags` | [Tier](/docs/explanation/tier), [Write Experiments](/docs/tutorials/write-experiment) |
For the integration tier and scope each capability corresponds to, see [Tier](/docs/explanation/tier).
## Reference implementations
[`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) provides five runnable non-intrusive integration examples (ai-sdk-v7, claude-sdk, codex-sdk, pi-sdk, langgraph), covering event-stream assertions, multi-turn isolation, HITL approve/reject, and the trace waterfall. Hand-writing `send` only requires implementing the transport and the mapping table; session continuation and HITL pause/resume are provided by `ctx.session`, and frame-by-frame driving can use the built-in implementation — see [Built-in agent capabilities](/docs/reference/builtin-agents).
## Related reading
* [Official adapters overview](/docs/reference/official-adapters) — Sandbox and non-Sandbox Adapters and their configuration options.
* [Write Send](/docs/tutorials/write-send) — the complete tutorial for hand-writing an Adapter: seven progressive steps, from sending one message to HITL, OTel, and flags.
* [Adapter](/docs/explanation/adapter) — the `send` contract: `TurnInput` / `AgentContext` / `Turn`, field by field.
* [OTel Integration](/docs/tutorials/connect-otel) — send the app's spans to [NiceEval](https://niceeval.com/) too, in exchange for the call waterfall in `niceeval view`.
* [Tier](/docs/explanation/tier) — the requirements and capabilities of the three integration tiers.
* [Write Experiments](/docs/tutorials/write-experiment) — the full `defineExperiment` fields.