Connect your Agent

Connecting a subject under test to NiceEval requires an Adapter. defineAgent wraps a send function: it receives the input, drives the Agent, and returns the turn’s result. This tutorial first gets a minimal integration running with an Adapter, an Experiment, and an eval, then explains how parameters flow from the Experiment to the Adapter and on to the app under test. Event streams, multi-turn, HITL, and tracing are optional capabilities, with the corresponding tutorials listed at the end.

Choose the integration path for your subject under test

AI SDK app

Apps built with the Vercel AI SDK can connect to an existing HTTP interface using the built-in adapter.

Agent

Evaluating a standalone Agent like Claude Code / Codex / bub: use the built-in Sandbox Agent.

Other AI Agent

Your own Agent needs to write Send. If your app already has OTel instrumentation, you can connect OTel.

Minimal integration example

Assume the project already has npx niceeval init run, so niceeval.config.ts and the evals/ directory exist. Three files, one job each: the Adapter connects to the system under test, the Experiment pins down the run configuration, and the eval defines the interaction and assertions. 1. Write the Adapter. The minimal integration only fills status and events: put the Agent’s reply into a message event.

// agents/my-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

export default defineAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    // Example URL: swap in your own agent's real endpoint (HTTP, CLI, or SDK all work,
    // as long as send can get the reply text)
    const r = await fetch("http://localhost:3000/chat", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ message: input.text, model: ctx.model }),  // ← experiment.model reaches here via ctx.model
      signal: ctx.signal,
    });
    const body = await r.json();
    return {
      status: r.ok ? "completed" : "failed",
      events: [{ type: "message", role: "assistant", text: body.reply }],
    };
  },
});

The minimal example starts with a hardcoded URL. When you need to pass the URL in per environment, use how parameters flow in below. The full send contract (every field of TurnInput / AgentContext / Turn) is in Adapter. Even when the Agent runtime and the evals live in the same codebase, still call the interface the way a frontend user would, and do not replace fetch with an in-process function call, because:

An in-process call is not the path your users take. The HTTP layer, serialization, middleware, and streaming are all bypassed; a passing eval does not mean production behavior is correct.
An Adapter that calls in-process cannot be reused across deployment environments. An HTTP Adapter connects to local, staging, or production just by swapping baseUrl (see the two Experiment files below); an in-process call is tied to the current codebase.

2. Experiment

// experiments/my-agent.ts
import { defineExperiment } from "niceeval";
import myAgent from "../agents/my-agent.ts";

export default defineExperiment({
  description: "my-agent baseline",
  agent: myAgent,
  model: "gpt-4o",
  attempts: 1,
});

3. Eval

// evals/refund-policy.eval.ts
import { defineEval } from "niceeval";
import { includes } from "niceeval/expect";

export default defineEval({
  description: "Refund policy Q&A",
  async test(t) {
    const turn = await t.send("What is your refund policy?");
    await turn.succeeded().stopOnFailure();
    t.check(t.reply, includes("30 days"));
    t.judge.autoevals.closedQA("Does the answer explain the refund window?").atLeast(0.7);
  },
});

npx niceeval exp my-agent        # run every eval under this experiment
npx niceeval exp my-agent refund # only run those whose ID starts with refund
npx niceeval view --experiment my-agent  # inspect results in the local viewer

What a successful run looks like: the terminal shows a live dashboard where the completed and queued counts update in place; failures, errors, and warnings stay in the output. When the run finishes, it prints the summary and receipt. You can use a receipt Run ID with npx niceeval view --run <runId> to inspect each eval’s per-turn inputs, events, and assertion details. If it doesn’t run, triage by where the error shows up, into three buckets:

fetch throws directly (connection refused, etc.): the app is not running, or the URL in send is wrong — first send the same request to that endpoint with curl to confirm.
t.succeeded() fails and the turn’s verdict is failed: the request went out, but the Turn the app returned is failed. Map the protocol’s failures onto Turn.status or a standard error event; keep any extra bounded context you need with ctx.diagnostic(...), not by printing the full response body.
Only content assertions fail: the integration itself already works — compare the actual value of t.reply in view, then adjust the assertion or the app.

Once these steps are done, text assertions and Judge assessments both work. Tool, multi-turn, and approval-flow assertions require adding the optional capabilities listed at the end.

Experiment flags

Configuration belongs to exactly two channels; keep them separate and the integration stays untangled:

Static configuration goes through the Adapter factory. Environment-level configuration such as URL, auth, and protocol details is written as factory parameters in the Experiment file. The agent field of defineExperiment receives an already-configured instance.
Per-turn dynamic values go through ctx. The model and flags declared by the experiment are handed to send verbatim via ctx on every turn; the Adapter does not interpret their meaning, it only forwards them to the app with the request.

Turn step 1’s my-agent from an instance with a hardcoded URL into a factory that receives configuration. Change the default export to a function that returns defineAgent(...), and have send read the factory parameters. The model and flags declared by the Experiment arrive via ctx on every turn, and send forwards them with the request:

// agents/my-agent.ts — factory: static configuration goes into the closure
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

export function myAgent(options: { baseUrl: string }): Agent {
  return defineAgent({
    name: "my-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      const r = await fetch(`${options.baseUrl}/chat`, {   // ← factory option: where to connect
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({
          message: input.text,
          model: ctx.model,     // ← experiment.model: forward it if the app's interface accepts a model choice
          flags: ctx.flags,   // ← experiment.flags: hand it to the app as-is; the app switches on it
        }),
        signal: ctx.signal,     // ← runner timeout and cancellation: attach to every request
      });
      const body = await r.json();
      return {
        status: r.ok ? "completed" : "failed",
        events: [{ type: "message", role: "assistant", text: body.reply }],
      };
    },
  });
}

Auth headers, protocol switches, and the like are static configuration too — add them to options the same way. The experiment side changes in two small places: a named import of the factory, and the agent field goes from referencing an instance to calling the factory:

// experiments/my-agent.ts — where parameters are declared
import { defineExperiment } from "niceeval";
import { myAgent } from "../agents/my-agent.ts";

export default defineExperiment({
  description: "my-agent baseline",
  agent: myAgent({ baseUrl: "http://localhost:3000" }),  // ← static configuration: passed at instantiation
  model: "gpt-5.4",                                    // ← dynamic value: reaches send via ctx.model
  flags: { promptVariant: "concise" },                // ← dynamic value: reaches send via ctx.flags
});

The ctx fields a turn may use, and how to consume them:

`ctx` field	Source	How the Adapter uses it
`signal`	The runner (timeout and cancellation)	Attach it to every outgoing request
`model`	The experiment’s `model`	Forward it with the request if the app’s interface accepts a model choice; ignore it otherwise
`flags`	The experiment’s `flags`	Forward as-is (request body or header both work); the app switches variants on it
`telemetry`	Present when OTel integration is configured	Touch only `headers`: a fresh W3C `traceparent` each turn, spread into the request headers. The receiving endpoint is the same on every run — pin it in `defineConfig` and point the app at it on startup; it is not passed from send — see OTel Integration
`session`	The runner (one per session line)	Session continuation and HITL held state live here: `id` / `capture()`, plus typed-slot `get()` / `set()` / `take()` — see Write Send
`progress(update)`	The runner (bound to the current `agent.run`)	Report the short-lived status of a turn or tool; the human dashboard may show it, but it is not saved with the results
`diagnostic(input)`	The runner (bound to the current `agent.run`)	Save warnings/errors such as protocol degradation or an incomplete response; inspect them through a planned Attempt page

Progress, diagnostics, and fatal errors in the Adapter

async send(input, ctx) {
  ctx.progress({ message: "Waiting for the upstream model" });
  const response = await callAgent(input, { signal: ctx.signal });

  if (response.eventsIncomplete) {
    ctx.diagnostic({
      code: "incomplete-event-stream",
      level: "warning",
      message: "The upstream response is missing tool result events",
      data: { requestId: response.requestId },
      dedupeKey: `incomplete-event-stream:${response.requestId}`,
    });
  }

  return toTurn(response);
}

progress is short-lived status that later updates overwrite; diagnostic is a bounded record you can still review after the run ends. Neither can specify a phase or output stream, and neither automatically changes Turn.status or the Attempt verdict. Infrastructure errors — a failed connection, parsing that cannot continue — should throw an exception; a normal failure of the Agent under test is expressed with Turn.status: "failed". The terminal shows only a one-layer error summary and Attempt identity. The full code, message, cause, stack, and diagnostics live in Attempt-owned channels; inspect them with niceeval show --run <runId> --page attempt-<attemptId>. An OTel trace only adds call relationships and timing — it is not a prerequisite for recording errors. To evaluate local and production separately, create two Experiment files and pass in different factory parameters:

// experiments/local.ts
export default defineExperiment({
  agent: myAgent({ baseUrl: "http://localhost:3000" }),
  model: "gpt-5.4",
});

// experiments/prod.ts
export default defineExperiment({
  agent: myAgent({ baseUrl: "https://api.example.com" }),
  model: "gpt-5.4",
});

npx niceeval exp local
npx niceeval exp prod

Do not put URLs into CLI positional arguments — the positional arguments after the experiment name are only for filtering eval IDs. For the full experiment fields (attempts, budget, concurrency, sandbox), see Write Experiments.

Add optional capabilities

Once the minimal integration is done, extend the Adapter as needed. Existing evals don’t need to change:

Target capability	Adapter change	Tutorial
Tool assertions (`calledTool` / `toolOrder` / negative assertions)	Map the app’s response into the standard event stream	Write Send, Events reference
Multi-turn conversations, `t.newSession()` isolation	Connect `ctx.session`: a typed slot or `id` + `capture()`	Write Send
Approval flows (HITL, human-in-the-loop)	Return `waiting` + `input.requested` to pause the turn, resume on the answer turn	HITL
The call waterfall in `niceeval view`	The app sends OTel spans to NiceEval (does not affect assertions)	OTel Integration
Feature A/B comparison	The app exposes variants as configuration switchable via `flags`	Tier, Write Experiments

For the integration tier and scope each capability corresponds to, see Tier.

Reference implementations

examples/zh/tier1 provides five runnable non-intrusive integration examples (ai-sdk-v7, claude-sdk, codex-sdk, pi-sdk, langgraph), covering event-stream assertions, multi-turn isolation, HITL approve/reject, and the trace waterfall. Hand-writing send only requires implementing the transport and the mapping table; session continuation and HITL pause/resume are provided by ctx.session, and frame-by-frame driving can use the built-in implementation — see Built-in agent capabilities.

Official adapters overview — Sandbox and non-Sandbox Adapters and their configuration options.
Write Send — the complete tutorial for hand-writing an Adapter: seven progressive steps, from sending one message to HITL, OTel, and flags.
Adapter — the send contract: TurnInput / AgentContext / Turn, field by field.
OTel Integration — send the app’s spans to NiceEval too, in exchange for the call waterfall in niceeval view.
Tier — the requirements and capabilities of the three integration tiers.
Write Experiments — the full defineExperiment fields.

Get Started