Connect Your Agent

Connecting a system under test to NiceEval requires an Adapter. defineAgent constructs an Adapter that directly calls a function, SDK, or service endpoint. Its send receives input, drives the Agent, and returns the result of that turn. This tutorial first gets a minimal integration working with an Adapter, Experiment, and eval, then explains how parameters travel from the Experiment to the Adapter and application under test. Event streams, multi-turn interaction, HITL, and tracing are optional capabilities; the matching tutorials are listed at the end.

Choose an integration path by system under test

AI SDK App

A Vercel AI SDK app can use the built-in Adapter to connect an existing HTTP interface.

Agent

To evaluate a standalone Agent such as Claude Code, Codex, or bub, use a built-in Sandbox Agent.

Other AI Agent

Your own Agent needs Write Send. If the app already has OTel instrumentation, connect OTel.

Minimal integration example

The following steps assume the project has already run npx niceeval init and contains niceeval.config.ts and an evals/ directory. Each of the three files has one responsibility: the Adapter connects the system under test, the Experiment fixes run configuration, and the eval defines interactions and Assertions. 1. Write the Adapter. The minimal integration fills only status and events, putting the Agent reply in one message event.

// agents/my-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

export default defineAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    // Example URL: replace it with the real endpoint for your Agent. HTTP, CLI, and SDK all work
    // as long as send can obtain reply text.
    const r = await fetch("http://localhost:3000/chat", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ message: input.text, model: ctx.model }),  // ← experiment.model reaches here through ctx.model
      signal: ctx.signal,
    });
    const body = await r.json();
    return {
      status: r.ok ? "completed" : "failed",
      events: [{ type: "message", role: "assistant", text: body.reply }],
    };
  },
});

The minimal example starts with a fixed URL. To pass a URL by environment, use Experiment Flags below. See Adapter for the complete send contract—every field in TurnInput, AgentContext, and Turn. When the system under test is an independent HTTP service, call the HTTP interface that users actually use even if the service source and eval live in the same repository. Do not replace this example’s fetch with an in-process function call, because:

An in-process call is not the path users take. It bypasses HTTP, serialization, middleware, and streaming. A passing eval does not prove production behavior is correct.
The Adapter cannot be reused in another deployment environment. An HTTP Adapter connects to local, staging, or production by changing baseUrl (see the two Experiment files below). An in-process call depends on the current repository.

When the Agent offers only an SDK, function, or CLI, call the matching entry point directly in send. Keep the same input, cancellation signal, and returned Turn shape. Do not create a service that does not exist merely to fit the HTTP example. 2. Experiment

// experiments/my-agent.ts
import { defineExperiment } from "niceeval";
import myAgent from "../agents/my-agent.ts";

export default defineExperiment({
  description: "my-agent baseline",
  agent: myAgent,
  model: "gpt-4o",
  attempts: 1,
});

3. Eval

// evals/refund-policy.eval.ts
import { defineEval, defineJudge } from "niceeval";
import { includes } from "niceeval/expect";

const refundPolicyQuality = defineJudge({
  name: "refund-policy-quality",
  rubric: "Assess whether the answer clearly explains the refund period based on the question.",
});

export default defineEval({
  judge: refundPolicyQuality,
  description: "Refund-policy Q&A",
  async test(t) {
    const turn = await t.send("What is your refund policy?");
    turn.succeeded().label("Request succeeds");
    t.check(turn.message, includes("30")).label("Explains the refund window");
    t.judge({ question: turn.input, answer: turn.message }, refundPolicyQuality)
      .gate(0.7)
      .label("Refund-policy quality");
  },
});

The assertion line is important: do not mechanically write includes("30 days"). Wording varies on every reply, and one exact phrase makes an open-ended answer fail even if the system answered correctly. For open wording, use a regular expression that accepts equivalent phrasing, a shape Assertion such as includesUrl() / hasSections(), or a Judge Match registered through check. Use exact Assertions only for deterministic results such as a concrete value or ID. See Evaluation Kinds and Assertions for the complete trade-off.

npx niceeval exp my-agent        # Run every eval under this Experiment
npx niceeval exp my-agent refund # Run only IDs that start with refund
pnpm exec niceeval view --run <run-id>  # Inspect a Run in the local viewer

Verify the result: the terminal shows a live dashboard whose completed and queued counts update in place. Failures, errors, and warnings remain in output. At the end, it prints a summary and receipt. Use the receipt’s Run ID with pnpm exec niceeval view --run <runId> to inspect every eval’s per-turn input, events, and scoring detail. When it does not run, triage by where the error appears:

fetch throws directly (such as connection refused): the application is not running, or the URL in send is wrong. First use curl to send the same request to that interface and confirm it.
t.succeeded() fails and the turn’s Verdict is failed: the request was sent, but the application returned a failed Turn. Map protocol failures to Turn.status or a standard error event. Preserve bounded extra context with ctx.diagnostic(...), not by printing the full response body.
Only a content Assertion fails: the integration itself works. Compare the actual t.reply value in view, then adjust the Assertion or application.

After these steps, text Assertions and Judge scoring are available. Tool, multi-turn, and approval-flow Assertions require the optional capabilities below.

Experiment Flags

Configuration has exactly two channels. Keep them separate and the integration stays clear:

Static configuration goes through the Adapter factory. Write environment-level configuration such as URL, authentication, and protocol details as factory parameters in the Experiment file. The agent field in defineExperiment receives an already-configured instance.
Dynamic values per turn go through ctx. model and flags declared by the Experiment travel unchanged to send through ctx on every turn. The Adapter does not interpret their meaning; it only forwards them with the request to the application.

Turn the fixed-URL my-agent from step 1 into a factory that accepts configuration. Change the default export to a function returning defineAgent(...), and read factory parameters in send. The Experiment’s model and flags reach every turn through ctx, then send forwards them with the request:

// agents/my-agent.ts — Factory: static configuration enters the closure
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

export function myAgent(options: { baseUrl: string }): Agent {
  return defineAgent({
    name: "my-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      const r = await fetch(`${options.baseUrl}/chat`, {   // ← Factory configuration: where to connect
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({
          message: input.text,
          model: ctx.model,     // ← experiment.model: forward when the application accepts model selection
          flags: ctx.flags,   // ← experiment.flags: give them to the application unchanged; it switches variants by parameter
        }),
        signal: ctx.signal,     // ← Runner timeout and cancellation: attach it to every request
      });
      const body = await r.json();
      return {
        status: r.ok ? "completed" : "failed",
        events: [{ type: "message", role: "assistant", text: body.reply }],
      };
    },
  });
}

Authentication headers and protocol switches are static configuration too, so add them to options the same way. The Experiment needs two small changes: import the factory by name and change agent from “refer to an instance” to “call the factory”:

// experiments/my-agent.ts — Parameter declaration
import { defineExperiment } from "niceeval";
import { myAgent } from "../agents/my-agent.ts";

export default defineExperiment({
  description: "my-agent baseline",
  agent: myAgent({ baseUrl: "http://localhost:3000" }),  // ← Static configuration: pass at instantiation
  model: "gpt-5.4",                                    // ← Dynamic value: reaches send through ctx.model
  flags: { promptVariant: "concise" },                // ← Dynamic value: reaches send through ctx.flags
});

The ctx fields a turn may use and how the Adapter consumes them:

`ctx` field	Source	Adapter use
`signal`	Runner, for timeout and cancellation	Attach it to every outbound request
`model`	The Experiment’s `model`	Forward it when the application accepts model selection; otherwise ignore it
`flags`	The Experiment’s `flags`	Forward unchanged in a request body or header; the application switches variants by parameter
`telemetry`	Present when OTel integration is configured	Touch only `headers`: every turn has a fresh W3C `traceparent`, which you spread into request headers. The receiver endpoint stays the same for each run; pin it in `defineConfig` and point the application to it at startup. Do not pass it through `send`; see OTel Integration.
`session`	Runner, one per conversation line	It holds accessors for session continuation and paused HITL state: `id` / `capture()` and typed-slot `get()` / `set()` / `take()`. See Write Send.
`progress(update)`	Runner, bound to the current `agent.run`	Report short-lived Turn/tool status. A human dashboard can show it, but results do not persist it.
`diagnostic(input)`	Runner, bound to the current `agent.run`	Save warnings/errors such as protocol degradation and incomplete responses. Review them by drilling down through a locator.

Progress, diagnostics, and fatal errors in an Adapter

async send(input, ctx) {
  ctx.progress({ message: "Waiting for the upstream model" });
  const response = await callAgent(input, { signal: ctx.signal });

  if (response.eventsIncomplete) {
    ctx.diagnostic({
      code: "incomplete-event-stream",
      level: "warning",
      message: "The upstream response is missing tool result events",
      data: { requestId: response.requestId },
      dedupeKey: `incomplete-event-stream:${response.requestId}`,
    });
  }

  return toTurn(response);
}

progress is short-lived status that later updates replace. diagnostic is a bounded record available after the run ends. Neither can choose a phase or output stream, and neither changes Turn.status or the Attempt Verdict automatically. Throw infrastructure errors that cannot continue, such as connection failure or unparseable input. Express a normal failure received from the Agent under test with Turn.status: "failed". The terminal displays only a one-layer error summary and Attempt identity. Full code, message, cause, stack, and diagnostics live in Attempt-owned channels; inspect them with niceeval show @<attempt-locator> or a fixed query operation. An OTel trace only adds call relationships and timing; it is not a prerequisite for error data. To evaluate local and production environments separately, create two Experiment files with different factory parameters:

// experiments/local.ts
export default defineExperiment({
  agent: myAgent({ baseUrl: "http://localhost:3000" }),
  model: "gpt-5.4",
});

// experiments/prod.ts
export default defineExperiment({
  agent: myAgent({ baseUrl: "https://api.example.com" }),
  model: "gpt-5.4",
});

npx niceeval exp local
npx niceeval exp prod

Do not put a URL in CLI positional arguments. Positionals after the Experiment name filter eval IDs only. See Write Experiments for the complete Experiment fields: attempts, budget, concurrency, and sandbox.

Add optional capabilities

After minimal integration, extend the Adapter as needed. Existing evals do not need to change:

Capability	Adapter change	Tutorial
Tool Assertions (`calledTool` / `toolOrder` / negative Assertions)	Map the application response to the standard event stream	Write Send, Event Stream Reference
Multi-turn conversation and `t.newSession()` isolation	Connect `ctx.session`: typed slots or `id` + `capture()`	Write Send
Approval flows (HITL, human-in-the-loop)	Return `waiting` + `input.requested` to pause a turn, then continue on the response turn	HITL
The call waterfall in `niceeval view`	The application sends OTel spans to NiceEval; Assertions stay unchanged	OTel Integration
Feature A/B comparison	The application exposes variants as configuration switchable by `flags`	Tier, Write Experiments

See Tier for the integration tier and scope corresponding to each capability.

Reference implementations

examples/zh/tier1 provides five runnable non-intrusive integration examples—ai-sdk-v7, claude-sdk, codex-sdk, pi-sdk, and langgraph. They cover event-stream Assertions, multi-turn isolation, HITL approval and rejection, and the trace waterfall. Hand-writing send requires only a transport and mapping table. ctx.session provides session continuation and HITL pause/resume; for frame-by-frame driving, use a built-in implementation. See Built-in Agent Capabilities.

Official Adapter Overview: Sandbox and non-Sandbox Adapters and their options.
Write Send: the complete seven-step hand-written Adapter tutorial, from sending one message to HITL, OTel, and flags.
Adapter: the send contract, field by field for TurnInput, AgentContext, and Turn.
OTel Integration: send application spans to NiceEval too in exchange for the niceeval view call waterfall.
Tier: requirements and capabilities for the three integration tiers.
Write Experiments: the complete defineExperiment fields.

Get Started