Connect Your Agent: Choose an Integration and Pass Configuration

The Adapter in the quickstart hard-codes a localhost address and only handles single-turn conversations. Real projects soon run into these problems: local and staging addresses differ, you want to compare two models, and you want to check tool calls. This page shows how to turn that minimal Adapter into an integration you can keep using.

Choose an integration first

AI SDK applications

For applications built on the Vercel AI SDK, use a built-in adapter to connect to the existing HTTP interface directly, with no handwritten mapping.

Coding Agents such as Claude Code and Codex

Use a built-in Sandbox Agent and run the Agent in an isolated environment.

Other AI applications

Handwrite an Adapter’s send function. If the application already has OTel instrumentation, you can also connect OTel to see the call waterfall.

When the system under test is not a conversational Agent but an application that offers business operations such as posting or generating images, use defineAdapter; see Evaluate a Custom Application.

Connect to the real entry point

When the system under test is a separate HTTP service, the Adapter should call the interface actually used in production, even if the service code and the evals live in the same repository. Do not switch to calling a function in-process for convenience:

The HTTP layer, serialization, middleware, and streaming are all bypassed, so a passing eval does not mean production behavior is correct.
An HTTP Adapter connects to local, staging, or production by changing one baseUrl; an in-process call can only run in the current codebase.

When the Agent only provides an SDK, function, or CLI in the first place, call it directly in send. Do not build an extra service just to follow the HTTP pattern. Starting and stopping the service under test is not the Adapter’s job. First start the service with the project’s own dev command, deploy script, or CI, confirm the health check passes, then run the evals. When the service cannot start, let the eval report the error honestly. Do not pass off a hard-coded reply as a run result. For services that need to start and stop automatically for each eval run, see Start Shared Experiment Services.

Two channels for configuration

The easiest thing to get confused about in an integration is “where should this value come from”. There are only two channels:

Where to connect and how to authenticate goes through the Adapter factory. Environment-level configuration such as URLs, auth headers, and protocol switches is written as factory parameters in the Experiment file.
Which configuration to run this time goes through ctx. NiceEval passes the Experiment’s model and flags to send through ctx every turn. The Adapter does not interpret them; it forwards them to the application unchanged.

Turn the quickstart Adapter into a factory:

// agents/my-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";
import type { Agent } from "niceeval/adapter";

export function myAgent(options: { baseUrl: string }): Agent {
  return defineAgent({
    name: "my-agent",
    evidenceCoverage: completeEvidenceCoverage,
    async send(input, ctx) {
      const r = await fetch(`${options.baseUrl}/chat`, { // Where to connect: from the factory parameter
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({
          message: input.text,
          model: ctx.model, // Which model to run: from the Experiment
          flags: ctx.flags, // Which switches to turn on: from the Experiment
        }),
        signal: ctx.signal, // Timeout and cancellation
      });
      const body = await r.json();
      return {
        status: r.ok ? "completed" : "failed",
        events: [{ type: "message", role: "assistant", text: body.reply }],
      };
    },
  });
}

Call the factory in the Experiment, then declare model and flags:

// experiments/local.ts
import { defineExperiment } from "niceeval";
import { myAgent } from "../agents/my-agent.ts";

export default defineExperiment({
  agent: myAgent({ baseUrl: "http://localhost:3000" }),
  model: "gpt-5.4",
  flags: { promptVariant: "concise" },
});

To evaluate local and production separately, write two Experiments that differ only in baseUrl:

// experiments/prod.ts
export default defineExperiment({
  agent: myAgent({ baseUrl: "https://api.example.com" }),
  model: "gpt-5.4",
});

pnpm exec niceeval exp local
pnpm exec niceeval exp prod

Do not put the URL on the command line. Positional arguments after the Experiment ID are only for filtering evals. Fields available on ctx each turn:

Field	Comes from	How the Adapter uses it
`signal`	NiceEval’s timeout and cancellation	Attach it to every request you send
`model`	The Experiment’s `model`	Forward it if the application interface accepts a model choice; otherwise ignore it
`flags`	The Experiment’s `flags`	Forward unchanged, in the request body or a header
`session`	One per conversation	Store and read state for multi-turn continuation and human intervention; see Write send
`telemetry`	Present when OTel integration is configured	Put the `traceparent` from `headers` into the request headers; see OTel integration
`progress(update)`	The current turn’s call	Shows what is happening in the terminal; not saved
`diagnostic(input)`	The current turn’s call	Records an anomaly to review later; saved with the results

Report progress and anomalies

When calling upstream is slow or the response is incomplete, report through ctx instead of calling console.log directly:

async send(input, ctx) {
  ctx.progress({ message: "Waiting for upstream model" });
  const response = await callAgent(input, { signal: ctx.signal });

  if (response.eventsIncomplete) {
    ctx.diagnostic({
      code: "incomplete-event-stream",
      level: "warning",
      message: "Upstream response is missing tool result events",
      data: { requestId: response.requestId },
      dedupeKey: `incomplete-event-stream:${response.requestId}`,
    });
  }

  return toTurn(response);
}

Handle three cases separately:

The Agent under test replied normally, but reports failure for this turn itself: return status: "failed". The run reaches a conclusion, usually failed.
The connection failed or the response cannot be parsed, so the eval cannot continue: throw an exception. The run is recorded as errored, and the terminal gives a locator.
It can continue, but there is an anomaly worth recording: write a ctx.diagnostic. It does not change pass or fail.

The terminal shows only a one-line error summary. For the full error message, stack, and diagnostics, use niceeval show @<locator>.

Where to look first when it does not work

fetch throws directly, for example connection refused: the application is not running, or the URL is wrong. First send one request to the same interface with curl to confirm.
t.succeeded() fails: the request went out, but the Adapter returned failed. Check how the application’s error response is mapped to status.
Only content assertions fail: the integration works. Use niceeval show @<locator> --execution to see what the application actually returned, then decide whether to change the assertion or the application.

Add more capabilities as needed

A minimal integration can only check reply text. Each capability below changes only the Adapter, and existing evals need no changes:

What to check	What the Adapter does	How
Which tools were called, in what order, and what was not called	Map the application’s tool calls to standard events	Write send · Step 4
Multi-turn conversations and parallel independent sessions	Continue context with `ctx.session`	Write send · Step 2
Tokens and cost	Return `usage`	Write send · Step 3
Flows that need human approval to continue	Return `waiting` and say what it is waiting for	HITL
Call waterfall in the browser	Have the application also send OTel spans to NiceEval	OTel integration
Compare two prompt versions or feature switches	The application makes variants switchable with `flags`	Organize Experiments

For how much cooperation each capability needs from the application, see Tier.

Reference implementations

examples/zh/tier1 has five runnable integration examples: AI SDK v7, Claude SDK, Codex SDK, Pi SDK, and LangGraph. They cover tool call assertions, multi-turn isolation, HITL approve and reject, and the call waterfall. For the full fields of send, see Adapter; for all official adapters, see Official Adapters.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting