Skip to main content
Connecting to NiceEval is not an all-or-nothing one-time project. Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers — each tier adds one kind of investment on top of the previous one and buys a new set of capabilities, and every eval you have already written keeps working unchanged the whole way up. Integration comes in three tiers: Tier 1 is send only, with zero application changes; Tier 2 adds OTel and buys the call waterfall in niceeval view; Tier 3 modifies the application to expose variants as flags for A/B testing.

​
Tier 1: send only

Your application code stays untouched; the integration is a single Adapter file — send talks to the same interface your user-facing frontend already uses. The full assertion set lives at this tier: text assertions and judges, structured output validation, multi-turn and session isolation, HITL approval flows, tool assertions from event mapping, and usage assertions from the usage that send returns — every verdict is based entirely on the Turn returned by send, and the next two tiers add no further assertions. If the application interface itself exposes model selection, model comparison experiments also live at this tier — the experiment’s model reaches the Adapter via ctx.model and is simply forwarded with the request. Most teams start here, and for many this tier is all they ever need.

​
Tier 2: send + OTel

Same send, same event mapping — the only change is that your application also sends its OTel spans to NiceEval. If the app is already instrumented (AI SDK telemetry, LangGraph, OpenLLMetry / OpenInference, hand-rolled gen_ai spans), this is zero change; if not, what you add is a piece of generic OTel initialization — that is observability infrastructure, not a modification built for eval purposes. What this tier buys is observability: the call waterfall in niceeval view — every model call and tool execution inside the application, each with its own latency and tokens, laid out as a per-turn timeline. Assertions are unaffected: spans are supplemental telemetry Observations that never feed assertions. See OTel Integration for the walkthrough.

​
Tier 3: application changes + experiment flags

To evaluate “which prompt / tool set / feature toggle works better” — a feature A/B test — the point of comparison lives inside the application, out of reach for the first two tiers. This tier changes the application’s internal code to expose the variants as externally selectable configuration: the experiment’s flags pass through ctx.flags to the Adapter, the Adapter hands them to the application with the request (an HTTP header, a request body field, or an environment variable all work), and the application switches variants based on the parameter. Note that what gets modified is the application (exposing variants as configuration), not the integration surface — the Adapter itself stays the same and still calls the application only through its external interface.

​
Non-intrusive is the baseline for the first two tiers

The first two tiers are both non-intrusive: you start the application your own way (pnpm start, deployed wherever) — the eval side never spawns the application process or opens a separate port. The Adapter only talks to the same interface your user-facing frontend already uses (an HTTP endpoint, an SSE stream), and reports a clear “start the application first” error when it cannot connect. The only difference between the two tiers is observability data: Tier 1 has only the events returned by send; Tier 2 adds a copy of spans sent to NiceEval, and the waterfall appears along with them.

​
How to move up

The three tiers are progressive, not mutually exclusive: start with Tier 1 to get baselines and model comparison running; move up to Tier 2 when you want the call waterfall; move up to Tier 3 when you need to compare variants inside the application. Each upgrade only adds something to the Adapter or the application — how to organize comparisons on the experiment side is covered in Experiments.
  • Connect Your Agent — The integration overview: minimal integration and parameter channels.
  • Adapter — The contract itself: what send receives and returns, and how ctx.model / ctx.telemetry / ctx.flags appear per tier.
  • OTel Integration — The full Tier 2 walkthrough.
  • Experiments — How model / flags comparisons are declared on the experiment side.