> ## Documentation Index > Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # OTel Integration > Send the OTel spans your app already emits to NiceEval, then view a per-turn call waterfall in niceeval view. Eval assertions still come from the events and usage send returns. OTel integration **does not change where assertion data comes from**. `t.calledTool`, `t.maxTokens`, and timing verdicts all read the `Turn` (`events` and `usage`) your adapter returns from `send` — see [Connect Your Agent](/docs/tutorials/connect-your-agent). OTel spans power **the call waterfall in `niceeval view`**. The waterfall shows model calls, tool execution, duration, and tokens per turn, helping you pinpoint the exact step where an eval failed or a turn slowed down. If your app already emits OTel traces — AI SDK telemetry, LangGraph's LangSmith export, OpenLLMetry / OpenInference auto-instrumentation, or your own spans following the GenAI semantic conventions — you are already producing the waterfall data: just have the app send a copy of its spans to [NiceEval](https://niceeval.com/) too. Application code stays untouched, and the integration remains non-intrusive (see [Tier](/docs/explanation/tier)). ## How it works (one paragraph) At run time [NiceEval](https://niceeval.com/) starts a local OTLP receiver. Spans the app sends are attributed to the corresponding `send` turn, normalized into GenAI semantics, and written to the Attempt-owned `niceeval.telemetry` JSONL event channel. A Report decodes and displays the waterfall only when it declares that requirement. **Spans never affect assertions** — they do not change Agent behavior, execution errors, or Verdicts. Missing instrumentation, late spans, or dropped batches affect that channel's coverage and waterfall completeness only; they never affect a Verdict. ## Wiring it up **1. Adapter side** — write `send` as usual (the event mapping is still your mapping), plus one extra line: forward this turn's `traceparent` with the request: ```ts theme={null} // agents/support-bot.ts import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter"; export default defineAgent({ name: "support-bot", evidenceCoverage: completeEvidenceCoverage, async send(input, ctx) { // Example URL: swap in your own agent's real endpoint const r = await fetch("http://localhost:5188/chat", { method: "POST", // Forward traceparent with the request: this turn's spans attach to NiceEval's trace, // which is what makes attribution precise under concurrency headers: { "content-type": "application/json", ...ctx.telemetry?.headers }, body: JSON.stringify({ message: input.text, model: ctx.model }), signal: ctx.signal, }); const body = await r.json(); return { status: r.ok ? "completed" : "failed", events: mapToEvents(body), // Assertions are based on this, same as without OTel }; }, }); ``` Built-ins do not need this step: `uiMessageStreamAgent` always merges `ctx.telemetry.headers` into the request headers automatically. **2. How the endpoint reaches your app.** [NiceEval](https://niceeval.com/)'s receiver endpoint is **startup-time configuration, never passed through `send`**. Standard OTel SDKs read `OTEL_*` environment variables only once, at process startup. Pick a configuration approach by deployment shape: * **Your own long-running service (most common)**: use **fixed-port mode** — pin the receiver port in `niceeval.config.ts`. Writing this config is what turns OTel integration on: ```ts theme={null} // niceeval.config.ts export default defineConfig({ telemetry: { port: 4318 }, // Receiver always listens on http://localhost:4318/v1/traces }); ``` Configure `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces` once when the service starts; it stays valid no matter how many eval runs follow. The trade-off: sharing a port means only one niceeval process can run on the machine at a time; OTel Collector fan-out setups point at this same fixed endpoint. If `port` is already taken by another process, [NiceEval](https://niceeval.com/) fails immediately with an error telling you to pick a free port — it never fails silently. The receiver hostname reported to your app defaults to `127.0.0.1`. Only when your app can already reach the host's receiver through a controlled tunnel or another reachable route should you report that address via `host`, e.g. `telemetry: { host: "otel.internal", port: 4318 }`. Docker Sandbox Agents start the receiver inside the same Sandbox by default, without relying on the container reaching back to the host. These two fields are the only way to configure the OTLP receiver in [NiceEval](https://niceeval.com/); it does not read environment variables. * **Child processes / processes [NiceEval](https://niceeval.com/) spawns** (CLI-style agents): nothing to do. `ctx.telemetry.env` (standard `OTEL_*` environment variables, ready to spread) is injected into the process environment; each run is a fresh process that reads the fresh endpoint. **3. App side** — a few lines of configuration, depending on your instrumentation ecosystem: The official OTel integration (`@ai-sdk/otel`, which emits standard GenAI semantics) is recommended; the older `experimental_telemetry` (`ai.*`) renders too: ```ts theme={null} import { generateText } from "ai"; const result = await generateText({ model, tools, messages, experimental_telemetry: { isEnabled: true }, }); ``` The exporter is the standard OTel Node SDK, with the endpoint pointed at [NiceEval](https://niceeval.com/) (the injected env or the fixed port). Runnable examples: app-side instrumentation in [`examples/zh/origin/ai-sdk-v7`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/ai-sdk-v7) (`src/backend/otel.ts`, official `@ai-sdk/otel`); the full eval project after integration in [`examples/zh/tier1/ai-sdk-v7`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7). The zero-dependency route: three environment variables: ```bash theme={null} LANGSMITH_TRACING=true \ LANGSMITH_OTEL_ENABLED=true \ LANGSMITH_OTEL_ONLY=true \ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \ node server.js # Endpoint value per "How the endpoint reaches your app": injected env or fixed port ``` `LANGSMITH_OTEL_ONLY` means OTLP only, no LangSmith cloud export; remove it to dual-send. The **Python** `langsmith` SDK registers the OTel hook automatically at import time. The **JS** version (`langsmith@0.7.x`) additionally needs an explicit call to `initializeOTEL()` (exported from `langsmith/experimental/otel/setup`) — without it, it only logs a warning and produces no spans. The three environment variables stay the same either way. Python backend example: [`examples/zh/origin/langgraph`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/origin/langgraph); the full eval project after integration: [`examples/zh/tier1/langgraph`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/langgraph). ```ts theme={null} import * as traceloop from "@traceloop/node-server-sdk"; traceloop.initialize({ disableBatch: true }); // Endpoint via the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable ``` ```python theme={null} from openinference.instrumentation.langchain import LangChainInstrumentor from phoenix.otel import register register() # Or the standard OTel SDK; endpoint via OTEL_EXPORTER_OTLP_ENDPOINT LangChainInstrumentor().instrument() ``` Instrument per the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/): name model-call spans `chat {model}` and tool spans `execute_tool {tool}`, with `gen_ai.operation.name`, `gen_ai.tool.name` / `gen_ai.tool.call.id` attributes. This is the target semantics [NiceEval](https://niceeval.com/) normalizes into, and it produces the most complete waterfall. ## How spans are attributed to turns When evals run in parallel, one receiver receives spans from multiple sessions at once. [NiceEval](https://niceeval.com/) attributes them to their turns via two paths: * **traceparent (recommended, concurrency-safe)**: when `send` makes its request, spread `ctx.telemetry.headers` (W3C trace context, a fresh `traceparent` per turn) into the request headers. If the app's instrumentation supports context propagation (standard OTel HTTP server instrumentation does), this turn's spans automatically attach under the trace [NiceEval](https://niceeval.com/) provided, and attribution is exact by traceId. * **Time window (fallback)**: when the app does not propagate trace context, spans are attributed by the time window around `send`. The window is only reliable when turns run serially, so in this case [NiceEval](https://niceeval.com/) executes this agent's turns serially and says so in the logs — it never silently mixes streams. Once traceparent is confirmed working, concurrency resumes automatically. Export promptly on the app side: what the waterfall cares about is "this turn's spans arrive in time". Use `SimpleSpanProcessor` (or flush every turn) — `BatchSpanProcessor`'s buffering makes spans arrive late across turns; when the waterfall occasionally has a missing tail, that is usually why. ## Keep your existing OTel backend and dual-send Your app most likely already sends traces to its own observability backend (Langfuse / SigNoz / a production collector). Connecting [NiceEval](https://niceeval.com/) **requires neither switching backends nor a second layer of instrumentation**: a TracerProvider supports multiple SpanProcessors — the same batch of spans, two exits: ```ts theme={null} // instrumentation.ts — initialize once at app startup: one set of instrumentation, two exits import { NodeTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; export const provider = new NodeTracerProvider({ spanProcessors: [ // Exit 1: your own backend, always on new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.MY_COLLECTOR_URL })), // Exit 2: NiceEval. Without this variable in production, this exit simply // does not exist — same code in production and in evals ...(process.env.OTEL_EXPORTER_OTLP_ENDPOINT ? [new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }))] : []), ], }); provider.register(); ``` If touching app code is inconvenient, fan out through an OTel Collector instead — the app sends only to the collector, and the collector configures two exporters (your backend + [NiceEval](https://niceeval.com/)'s fixed endpoint). The cost is one more component to operate. ## Controlling waterfall content with semantic mapping [NiceEval](https://niceeval.com/) normalizes every span into GenAI semantics before drawing the waterfall. The mapping reads `gen_ai.operation.name` (the standard operation name) and the normalized `kind` (semantic role): | Span semantics | Waterfall content | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gen_ai.operation.name: "chat"` (or `text_completion` / `embeddings`) | Colored and grouped as a **model call** | | `gen_ai.operation.name: "execute_tool"`, or a `tool_name` attribute | Colored and grouped as a **tool execution** | | `gen_ai.operation.name: "invoke_agent"` / `"create_agent"` | Colored and grouped as a **sub-agent** | | A `call_id` attribute equal to the `operationId` in one of your send events | That tool span gets the real input/output backfilled (`io.input` / `io.output`, joined by `operationId` from the events `send` returned — many instrumentations do not capture content by default, and this path is unaffected) | | Span status = error | Marked red | | Nothing recognizable (normalized to `other`) | The timeline is still there; in large traces (>150 spans) it is folded away as internal noise | | Any other attributes | Preserved as-is, drillable in the view | If your app instruments directly per the GenAI semconv (the "Hand-rolled gen\_ai" tab above), all of this holds automatically; the common shapes of mainstream formats (AI SDK, LangSmith, OpenLLMetry / OpenInference) are also within the generic fallback's recognition range. ### Private instrumentation: write your own mapping When your instrumentation has an app-private shape the generic fallback cannot recognize, there are two routes: * **Fix the instrumentation (recommended)**: add a `gen_ai.operation.name` attribute to the spans on the app side — a one-line change, and your own observability backend benefits equally. * **Write a `spanMapper`**: when touching the app is inconvenient, declare a pure function on the agent that translates private spans into the semantics of the table above before rendering. Both `tagSpan` (writes the verdict back onto the span; existing attributes are only added to, never changed) and `heuristicTag` (the generic fallback verdict) are exported from `niceeval/adapter`: ```ts theme={null} import { completeEvidenceCoverage, defineAgent, tagSpan, heuristicTag } from "niceeval/adapter"; import type { TraceSpan } from "niceeval"; function mapMySpans(spans: TraceSpan[]): TraceSpan[] { return spans.map((span) => { // Private naming → standard semantics: op goes into gen_ai.operation.name, kind sets the coloring if (span.name === "my.llm.request") return tagSpan(span, { op: "chat", kind: "model" }); if (span.name === "my.tool.exec") { // Copy the private call id into call_id so tool I/O can be joined from send events const attributes = { ...span.attributes, call_id: String(span.attributes?.["my.callId"] ?? "") }; return tagSpan({ ...span, attributes }, { op: "execute_tool", kind: "tool" }); } return tagSpan(span, heuristicTag(span)); // leave the rest to the generic fallback }); } export default defineAgent({ name: "my-agent", evidenceCoverage: completeEvidenceCoverage, spanMapper: mapMySpans, async send(input, ctx) { /* same wiring as above */ }, }); ``` `mapCodexSpans`, exported from `niceeval/adapter`, is a ready-made `spanMapper` implementation you can use as a reference. A `spanMapper` only affects observability display; a mapping mistake makes the waterfall's grouping or coloring inaccurate — it never affects assertions. ## Boundaries * **All assertion data comes from `send`**. Tool calls need to be mapped into `events` (using a built-in converter or a hand-written mapping, see [Write send](/docs/tutorials/write-send)); usage needs to be included in the `send` return value. Data that exists in a span but is missing from `events` never participates in assertions. * **Multi-turn sessions and HITL are not span concerns.** Spans have no "waiting for human input" semantics, and session continuation is application-protocol work — both keep happening in `send` as usual (session continuation in [Write send](/docs/tutorials/write-send), the HITL concept in [HITL](/docs/explanation/hitl)). * **You get a warning when no spans arrive.** A run that produces zero spans overall is usually because the endpoint is not hooked up (env not injected, service not restarted); [NiceEval](https://niceeval.com/) says so in the logs. The waterfall is empty; assertions judge as usual. ## Related reading * [Connect Your Agent](/docs/tutorials/connect-your-agent) — the `send` event mapping and where assertion data comes from. * [Write send](/docs/tutorials/write-send) — the full tutorial for hand-writing an adapter; step six is this page's adapter-side wiring. * [Events reference](/docs/reference/events) — the event structure assertions read.