Connect OTel: See a Per-Turn Call Waterfall in the Browser

Where spans come from and where they go

Send spans to NiceEval

// agents/support-bot.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

export default defineAgent({
  name: "support-bot",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    // Example address: replace with your own agent's real endpoint
    const r = await fetch("http://localhost:5188/chat", {
      method: "POST",
      // Carry traceparent with the request: this turn's spans attach to NiceEval's trace, so attribution stays exact under concurrency
      headers: { "content-type": "application/json", ...ctx.telemetry?.headers },
      body: JSON.stringify({ message: input.text, model: ctx.model }),
      signal: ctx.signal,
    });
    const body = await r.json();
    return {
      status: r.ok ? "completed" : "failed",
      events: mapToEvents(body),   // Assertions are based on this, the same as without OTel
    };
  },
});

• A service you start yourself that keeps running (most common): fix the receiver port in niceeval.config.ts. Writing this configuration turns on OTel integration:
  // niceeval.config.ts
  export default defineConfig({
    telemetry: { port: 4318 },   // The receiver always listens on http://localhost:4318/v1/traces
  });
  
  Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces once when the service starts, and you never need to change it no matter how many evals you run. When forwarding through an OTel Collector, point it at this address too. Because the port is fixed, only one niceeval process can run on the same machine at a time. When another process holds the port, NiceEval reports an error directly and tells you to pick a free port. The receiver address given to the application defaults to 127.0.0.1. Only when the application has to go through a tunnel or other routing to reach the machine running NiceEval, use host to write that address, for example telemetry: { host: "otel.internal", port: 4318 }. A Docker Sandbox Agent starts the receiver inside the same Sandbox by default, so the container does not need to connect back to the host. OTLP receiving is configured only through these two fields and does not read environment variables.
• A child process started by NiceEval (CLI-style Agents): nothing to do. Put ctx.telemetry.env (standard OTEL_* environment variables, ready to spread) into the process environment. Each run is a new process, so it reads this run’s address.

import { generateText } from "ai";

const result = await generateText({
  model, tools, messages,
  experimental_telemetry: { isEnabled: true },
});

LANGSMITH_TRACING=true \
LANGSMITH_OTEL_ENABLED=true \
LANGSMITH_OTEL_ONLY=true \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \
node server.js   # Choose the endpoint value as described in "Tell the application where to send": injected env or fixed port

import * as traceloop from "@traceloop/node-server-sdk";

traceloop.initialize({ disableBatch: true });
// The endpoint uses the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable

from openinference.instrumentation.langchain import LangChainInstrumentor
from phoenix.otel import register

register()  # Or the standard OTel SDK; the endpoint uses OTEL_EXPORTER_OTLP_ENDPOINT
LangChainInstrumentor().instrument()

Attribute spans to Turns

• By traceparent (recommended, accurate under concurrency): when send makes the request, spread ctx.telemetry.headers into the request headers so each turn carries a new traceparent. When the application’s instrumentation supports trace context propagation (all standard OTel HTTP server instrumentation does), this turn’s spans attach under the trace NiceEval provided and match exactly by traceId.
• By time (fallback): when the application does not propagate trace context, attribution uses the start and end times of send. This is only reliable when turns run one at a time, so NiceEval runs this Agent’s turns one at a time and notes it in the logs, instead of mixing spans from different conversations. Once traceparent is confirmed to work, concurrency resumes automatically.

Keep your existing OTel backend and send to both

// instrumentation.ts — initialize once at application startup: one set of instrumentation, two outputs
import { NodeTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

export const provider = new NodeTracerProvider({
  spanProcessors: [
    // Output 1: your own backend, always on
    new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.MY_COLLECTOR_URL })),
    // Output 2: NiceEval. In production this variable is absent, so this output does not exist; same code online and offline
    ...(process.env.OTEL_EXPORTER_OTLP_ENDPOINT
      ? [new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }))]
      : []),
  ],
});
provider.register();

Control waterfall content with semantic mapping

Private instrumentation: write your own mapping

• Change the instrumentation (recommended): add a gen_ai.operation.name attribute to spans in the application. It is a one-line change, and your own observability backend benefits too.
• Write a spanMapper: when changing the application is inconvenient, declare a pure function on the Agent that translates custom spans to the semantics in the table above before drawing. tagSpan writes the category back to the span, only adding attributes and never changing existing ones; heuristicTag is the same logic automatic recognition uses. Both are exported from niceeval/adapter:

import { completeEvidenceCoverage, defineAgent, tagSpan, heuristicTag } from "niceeval/adapter";
import type { TraceSpan } from "niceeval";

function mapMySpans(spans: TraceSpan[]): TraceSpan[] {
  return spans.map((span) => {
    // Private naming → standard semantics: op goes into gen_ai.operation.name, kind decides the color
    if (span.name === "my.llm.request") return tagSpan(span, { op: "chat", kind: "model" });
    if (span.name === "my.tool.exec") {
      // Map the private call id to call_id so tool I/O can link to send events
      const attributes = { ...span.attributes, call_id: String(span.attributes?.["my.callId"] ?? "") };
      return tagSpan({ ...span, attributes }, { op: "execute_tool", kind: "tool" });
    }
    return tagSpan(span, heuristicTag(span));   // Leave the rest to automatic recognition
  });
}

export default defineAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  spanMapper: mapMySpans,
  async send(input, ctx) { /* Same as the integration above */ },
});

Check the result

• The waterfall is empty: not a single span arrived during the whole run, usually because the application did not reach the receiver address, for example the environment variable was not set, or the service was not restarted after a config change. NiceEval notes this in the logs, and assertions are judged as usual.
• The waterfall occasionally misses its tail: first check whether the application uses BatchSpanProcessor; see “Attribute spans to Turns” above.
• All spans show as other: see “Private instrumentation: write your own mapping” above.

What OTel integration does not handle

• All assertion data comes from send. Tool calls must be mapped into events (with a built-in converter or handwritten mapping; see Write send), and usage goes in the return value of send. Data that appears only in spans and not in events does not take part in assertions.
• Multi-turn sessions and HITL still happen in send. Spans do not express “waiting for human input”, and session continuation is decided by the application interface (for session continuation see Write send; for HITL see HITL).

Related reading

• Connect Your Agent: send event mapping and the data source for assertions.
• Write send: the full tutorial for a handwritten Adapter; step six is the Adapter-side configuration for this page.
• Events reference: the event structure assertions read.