OTel Integration

​

How it works (one paragraph)

​

Wiring it up

// agents/support-bot.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

export default defineAgent({
  name: "support-bot",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    // Example URL: swap in your own agent's real endpoint
    const r = await fetch("http://localhost:5188/chat", {
      method: "POST",
      // Forward traceparent with the request: this turn's spans attach to NiceEval's trace,
      // which is what makes attribution precise under concurrency
      headers: { "content-type": "application/json", ...ctx.telemetry?.headers },
      body: JSON.stringify({ message: input.text, model: ctx.model }),
      signal: ctx.signal,
    });
    const body = await r.json();
    return {
      status: r.ok ? "completed" : "failed",
      events: mapToEvents(body),   // Assertions are based on this, same as without OTel
    };
  },
});

• Your own long-running service (most common): use fixed-port mode — pin the receiver port in niceeval.config.ts. Writing this config is what turns OTel integration on:
  // niceeval.config.ts
  export default defineConfig({
    telemetry: { port: 4318 },   // Receiver always listens on http://localhost:4318/v1/traces
  });
  
  Configure OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces once when the service starts; it stays valid no matter how many eval runs follow. The trade-off: sharing a port means only one niceeval process can run on the machine at a time; OTel Collector fan-out setups point at this same fixed endpoint. If port is already taken by another process, NiceEval fails immediately with an error telling you to pick a free port — it never fails silently. The receiver hostname reported to your app defaults to 127.0.0.1. Only when your app can already reach the host’s receiver through a controlled tunnel or another reachable route should you report that address via host, e.g. telemetry: { host: "otel.internal", port: 4318 }. Docker Sandbox Agents start the receiver inside the same Sandbox by default, without relying on the container reaching back to the host. These two fields are the only way to configure the OTLP receiver in NiceEval; it does not read environment variables.
• Child processes / processes NiceEval spawns (CLI-style agents): nothing to do. ctx.telemetry.env (standard OTEL_* environment variables, ready to spread) is injected into the process environment; each run is a fresh process that reads the fresh endpoint.

import { generateText } from "ai";

const result = await generateText({
  model, tools, messages,
  experimental_telemetry: { isEnabled: true },
});

LANGSMITH_TRACING=true \
LANGSMITH_OTEL_ENABLED=true \
LANGSMITH_OTEL_ONLY=true \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces \
node server.js   # Endpoint value per "How the endpoint reaches your app": injected env or fixed port

import * as traceloop from "@traceloop/node-server-sdk";

traceloop.initialize({ disableBatch: true });
// Endpoint via the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable

from openinference.instrumentation.langchain import LangChainInstrumentor
from phoenix.otel import register

register()  # Or the standard OTel SDK; endpoint via OTEL_EXPORTER_OTLP_ENDPOINT
LangChainInstrumentor().instrument()

​

How spans are attributed to turns

• traceparent (recommended, concurrency-safe): when send makes its request, spread ctx.telemetry.headers (W3C trace context, a fresh traceparent per turn) into the request headers. If the app’s instrumentation supports context propagation (standard OTel HTTP server instrumentation does), this turn’s spans automatically attach under the trace NiceEval provided, and attribution is exact by traceId.
• Time window (fallback): when the app does not propagate trace context, spans are attributed by the time window around send. The window is only reliable when turns run serially, so in this case NiceEval executes this agent’s turns serially and says so in the logs — it never silently mixes streams. Once traceparent is confirmed working, concurrency resumes automatically.

​

Keep your existing OTel backend and dual-send

// instrumentation.ts — initialize once at app startup: one set of instrumentation, two exits
import { NodeTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

export const provider = new NodeTracerProvider({
  spanProcessors: [
    // Exit 1: your own backend, always on
    new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.MY_COLLECTOR_URL })),
    // Exit 2: NiceEval. Without this variable in production, this exit simply
    // does not exist — same code in production and in evals
    ...(process.env.OTEL_EXPORTER_OTLP_ENDPOINT
      ? [new SimpleSpanProcessor(new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }))]
      : []),
  ],
});
provider.register();

​

Controlling waterfall content with semantic mapping

​

Private instrumentation: write your own mapping

• Fix the instrumentation (recommended): add a gen_ai.operation.name attribute to the spans on the app side — a one-line change, and your own observability backend benefits equally.
• Write a spanMapper: when touching the app is inconvenient, declare a pure function on the agent that translates private spans into the semantics of the table above before rendering. Both tagSpan (writes the verdict back onto the span; existing attributes are only added to, never changed) and heuristicTag (the generic fallback verdict) are exported from niceeval/adapter:

import { completeEvidenceCoverage, defineAgent, tagSpan, heuristicTag } from "niceeval/adapter";
import type { TraceSpan } from "niceeval";

function mapMySpans(spans: TraceSpan[]): TraceSpan[] {
  return spans.map((span) => {
    // Private naming → standard semantics: op goes into gen_ai.operation.name, kind sets the coloring
    if (span.name === "my.llm.request") return tagSpan(span, { op: "chat", kind: "model" });
    if (span.name === "my.tool.exec") {
      // Copy the private call id into call_id so tool I/O can be joined from send events
      const attributes = { ...span.attributes, call_id: String(span.attributes?.["my.callId"] ?? "") };
      return tagSpan({ ...span, attributes }, { op: "execute_tool", kind: "tool" });
    }
    return tagSpan(span, heuristicTag(span));   // leave the rest to the generic fallback
  });
}

export default defineAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  spanMapper: mapMySpans,
  async send(input, ctx) { /* same wiring as above */ },
});

​

Boundaries

• All assertion data comes from send. Tool calls need to be mapped into events (using a built-in converter or a hand-written mapping, see Write send); usage needs to be included in the send return value. Data that exists in a span but is missing from events never participates in assertions.
• Multi-turn sessions and HITL are not span concerns. Spans have no “waiting for human input” semantics, and session continuation is application-protocol work — both keep happening in send as usual (session continuation in Write send, the HITL concept in HITL).
• You get a warning when no spans arrive. A run that produces zero spans overall is usually because the endpoint is not hooked up (env not injected, service not restarted); NiceEval says so in the logs. The waterfall is empty; assertions judge as usual.

​

Related reading

• Connect Your Agent — the send event mapping and where assertion data comes from.
• Write send — the full tutorial for hand-writing an adapter; step six is this page’s adapter-side wiring.
• Events reference — the event structure assertions read.