Standard Event Stream Reference

An Adapter’s send returns a Turn. Its raw events: StreamEvent[] is the source of truth. NiceEval derives immutable toolCalls and eventOccurrences collections for managed Matches. Raw events remains an ordinary Value subject for check; occurrence quantifiers and order Matches use eventOccurrences. Translate what your Agent did in this turn truthfully into the stream, and the complete event-assertion surface is available.

Turn: the return value of send

interface Turn {
  events: StreamEvent[];                          // this turn's events, in their actual order
  data?: JsonValue;                               // structured output, which t.check(...) can compare
  status: "completed" | "failed" | "waiting";     // waiting = paused for human input (HITL)
  usage?: Usage;                                  // → maxTokens / maxCost / cost reports; full fields below
}

data means “this turn’s structured output.” Fill it only when the application’s response itself is a structured object—for extraction, classification, or form completion—so an Eval author can compare it with t.check(turn.data, ...). An application that returns only text leaves it unset. Do not copy the raw response body or message text into it just to fill it, and do not serialize structured output into events instead. Include usage when it is available, and omit it when it is not—never invent numbers. The full fields of usage (the Usage type):

`inputTokens`

inputTokens?: number;

Input tokens that did not hit cache and are billed at the full rate. They are mutually exclusive with the two cache buckets.

`outputTokens`

outputTokens?: number;

The number of output (completion) tokens.

`cacheReadTokens`

cacheReadTokens?: number;

Input tokens served from a prompt-cache hit. This is a separate billing bucket and is not included in inputTokens. Omission means the Agent does not report it.

`cacheCreationTokens`

cacheCreationTokens?: number;

Input tokens written to a prompt cache. This is a separate billing bucket and is not included in inputTokens. Omission means the Agent does not report it.

`reasoningTokens`

reasoningTokens?: number;

The number of reasoning (thinking) tokens. It is a detail already included in outputTokens and is listed separately for display. It exists only when the protocol actually supplies it.

`requests`

requests?: number;

The number of model requests that actually occurred. Omit it when the protocol does not provide a request count; never write 1 just to fill the field.

`costUSD`

costUSD?: number;

The actual USD cost measured by the gateway or Adapter. It can only be explicitly returned through Turn.usage.costUSD; it is never derived from token usage or an OTel span. It is separate from the top-level estimatedCostUSD (a price-table estimate). This is a one-way field contract: this field stores only the observed value; estimatedCostUSD is always the estimate from estimateCost(model, usage, pricing), computed independently even when an observed value exists. Neither value overrides or fills in for the other; an observed value never replaces or triggers an estimate.

Cost boundary: Usage.costUSD represents only observed USD cost returned by a Provider or Adapter. It is never derived from tokens, a model catalog, or a local price table. The Runner independently calculates estimatedCostUSD from the model, reported token usage, and the Config/runtime price table even when an observed cost exists. The Experiment budget and t.maxCost() consume that estimate; Usage.costUSD does not. A Report cost projection accepts only an explicit PricingProfile and sealed Usage; it does not read the Runner estimate.

StreamEvent variants at a glance

The ten StreamEvent variants are listed field by field below. For assertions that consume them and usage details, see the Event table and Event details sections that follow.

`message`

{ type: "message"; role: "assistant"; text: string; loc?: SourceLoc }

`message`

{ type: "message"; role: "user"; text: string; loc?: SourceLoc; sourceOrder?: number }

`operation.started`

{
  type: "operation.started";
  operationId: string;
  operation:
    | {
        kind: "tool";
        name: string;
        input: JsonValue;
        tool?: ToolName;
        /**
         * The Adapter's protocol-level classification as command / not-command. It is temporarily
         * optional to support third-party Adapters that have not migrated. When it is absent, core
         * must not fabricate it from the name, input, or shell text.
         */
        command?: CommandProjection;
      }
    | { kind: "subagent"; name: string; remoteUrl?: string };
}

`operation.finished`

{
  type: "operation.finished";
  operationId: string;
  kind: "tool";
  output?: JsonValue;
  status: "completed" | "failed" | "rejected";
}

`operation.finished`

{
  type: "operation.finished";
  operationId: string;
  kind: "subagent";
  output?: JsonValue;
  status: "completed" | "failed";
}

`skill.loaded`

{ type: "skill.loaded"; skill: string; operationId?: string }

`input.requested`

{ type: "input.requested"; request: InputRequest }

`thinking`

{ type: "thinking"; text: string }

`context.injected`

{ type: "context.injected"; text: string; source?: string }

`compaction`

{ type: "compaction"; reason?: string }

`error`

{ type: "error"; message: string }

Event table

Event	What it says	Assertion / API that consumes it
`message`	The Agent (or user) said something	`turn.message`, `turn.check(turn.eventOccurrences, eventMatch("message", ...))`, Judge material
`operation.started` (`kind: "tool"`)	Starts a tool call	`turn.calledTool(...)`, `turn.toolOrder(...)`, `turn.check(turn.eventOccurrences, eventMatch(...))`
`operation.finished` (`kind: "tool"`)	The result of that tool call	`ToolMatch` with a status condition
`operation.started` (`kind: "subagent"`)	Delegates to a subagent	Raw event and Report; there is currently no subagent-specific assertion
`operation.finished` (`kind: "subagent"`)	The subagent returns	Raw event and Report; a failure enters the aggregate fact for `noFailedActions()`
`input.requested`	Stops to wait for human input (HITL)	`t.check(turn.status, equals("waiting"))`, `t.requireInputRequest()`
`thinking`	Thinking text	Displayed by view; there is currently no `eventMatch` selector
`compaction`	Context was compacted	Displayed by view; there is currently no `eventMatch` selector
`error`	This turn failed	Displayed by view; there is currently no `eventMatch` selector

eventMatch currently has three shapes: message, tool operation.started, and tool operation.finished. Apply occurrence quantifiers to this Match, then register it against the managed eventOccurrences collection. Use inOrder([...]) against a Turn or Session collection for ordering. You cannot use every raw StreamEvent as an occurrence selector, but you can still pass raw events to check with an ordinary Value Match.

import { eventMatch, inOrder, satisfies } from "niceeval/expect";

turn.check(turn.eventOccurrences, eventMatch("message").atLeast(2));
turn.check(
  turn.eventOccurrences,
  inOrder([eventMatch("operation.started"), eventMatch("operation.finished")]),
);
t.check(turn.events, satisfies("contains thinking", (events) =>
  events.some((event) => event.type === "thinking"),
));

Event details

`message` — what was said

{ type: "message", role: "assistant" | "user", text: string }

Emit one role: "assistant" message for every segment of assistant text. A tool result is not an assistant message. Do not wrap tool output in message, or t.reply will read the wrong content. NiceEval automatically records user-input message events; an Adapter does not need to emit them.

Tool `operation.started` / `operation.finished` — which tool ran and what happened

{ type: "operation.started", operationId: string,
  operation: { kind: "tool", name: string, input: JsonValue,
    tool?: ToolName, command?: CommandProjection } }
{ type: "operation.finished", operationId: string, kind: "tool",
  output?: JsonValue, status: "completed" | "failed" | "rejected" }

Pair each tool operation.started with a tool operation.finished that has the same operationId. That prevents concurrent calls from being paired incorrectly. Use the explicit ID in your Agent’s return when it has one—such as AI SDK’s toolCallId or Anthropic’s tool_use.id—and synthesize it by order only as a last resort.
Set status truthfully. A failed tool execution is "failed"; a human rejection is "rejected". Eval authors use ToolMatch status conditions to distinguish them precisely, so do not conflate the two.
Use the tool’s original name for name.
With only started, the logical occurrence has pending status and unavailable output. A finished event that omits output also means unavailable; do not turn missing data into empty JSON or an ordinary mismatch.

Pass tool events to `ToolMatch`

The same shell tool might run git status and pnpm test in sequence. Its name tells you only the tool category. When an Eval author needs to identify the command, use the normalized command projection:

import { commandMatch } from "niceeval/expect";

turn.calledTool(commandMatch("pnpm", { argsStart: ["test"] }));

ToolMatch compares one logical occurrence at a time. For complete selectors, JSON, paths, output, and counting rules, see Scoped assertions. When the native protocol directly supplies structured argv for one invocation, call the public constructor from niceeval/adapter:

import { commandProjection } from "niceeval/adapter";

events.push({
  type: "operation.started",
  operationId: call.id,
  operation: {
    kind: "tool",
    name: call.name,
    input: call.input,
    command: commandProjection({
      state: "available",
      executable: call.executable,
      args: call.args,
    }),
  },
});

commandProjection() retains original tokens confirmed by the Adapter and uses the same normalizeLogicalCommand() to produce a logical-command/v1 projection. pnpm exec niceeval view can be matched precisely by commandMatch("niceeval", { argsStart: ["view"] }). Mark the original command available only when the native protocol already supplies argv, or the protocol grammar can unambiguously produce one invocation. Use opaqueCommandProjection(reason) when the protocol supplies only shell source or when content is truncated or redacted. Use notCommandProjection() when you can confirm that it is not a command. When you cannot confirm command / not-command, downgrade actions coverage; do not guess from a tool name, input, or shell text.

Subagent `operation.started` / `operation.finished` — who was delegated to

{ type: "operation.started", operationId: string,
  operation: { kind: "subagent", name: string, remoteUrl?: string } }
{ type: "operation.finished", operationId: string, kind: "subagent",
  output?: JsonValue, status: "completed" | "failed" }

When the system under test delegates a task to a subagent and waits for it to return, emit this pair. The operationId pairing rule is the same as above. These events remain in the raw event stream and Report, but the public assert-first API currently has no calledSubagent, and eventMatch(...) cannot select a subagent operation. Do not invent a different selector.

`input.requested` — stop and wait for a person (HITL)

{ type: "input.requested", request: {
    id?: string,
    action?: string,        // the action where it paused (such as a tool name)
    input?: JsonValue,      // the action's input
    prompt?: string,        // the question for the person
    options?: { id: string, label?: string }[],   // optional choices (approve / deny…)
} }

When the Agent stops a turn to wait for a person, emit one event for each pending question and return "waiting" as that Turn’s status. The t.requireInputRequest(filter) filter matches this request field by field. Fill in as many fields as you can, or the Eval cannot select it. See the HITL section of the connection tutorial.

`thinking` / `compaction` / `error`

{ type: "thinking", text: string }
{ type: "compaction", reason?: string }   // context compression; an Adapter parser emits it; there is no declaration layer
{ type: "error", message: string }

Emit these events when you have them; do not fabricate them when you do not. compaction mostly comes from coding-Agent CLIs when a full context is compressed automatically. These raw events are retained for Reports and diagnostics; the public eventMatch cannot currently select them.

Three mapping rules

Order is fact: lay events out in the order they actually occurred. toolOrder / eventOrder use a monotonic cursor to match subsequences of different occurrences. The wrong order makes assertions misleading. eventOrder only orders events supported by public eventMatch.
Pair operationIds: every started operation needs a finished operation with the same kind and ID. Only a complete pair lets the framework assign status and input to the same logical tool occurrence.
Completeness must be explicit: official converters declare evidence coverage according to actual Adapter capability. A negative assertion with incomplete relevant input or action coverage is unavailable; it never turns “not observed” into “did not happen.” Manual mappings must likewise state their coverage honestly. See the Capabilities reference.

A complete mapping example

When an Agent return has step records, the mapping is a small loop:

import type { StreamEvent } from "niceeval";

function toStreamEvents(body: MyAgentResponse): StreamEvent[] {
  const events: StreamEvent[] = [];
  for (const step of body.steps) {
    if (step.type === "tool_call") {
      events.push({
        type: "operation.started",
        operationId: step.id,
        operation: { kind: "tool", name: step.tool, input: step.args },
      });
      events.push({
        type: "operation.finished",
        operationId: step.id,
        kind: "tool",
        output: step.result,
        status: step.error ? "failed" : "completed",
      });
    }
    if (step.type === "text") events.push({ type: "message", role: "assistant", text: step.text });
  }
  return events;
}

Connect your Agent — the tutorial for making an integration work from scratch.
Capabilities — what declaring a complete event stream means.
Authoring Evals — the complete assertion surface that consumes this stream.

​Turn: the return value of send

​inputTokens

​outputTokens

​cacheReadTokens

​cacheCreationTokens

​reasoningTokens

​requests

​costUSD

​StreamEvent variants at a glance

​message

​message

​operation.started

​operation.finished

​operation.finished

​skill.loaded

​input.requested

​thinking

​context.injected

​compaction

​error

​Event table

​Event details

​message — what was said

​Tool operation.started / operation.finished — which tool ran and what happened

​Pass tool events to ToolMatch

​Subagent operation.started / operation.finished — who was delegated to

​input.requested — stop and wait for a person (HITL)

​thinking / compaction / error

​Three mapping rules

​A complete mapping example

​Related reading

Turn: the return value of send

`inputTokens`

`outputTokens`

`cacheReadTokens`

`cacheCreationTokens`

`reasoningTokens`

`requests`

`costUSD`

StreamEvent variants at a glance

`message`

`message`

`operation.started`

`operation.finished`

`operation.finished`

`skill.loaded`

`input.requested`

`thinking`

`context.injected`

`compaction`

`error`

Event table

Event details

`message` — what was said

Tool `operation.started` / `operation.finished` — which tool ran and what happened

Pass tool events to `ToolMatch`

Subagent `operation.started` / `operation.finished` — who was delegated to

`input.requested` — stop and wait for a person (HITL)

`thinking` / `compaction` / `error`

Three mapping rules

A complete mapping example

Related reading