Standard Event Stream Reference

An Adapter’s send returns a Turn, and its events: StreamEvent[] is this turn’s standard event stream; event assertions read their facts from it. turn.calledTool(...), turn.toolOrder(...), turn.event(...), and turn.succeeded() all judge from this turn’s events and the tool calls within them, and register an assertion directly when called. Translate what the agent did in this turn truthfully into the stream, and every event assertion works.

Turn: the return value of send

interface Turn {
  events: StreamEvent[];                          // this turn's events, in their actual order
  data?: JsonValue;                               // structured output, which t.check(...) can compare
  status: "completed" | "failed" | "waiting";     // waiting = paused for human input (HITL)
  usage?: Usage;                                  // → maxTokens / maxCost / cost reports; full fields below
}

data holds this turn’s structured output. Fill it only when the application’s answer itself is a structured object, such as extraction, classification, or form filling, so an Eval author can compare it with t.check(turn.data, ...). An application that returns only text leaves data unset. Do not copy the raw response body or message text into it, and do not serialize structured output into events. Fill usage when you have it and leave it empty when you do not; do not estimate a number to fill it. The full fields of usage (the Usage type):

`inputTokens`

inputTokens?: number;

Input tokens that missed the cache and are billed at full price; mutually exclusive with the two cache buckets.

`outputTokens`

outputTokens?: number;

The number of output (completion) tokens.

`cacheReadTokens`

cacheReadTokens?: number;

Input tokens served from a prompt-cache hit; a separate billing bucket, not included in inputTokens (omitted means the agent does not report it).

`cacheCreationTokens`

cacheCreationTokens?: number;

Input tokens written to the prompt cache; a separate billing bucket, not included in inputTokens (omitted means the agent does not report it).

`reasoningTokens`

reasoningTokens?: number;

The number of reasoning (thinking) tokens, a detail already included in outputTokens and listed separately for display; present only when the protocol actually provides it.

`requests`

requests?: number;

The number of model requests that actually occurred. Omit it when the protocol does not provide a request count; never write 1 to fill it.

`costUSD`

costUSD?: number;

The actual USD cost measured by the gateway/adapter (it can only be brought back explicitly through Turn.usage.costUSD, and is never derived from token usage or OTel spans). It and the top-level estimatedCostUSD (a price-table estimate) are two independent facts under a one-way field contract: this field stores only the observed value; estimatedCostUSD always equals the estimate from estimateCost(model, usage, pricing), computed independently even when an observed value exists. Neither overrides or backs up the other; an observed value never replaces or triggers the estimate.

How cost is calculated: Usage.costUSD records only the USD cost actually returned by a Provider or Adapter; it is never derived from token counts, a model catalog, or a local price table. Whether or not that value exists, the Runner separately computes an estimated cost, estimatedCostUSD, from the model, the reported token usage, and the configured price table. The Experiment budget and t.maxCost() look at this estimate, not Usage.costUSD. Costs in reports are computed only from a PricingProfile you specify explicitly and the recorded usage; they do not read the Runner’s estimate either.

StreamEvent variants at a glance

StreamEvent has ten variants, listed field by field below. For which assertions read them and what to watch for when using them, see the “Event table” and “Event details” sections that follow.

`message`

{ type: "message"; role: "assistant"; text: string; loc?: SourceLoc }

`message`

{ type: "message"; role: "user"; text: string; loc?: SourceLoc; sourceOrder?: number }

`operation.started`

{
  type: "operation.started";
  operationId: string;
  operation:
    | {
        kind: "tool";
        name: string;
        input: JsonValue;
        tool?: ToolName;
        /**
         * Adapter 对 command / not-command 的协议级分类。暂时可选以兼容尚未迁移的
         * 第三方 Adapter；缺失不得在 core 侧由 name、input 或 shell text 补造。
         */
        command?: CommandProjection;
      }
    | { kind: "subagent"; name: string; remoteUrl?: string };
}

`operation.finished`

{
  type: "operation.finished";
  operationId: string;
  kind: "tool";
  output?: JsonValue;
  status: "completed" | "failed" | "rejected";
}

`operation.finished`

{
  type: "operation.finished";
  operationId: string;
  kind: "subagent";
  output?: JsonValue;
  status: "completed" | "failed";
}

`skill.loaded`

{ type: "skill.loaded"; skill: string; operationId?: string }

`input.requested`

{ type: "input.requested"; request: InputRequest }

`thinking`

{ type: "thinking"; text: string }

`context.injected`

{ type: "context.injected"; text: string; source?: string }

`compaction`

{ type: "compaction"; reason?: string }

`error`

{ type: "error"; message: string }

Event table

Event	What it says	Assertion / API that reads it
`message`	The agent (or user) said something	`turn.reply`, `turn.event(eventMatch("message", ...))`, Judge material
`operation.started` (`kind: "tool"`)	Starts a tool call	`turn.calledTool(...)`, `turn.toolOrder(...)`, `turn.event(...)`
`operation.finished` (`kind: "tool"`)	The result of that tool call	`ToolMatch` with a status condition
`operation.started` (`kind: "subagent"`)	Delegates to a subagent	Kept in raw events and reports; there is no subagent-specific assertion
`operation.finished` (`kind: "subagent"`)	The subagent returns	Kept in raw events and reports; a failure counts toward `noFailedActions()`
`input.requested`	Stops to wait for human input (HITL)	`t.check(turn.status, equals("waiting"))`, `t.requireInputRequest()`
`thinking`	Thinking text	Shown in the view; check it with `t.check(turn.events, ...)` when needed
`compaction`	Context was compacted	Shown in the view; check it with `t.check(turn.events, ...)` when needed
`error`	This turn failed	Shown in the view; check it with `t.check(turn.events, ...)` when needed

Event assertions have three entry points: event(match), notEvent(match), and eventOrder(matches), all of which take a match condition built with eventMatch(...). eventMatch supports three events: message, tool operation.started, and tool operation.finished. To require a number of occurrences, use the match condition’s own methods such as .atLeast() and .exactly(). For other events, treat turn.events as an ordinary value and check it with check(turn.events, valueMatch).

Event details

`message` — what was said

{ type: "message", role: "assistant" | "user", text: string }

Each segment of assistant text produces one message with role: "assistant". A tool result is not an assistant message; do not wrap tool output in message, or t.reply will read the wrong content. NiceEval records user-input message events automatically, so the Adapter does not need to emit them.

Tool `operation.started` / `operation.finished` — which tool ran and what happened

{ type: "operation.started", operationId: string,
  operation: { kind: "tool", name: string, input: JsonValue,
    tool?: ToolName, command?: CommandProjection } }
{ type: "operation.finished", operationId: string, kind: "tool",
  output?: JsonValue, status: "completed" | "failed" | "rejected" }

Every tool operation.started needs a tool operation.finished with the same operationId; concurrent calls are matched up by this ID. Use the ID already in the agent’s return when there is one, such as AI SDK’s toolCallId or Anthropic’s tool_use.id; generate one by order only when there is none.
Set status to what actually happened: "failed" when the tool execution failed, "rejected" when a person rejected it. Eval authors use ToolMatch status conditions to tell the two apart, so do not mix them up.
Set name to the tool’s original name.
With only started and no finished yet, the call’s status is pending and its output is recorded as unavailable. A finished event without output also means the output is unavailable. Do not write missing output as empty JSON, and do not let it look like an ordinary mismatch.

Pass tool events to `ToolMatch`

The same shell tool might first run git status and then pnpm test. The tool name only says which kind of tool it is. To tell the specific commands apart, Eval authors match on the normalized command information (command projection):

import { commandMatch } from "niceeval/expect";

turn.calledTool(commandMatch("pnpm", { argsStart: ["test"] }));

ToolMatch compares one tool call at a time. For the complete filter conditions, and the JSON, path, output, and counting rules, see Scoped assertions. When the native protocol directly provides a command’s structured argv, generate the command information with the constructor exported from niceeval/adapter:

import { commandProjection } from "niceeval/adapter";

events.push({
  type: "operation.started",
  operationId: call.id,
  operation: {
    kind: "tool",
    name: call.name,
    input: call.input,
    command: commandProjection({
      state: "available",
      executable: call.executable,
      args: call.args,
    }),
  },
});

commandProjection() keeps the original arguments the Adapter confirmed and normalizes them into the logical-command/v1 format by NiceEval’s shared rules. For example, pnpm exec niceeval view is matched exactly by commandMatch("niceeval", { argsStart: ["view"] }). Mark the original command available (state: "available") only when the native protocol itself provides argv, or when one command can be split out unambiguously by the protocol’s grammar. When the protocol provides only shell source, or the content is truncated or redacted, use opaqueCommandProjection(reason). When you can confirm the call is not a command, use notCommandProjection(). When you cannot tell whether it is a command, downgrade actions coverage; do not guess from the tool name, input, or shell text.

Subagent `operation.started` / `operation.finished` — who was delegated to

{ type: "operation.started", operationId: string,
  operation: { kind: "subagent", name: string, remoteUrl?: string } }
{ type: "operation.finished", operationId: string, kind: "subagent",
  output?: JsonValue, status: "completed" | "failed" }

When the system under test delegates a task to a subagent and waits for it to return, emit this pair; the operationId pairing rule is the same as above. They stay in the raw event stream and reports. To check them, use an ordinary value match on turn.events; the tool matching in eventMatch(...) does not treat a subagent as a tool call.

`input.requested` — stop and wait for a person (HITL)

{ type: "input.requested", request: {
    id?: string,
    action?: string,        // the action where it paused (such as a tool name)
    input?: JsonValue,      // the action's input
    prompt?: string,        // the question for the person
    options?: { id: string, label?: string }[],   // choices (approve / deny…)
} }

When the agent stops to wait for a person, each question awaiting an answer produces one input.requested, and the Turn’s status is "waiting". t.requireInputRequest(filter) matches request field by field, so fill in as many fields as you can, or the Eval cannot select it. For how to wire it up, see the HITL section of the connection tutorial.

`thinking` / `compaction` / `error`

{ type: "thinking", text: string }
{ type: "compaction", reason?: string }   // context compaction; emitted when the Adapter parses it
{ type: "error", message: string }

Emit them when they exist; do not make them up when they do not. compaction mostly comes from coding agent CLIs, which compact automatically when the context fills up. These events are mainly for reports and diagnostics; when you need to assert on them, pass turn.events to check as an ordinary value.

Three rules for mapping events

Keep the real order: event order is the order things happened. toolOrder and eventOrder match different calls from front to back, so the wrong order gives the wrong assertion result. eventOrder can only order events that eventMatch supports.
Pair with operationId: every started needs a finished with the same kind and ID. Only with complete pairs can NiceEval attribute status and input to the same tool call.
Declare completeness honestly: official converters declare event coverage (evidence coverage) according to what the Adapter can actually do. When the relevant input or action coverage is incomplete, a negative assertion (such as “did not call a tool”) results in unavailable; “not seen” is never treated as “did not happen.” Manual mappings must also declare coverage honestly; see the Capabilities reference.

A complete mapping example

When the agent’s return includes step records, the mapping is a simple loop:

import type { StreamEvent } from "niceeval";

function toStreamEvents(body: MyAgentResponse): StreamEvent[] {
  const events: StreamEvent[] = [];
  for (const step of body.steps) {
    if (step.type === "tool_call") {
      events.push({
        type: "operation.started",
        operationId: step.id,
        operation: { kind: "tool", name: step.tool, input: step.args },
      });
      events.push({
        type: "operation.finished",
        operationId: step.id,
        kind: "tool",
        output: step.result,
        status: step.error ? "failed" : "completed",
      });
    }
    if (step.type === "text") events.push({ type: "message", role: "assistant", text: step.text });
  }
  return events;
}

Connect your agent — the tutorial for getting it working from scratch.
Capabilities — what declaring “the event stream is complete” means.
Authoring Evals — the full set of assertions that read this stream.

Turn: the return value of send

inputTokens

outputTokens

cacheReadTokens

cacheCreationTokens

reasoningTokens

requests

costUSD

StreamEvent variants at a glance

message

message

operation.started

operation.finished

operation.finished

skill.loaded

input.requested

thinking

context.injected

compaction

error

Event table

Event details

message — what was said

Tool operation.started / operation.finished — which tool ran and what happened

Pass tool events to ToolMatch

Subagent operation.started / operation.finished — who was delegated to

input.requested — stop and wait for a person (HITL)

thinking / compaction / error

Three rules for mapping events

A complete mapping example

Related reading

`inputTokens`

`outputTokens`

`cacheReadTokens`

`cacheCreationTokens`

`reasoningTokens`

`requests`

`costUSD`

`message`

`message`

`operation.started`

`operation.finished`

`operation.finished`

`skill.loaded`

`input.requested`

`thinking`

`context.injected`

`compaction`

`error`

`message` — what was said

Tool `operation.started` / `operation.finished` — which tool ran and what happened

Pass tool events to `ToolMatch`

Subagent `operation.started` / `operation.finished` — who was delegated to

`input.requested` — stop and wait for a person (HITL)

`thinking` / `compaction` / `error`