Capabilities and Evidence Coverage Reference

Capabilities answer two different questions:

What can the Runner construct or call? An Agent’s construction and runtime behavior provide Sandbox access, tracing, session continuation, and HITL.
What conclusions can assertions trust? Every Agent must use evidenceCoverage to declare the completeness of six evidence channels.

There is no separate capabilities field on defineAgent / defineSandboxAgent. That does not mean “nothing needs to be declared”: evidenceCoverage is required on both definitions so missing evidence never becomes an ambiguous fourth state.

Construction and runtime capabilities

Capability	How to obtain it	Effect on evidence
`t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions	Construct with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`)	Sandbox-result assertions consume the Attempt’s agent-attributed diff; they do not correspond to an `evidenceCoverage` channel
`tracing` (OTLP receiver → `niceeval view` waterfall)	Configure `tracing` for a CLI-style Agent. Configure a fixed `telemetry` port for a long-running application	Spans go only to the timing waterfall; they cannot fill a gap in behavior evidence
Cross-turn continuation and `t.newSession()` isolation	Continue the backend session with `ctx.session.id` / `capture`, or save typed adapter-private history with `ctx.session.get` / `set`	The session connection determines actual behavior; do not infer it from an evidence bit
HITL (`t.respond()`)	`send` returns `status: "waiting"` and `input.requested`	The next response is matched by structured request ID
Compaction visibility	An official parser or complete manual mapping emits a `compaction` event	Raw events go to reports and diagnostics; there is currently no public `eventMatch` selector

The required six-channel declaration

Both defineAgent and defineSandboxAgent require an EvidenceCoverage:

interface EvidenceCoverage {
  readonly events: EvidenceCoverageEntry;
  readonly actions: EvidenceCoverageEntry;
  readonly messages: EvidenceCoverageEntry;
  readonly usage: EvidenceCoverageEntry;
  readonly status: EvidenceCoverageEntry;
  readonly data: EvidenceCoverageEntry;
}

type EvidenceCoverageEntry =
  | { readonly status: "complete"; readonly reason?: never }
  | {
      readonly status: "partial" | "unavailable";
      readonly reason: string;
    };

Use completeEvidenceCoverage only when the Adapter really captures every channel completely. An official converter provides its coverage declaration for the protocol it normalizes. A manual mapping must state all six channels honestly:

import type { EvidenceCoverage } from "niceeval/adapter";

const evidenceCoverage = {
  events: { status: "partial", reason: "the endpoint emits final events only" },
  actions: { status: "unavailable", reason: "the endpoint does not expose tool lifecycles" },
  messages: { status: "complete" },
  usage: { status: "unavailable", reason: "the endpoint does not return token usage" },
  status: { status: "complete" },
  data: { status: "complete" },
} satisfies EvidenceCoverage;

Turn.evidenceCoverage is an optional per-turn downgrade. List only channels that are worse for that turn than the Agent default, such as when a stream ends before usage arrives. Omitted channels inherit the Agent declaration. A Turn cannot upgrade the default coverage.

How coverage changes assertion results

Coverage prevents “not collected” from looking like “confirmed not to have happened”:

Check	Required channel is complete	Required channel is partial / unavailable
A positive assertion finds matching evidence	`passed`	`passed` — evidence that exists is still valid
A positive assertion finds no match	`failed`	`unavailable` — a collection gap cannot prove the Agent did not act
A negative assertion such as `notCalledTool()` / `notEvent()`	Evaluated normally	`unavailable`
An upper-bound assertion such as `maxTokens()` / `maxCost()`	Evaluated normally	`unavailable`

An unavailable assertion is always persisted with a machine-readable reason. It is never silently discarded or folded into a pass:

A non-optional unavailable assertion makes the Attempt errored.
An assertion explicitly chained with .optional() remains visible as unavailable, but does not affect the Verdict.

Judge follows the same Verdict rule, but has its own unavailable reasons: an unresolved Judge model or API key, a failed Judge call, or a response without an extractable score all produce unavailable. Judge is therefore not “always available.”

What happens when a manual mapping misses events

If a manual mapping misses events but declares events or actions complete, negative assertions can appear to pass on an incomplete picture. That is an incorrect completeness declaration. Until the mapping covers successful, failed, rejected, and concurrent tool lifecycles, declare partial or unavailable honestly and give a reason. An official converter guarantees only the surface its protocol contract explicitly covers. OTel spans cannot repair an incomplete event mapping: spans go only to the waterfall, not assertions.

Event stream reference — the standard event stream contract.
Adapter concepts — the full ctx / Turn contract.
defineAgent and defineSandboxAgent — Agent fields and examples.
Assertions — unavailable, .optional(), and Verdict propagation.
OTel integration — why spans feed visualization only, not assertions.
Built-in Agent capabilities — what each built-in Adapter actually captures.

​Construction and runtime capabilities

​The required six-channel declaration

​How coverage changes assertion results

​What happens when a manual mapping misses events

​Related reading

Construction and runtime capabilities

The required six-channel declaration

How coverage changes assertion results

What happens when a manual mapping misses events

Related reading