Capabilities and evidence coverage

Capabilities answer two separate questions:

What can the runner construct or call? Agent construction and runtime behavior provide Sandbox access, tracing, session continuation, and HITL.
What conclusions can assertions trust? Every Agent declares the completeness of six evidence channels with the required evidenceCoverage field.

There is no separate capabilities field on defineAgent or defineSandboxAgent. That does not mean there is nothing to declare: evidenceCoverage is required on both definitions so missing evidence never becomes an implicit fourth state.

Construction and runtime capabilities

Capability	How it is obtained	Evidence implication
`t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions	Construct the Agent with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`)	Sandbox result assertions consume the Attempt’s agent-attributed diff, not an `evidenceCoverage` channel
`tracing` (OTLP receiver → `niceeval view` waterfall)	CLI-style agents configure `tracing`; long-running apps configure a fixed `telemetry` port	Spans feed the timing waterfall only; they never fill gaps in behavior evidence
Cross-turn continuation and `t.newSession()` isolation	Continue the backend session with `ctx.session.id` / `capture`, or store typed adapter-private history with `ctx.session.get` / `set`	Session wiring determines behavior; it is not inferred from an evidence bit
HITL (`t.respond()`)	`send` returns `status: "waiting"` plus `input.requested`	The next response is matched to the structured request ID
Compaction visibility	An official parser or a complete manual mapping emits a `compaction` event	`t.event("compaction")` also depends on the `events` coverage declared below

The required six-channel declaration

defineAgent and defineSandboxAgent both require an EvidenceCoverage value:

interface EvidenceCoverage {
  readonly events: EvidenceCoverageEntry;
  readonly actions: EvidenceCoverageEntry;
  readonly messages: EvidenceCoverageEntry;
  readonly usage: EvidenceCoverageEntry;
  readonly status: EvidenceCoverageEntry;
  readonly data: EvidenceCoverageEntry;
}

type EvidenceCoverageEntry =
  | { readonly status: "complete"; readonly reason?: never }
  | {
      readonly status: "partial" | "unavailable";
      readonly reason: string;
    };

Use completeEvidenceCoverage only when the adapter really captures every channel completely. Official converters expose a coverage declaration for the protocol they normalize; a manual mapping must declare all six channels honestly:

import type { EvidenceCoverage } from "niceeval/adapter";

const evidenceCoverage = {
  events: { status: "partial", reason: "the endpoint emits final events only" },
  actions: { status: "unavailable", reason: "tool lifecycle events are not exposed" },
  messages: { status: "complete" },
  usage: { status: "unavailable", reason: "the endpoint omits token usage" },
  status: { status: "complete" },
  data: { status: "complete" },
} satisfies EvidenceCoverage;

Turn.evidenceCoverage is an optional per-turn downgrade. List only channels that were worse than the Agent default for that turn, such as a stream that disconnected before usage arrived. Omitted channels inherit the Agent declaration; a Turn cannot upgrade it.

How coverage changes an assertion result

Coverage prevents missing telemetry from looking like proof that nothing happened:

Check	Complete required channel	Partial / unavailable required channel
Positive assertion finds matching evidence	`passed`	`passed` — evidence that exists is still evidence
Positive assertion finds no match	`failed`	`unavailable` — missing collection cannot prove the Agent did not do it
Negative assertion such as `notCalledTool()` / `notEvent()`	Evaluated normally	`unavailable`
Upper-bound assertion such as `maxTokens()` / `maxCost()`	Evaluated normally	`unavailable`

An unavailable assertion is always recorded with a machine-readable reason. It is never silently discarded or folded into a pass:

a non-optional unavailable assertion makes the Attempt errored;
an assertion explicitly chained with .optional() stays visible as unavailable but does not affect the Verdict.

Judge availability follows the same Verdict rule but has its own causes: an unresolved Judge model or API key, a failed Judge call, or an unparseable score produces unavailable. Judge is therefore not “always available.”

What happens when a manual mapping misses events

If a manual mapping claims events or actions are complete while omitting events, negative assertions can appear to pass on an incomplete picture. That is a false completeness declaration. Declare partial or unavailable with a reason until the mapping covers success, failure, rejection, and concurrent tool lifecycles. Official converters guarantee only the protocol surface stated by their contract. OTel spans cannot repair an incomplete event mapping: spans feed the waterfall, never assertions.

Events reference — the standard event stream contract.
Adapter concept — the full ctx / Turn contract.
defineAgent and defineSandboxAgent — Agent fields and examples.
Assertions — unavailable, .optional(), and Verdict propagation.
OTel integration — why spans feed visualization rather than assertions.
Built-in agent capabilities — what each built-in adapter captures.

API

Adapters

Results & Reports

CLI

Capabilities and evidence coverage

Construction and runtime capabilities

The required six-channel declaration

How coverage changes an assertion result

What happens when a manual mapping misses events

​Construction and runtime capabilities

​The required six-channel declaration

​How coverage changes an assertion result

​What happens when a manual mapping misses events

​Related reading

Construction and runtime capabilities

The required six-channel declaration

How coverage changes an assertion result

What happens when a manual mapping misses events

Related reading