> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Capabilities and evidence coverage

> How construction, runtime behavior, and the required six-channel evidenceCoverage declaration determine which NiceEval assertions can be trusted.

Capabilities answer two separate questions:

1. **What can the runner construct or call?** Agent construction and runtime behavior provide Sandbox access, tracing, session continuation, and HITL.
2. **What conclusions can assertions trust?** Every Agent declares the completeness of six evidence channels with the required `evidenceCoverage` field.

There is no separate `capabilities` field on `defineAgent` or `defineSandboxAgent`. That does **not** mean there is nothing to declare: `evidenceCoverage` is required on both definitions so missing evidence never becomes an implicit fourth state.

## Construction and runtime capabilities

| Capability                                                              | How it is obtained                                                                                                                    | Evidence implication                                                                                     |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions | Construct the Agent with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`)                                                    | Sandbox result assertions consume the Attempt's agent-attributed diff, not an `evidenceCoverage` channel |
| `tracing` (OTLP receiver → `niceeval view` waterfall)                   | CLI-style agents configure `tracing`; long-running apps configure a fixed `telemetry` port                                            | Spans feed the timing waterfall only; they never fill gaps in behavior evidence                          |
| Cross-turn continuation and `t.newSession()` isolation                  | Continue the backend session with `ctx.session.id` / `capture`, or store typed adapter-private history with `ctx.session.get` / `set` | Session wiring determines behavior; it is not inferred from an evidence bit                              |
| HITL (`t.respond()`)                                                    | `send` returns `status: "waiting"` plus `input.requested`                                                                             | The next response is matched to the structured request ID                                                |
| Compaction visibility                                                   | An official parser or a complete manual mapping emits a `compaction` event                                                            | `t.event("compaction")` also depends on the `events` coverage declared below                             |

## The required six-channel declaration

`defineAgent` and `defineSandboxAgent` both require an `EvidenceCoverage` value:

```ts theme={null}
interface EvidenceCoverage {
  readonly events: EvidenceCoverageEntry;
  readonly actions: EvidenceCoverageEntry;
  readonly messages: EvidenceCoverageEntry;
  readonly usage: EvidenceCoverageEntry;
  readonly status: EvidenceCoverageEntry;
  readonly data: EvidenceCoverageEntry;
}

type EvidenceCoverageEntry =
  | { readonly status: "complete"; readonly reason?: never }
  | {
      readonly status: "partial" | "unavailable";
      readonly reason: string;
    };
```

Use `completeEvidenceCoverage` only when the adapter really captures every channel completely. Official converters expose a coverage declaration for the protocol they normalize; a manual mapping must declare all six channels honestly:

```ts theme={null}
import type { EvidenceCoverage } from "niceeval/adapter";

const evidenceCoverage = {
  events: { status: "partial", reason: "the endpoint emits final events only" },
  actions: { status: "unavailable", reason: "tool lifecycle events are not exposed" },
  messages: { status: "complete" },
  usage: { status: "unavailable", reason: "the endpoint omits token usage" },
  status: { status: "complete" },
  data: { status: "complete" },
} satisfies EvidenceCoverage;
```

`Turn.evidenceCoverage` is an optional per-turn **downgrade**. List only channels that were worse than the Agent default for that turn, such as a stream that disconnected before usage arrived. Omitted channels inherit the Agent declaration; a Turn cannot upgrade it.

## How coverage changes an assertion result

Coverage prevents missing telemetry from looking like proof that nothing happened:

| Check                                                       | Complete required channel | Partial / unavailable required channel                                  |
| ----------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- |
| Positive assertion finds matching evidence                  | `passed`                  | `passed` — evidence that exists is still evidence                       |
| Positive assertion finds no match                           | `failed`                  | `unavailable` — missing collection cannot prove the Agent did not do it |
| Negative assertion such as `notCalledTool()` / `notEvent()` | Evaluated normally        | `unavailable`                                                           |
| Upper-bound assertion such as `maxTokens()` / `maxCost()`   | Evaluated normally        | `unavailable`                                                           |

An `unavailable` assertion is always recorded with a machine-readable reason. It is never silently discarded or folded into a pass:

* a non-optional unavailable assertion makes the Attempt `errored`;
* an assertion explicitly chained with `.optional()` stays visible as unavailable but does not affect the Verdict.

Judge availability follows the same Verdict rule but has its own causes: an unresolved Judge model or API key, a failed Judge call, or an unparseable score produces `unavailable`. Judge is therefore not "always available."

## What happens when a manual mapping misses events

If a manual mapping claims `events` or `actions` are complete while omitting events, negative assertions can appear to pass on an incomplete picture. That is a false completeness declaration. Declare `partial` or `unavailable` with a reason until the mapping covers success, failure, rejection, and concurrent tool lifecycles.

Official converters guarantee only the protocol surface stated by their contract. OTel spans cannot repair an incomplete event mapping: spans feed the waterfall, never assertions.

## Related reading

* [Events reference](/docs/reference/events) — the standard event stream contract.
* [Adapter concept](/docs/explanation/adapter) — the full `ctx` / `Turn` contract.
* [defineAgent and defineSandboxAgent](/docs/reference/define-agent) — Agent fields and examples.
* [Assertions](/docs/explanation/assert) — unavailable, `.optional()`, and Verdict propagation.
* [OTel integration](/docs/tutorials/connect-otel) — why spans feed visualization rather than assertions.
* [Built-in agent capabilities](/docs/reference/builtin-agents) — what each built-in adapter captures.