Write a Custom Report

A custom Report never receives a Record path or looks up data while a page renders. Its fixed flow is:

core-only Sample → ReportScope → ReportPlan
                                  ↓
                              ReportInput
                                  ↓
                         ReportExecution (once)
                           ↙       ↓       ↘
                         show     view    export

Define required facts

Prefer the built-in requirements NiceEval exports. Define a custom JSON fact with defineJsonFact() and an explicit owner:

import { defineJsonFact } from "niceeval/report";

const attemptEnergy = defineJsonFact({
  id: "attempt-energy",
  owner: "attempt",
  name: "com.example.energy",
  parse(document) {
    if (typeof document.value !== "number") {
      throw new Error("energy must be a number");
    }
    return document.value;
  },
});

A custom name uses a reverse-domain namespace and cannot start with niceeval.. Its parser receives only a validated { observedAt, value } document, never bytes, descriptors, paths, or blobs. It runs once for each owner and requirement combination during execution and must return a JSON value synchronously.

Plan pages first

plan(scope) reads core-only slots only. It can enumerate detail routes by Run, slot, Attempt ID, and locator, but cannot read a Verdict, Usage, or another business value.

import {
  defineReport,
  verdict,
} from "niceeval/report";

export default defineReport({
  id: "quality",
  plan(scope) {
    return {
      calculations: [qualitySummary],
      pages: [
        {
          id: "overview",
          route: { pathname: "/", parameters: {} },
          title: "Overview",
          inputs: [verdict, attemptEnergy],
          calculations: [qualitySummary],
          render(input) {
            return overviewPage(input);
          },
        },
      ],
      downloads: [],
    };
  },
});

Each Page, Calculation, and Download declares its own inputs. Execution cannot read an undeclared fact later; the host reports that call as a plan-invalid error.

Set it as the project default

Import the definition directly in niceeval.config.ts:

import { defineConfig } from "niceeval";
import quality from "./reports/quality";

export default defineConfig({
  report: quality,
});

Afterward, show and view use it when --report is absent. Pass --report standard for one invocation to use the built-in Report again.

Define a Calculation

const qualitySummary = defineCalculation({
  id: "quality-summary",
  inputs: [verdict, attemptEnergy],
  completeness: "allowPartial",
  evaluate(input) {
    return calculateQuality(input);
  },
});

allowPartial can use successfully read values, but the result must retain the full Sample denominator, observed count, and a partial marker. requireComplete can be available only when durable collection and decoding are complete.

Read data from a page

A page can read only its inputs and Calculations:

render(input) {
  const rows = input.report.sample.included.map((slot) => ({
    locator: slot.locator,
    verdict: input.readAttempt(slot, verdict),
    energy: input.readAttempt(slot, attemptEnergy),
  }));

  return {
    title: "Overview",
    body: { rows },
    textAlternative: renderRowsAsText(rows),
  };
}

A page returns structured JSON and a text equivalent, not a React component, script, CSS, or file path. NiceEval’s built-in runtime provides the browser host.

Isolate failures

If page A requests a valid Verdict but page B requests damaged Usage:

A renders normally.
B becomes input-invalid with its channel and issues, and local view shows a named error page.
Other pages do not reread the Record.
Static export does not publish because a planned consumer failed.

When user rendering, a Calculation, or a Download throws, its state is execution-failed. It does not pretend to be invalid input.

Downloads

A Download is also a named consumer. It declares inputs and Calculations, obtains a constrained path through defineReportDownloadPath("downloads/..."), and returns a complete Uint8Array in the single execution phase. Static export writes those already-produced bytes; it does not run user code again.

Capability boundaries

A custom Report does not provide:

Arbitrary browser scripts, styles, fonts, workers, or WASM.
Runtime network requests.
File-path or blob access from a Report.
JSONL, append, or documents larger than 65,536 UTF-8 bytes through generic ctx.fact().

Large text and binary runtime facts use a NiceEval-approved, named Attempt channel and blob. A built-in decoder delivers an ordinary value.

Run and publish

npx niceeval show --run <run-id> --report ./reports/quality.ts
npx niceeval view --run <run-id> --report ./reports/quality.ts --no-open
npx niceeval view --run <run-id> --report ./reports/quality.ts --out ./quality-site

--out must point to a directory that does not exist. See Publish a Static Report and Report Components API for details.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Write a Custom Report

Define required facts

Plan pages first

Set it as the project default

Define a Calculation

Read data from a page

Isolate failures

Downloads

Capability boundaries

Run and publish

​Define required facts

​Plan pages first

​Set it as the project default

​Define a Calculation

​Read data from a page

​Isolate failures

​Downloads

​Capability boundaries

​Run and publish

Define required facts

Plan pages first

Set it as the project default

Define a Calculation

Read data from a page

Isolate failures

Downloads

Capability boundaries

Run and publish