defineEval: declare, configure, and run NiceEval evals

defineEval is the main entry point for authoring evals. Each eval file calls it once, passing a description and test(t), and default-exports the result.

import { defineEval } from "niceeval";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    const turn = await t.send("What's the weather like in Brooklyn today?");
    await turn.succeeded().stopOnFailure();
  },
});

Do not provide id or name. NiceEval derives the eval ID from the file path.

`defineEval` options

`id`

id?: string;

Derived from the file path; hand-writing it in the definition is not allowed.

`description`

description?: string;

One-line description, shown in niceeval list and the view; purely informational — it does not affect scheduling or assertion evaluation.

`tags`

tags?: string[];

Tags, for the CLI’s --tag filter and view categorization; a filtering dimension independent from ID-prefix filtering.

`environment`

environment?: string;

The environment profile id this eval needs (provider-neutral, e.g. "python-3.9-astropy-4.2"); translated into that provider’s prebuilt artifact via the Sandbox spec’s environments table.

`judge`

judge?: JudgeConfig;

Overrides the project-level Config.judge, taking effect only for this one eval (e.g. to switch to a more expensive judge model).

`reporters`

reporters?: Reporter[];

Overrides / adds to the project-level Config.reporters, taking effect only for this one eval.

`timeoutMs`

timeoutMs?: number;

Overrides the project-level / CLI timeout for a single attempt (milliseconds), taking effect only for this one eval.

`metadata`

metadata?: Record<string, JsonValue>;

Arbitrary extra metadata, saved as Attempt Provenance; does not participate in scheduling or assertion evaluation, and is meant for custom reporters to consume.

`diff`

diff?: { include?: string[]; ignore?: string[] };

Adjusts the attribution exclusion list for the agent diff (Sandbox agents only; see docs/feature/eval/README.md): both arrays are gitignore-style globs (relative to workdir). By default, .git, node_modules, build artifacts, and package-manager caches are excluded; ignore adds further exclusions on top of the default list; include has the highest priority and explicitly adds matching paths back. The composition rule is fixed as “default ∪ ignore, then punched through by include”; the list is frozen at the ledger’s anchor point.

`setup`

setup?: (sandbox: Sandbox, ctx: SandboxHookContext) => Promise<void> | void;

Eval-level preparation: receives the Sandbox (after the workspace is uploaded and the git baseline is set, before dependencies are installed). Commands default to the environment’s own declared identity (the Docker image USER, the Compose service user:, and so on); when installing system dependencies, pass { user: "root" } to runCommand (e.g. runCommand("apt-get", ["install", …], { user: "root" })) — semantics are consistent across providers. The second parameter is a narrow context bound to eval.setup (ctx.progress / ctx.diagnostic, see docs/feature/eval/README.md). setup returns no value; to pass an artifact through to teardown, key it off the sandbox instance (concurrent attempts share the same module, so a plain module-level variable would get overwritten by other attempts).

`teardown`

teardown?: (sandbox: Sandbox, ctx: SandboxHookContext) => Promise<void> | void;

Eval-level cleanup: the first link in the attempt’s teardown chain (eval.teardown → agent.teardown → sandbox.teardown), with the Sandbox still alive at this point. Runs if and only if eval.setup’s point in time was reached — an error thrown by setup or test does not exempt it, and not declaring setup does not affect whether it fires. An error thrown, or exceeding the 30s cleanup budget, only records a teardown-failed diagnostic and does not change the verdict. Use it for temporary Fixtures outside the Sandbox (a temporary repo / bucket); things inside the Sandbox are reclaimed automatically when it’s destroyed and don’t need this.

`test`

test(t: TestContext): Promise<void> | void;

The eval body: receives the TestContext, drives the conversation / Sandbox operations, and asserts in place.

Test context: `t`

t (TestContext) is the high-level context the eval author receives. The runner assembles it according to the agent’s actual capabilities — fields like t.sandbox are only meaningful on agents constructed with defineSandboxAgent; the full rule set is in the capabilities reference. All members:

`send`

send(input: SendInput): Promise<TurnHandle>;

Sends a message on the default session (a string or a structured message), returning that turn’s TurnHandle. The events are also accumulated into the default session’s cumulative event stream, for the scoped assertions below to use.

`sendFile`

sendFile(path: string, text?: string): Promise<TurnHandle>;

Sends a message with an attached file (image, etc. — multimodal input). path is relative to the project root; after reading, it is base64-encoded and handed to the adapter via TurnInput.files.

`requireInputRequest`

requireInputRequest(filter?: InputRequestFilter): InputRequest;

Retrieves a pending HITL input request on the default session; without a filter it requires exactly one, and throws if it can’t get one.

`respond`

respond(...responses: (string | RespondAnswer)[]): Promise<TurnHandle>;

Answers the pending input request(s) on the default session, returning the continuation’s TurnHandle. String form matches requests in order; when multiple requests are paused together and you need to name which one you’re answering, use the RespondAnswer object form (see its type comment).

`respondAll`

respondAll(optionId: string): Promise<TurnHandle>;

Answers every pending input request on the default session in bulk, using the same optionId.

`reply`

readonly reply: string;

The assistant’s reply text for the default session’s most recent turn.

`sessionId`

readonly sessionId: string | undefined;

The adapter-side default session id (only present for agents with a session concept).

`events`

readonly events: readonly StreamEvent[];

The default session’s cumulative event stream (across every turn on that session).

`newSession`

newSession(): SessionHandle;

Opens another, independent session line, returning its SessionHandle — not void. This is the key entry point for multi-session isolation: sending and asserting on the new session are isolated from the default session and from every other newSession() session — commonly used for scenarios like “multiple users in parallel conversations” or “one main line plus one side branch.”

`signal`

readonly signal: AbortSignal;

The abort signal for this attempt; fires on timeout / EarlyExit / user Ctrl-C, and is passed to the adapter’s long-running calls for cancellation.

`model`

readonly model?: string;

The model name used for this attempt (decided by the experiment/CLI flag); omitted means the agent’s native default, not “no model.”

`reasoningEffort`

readonly reasoningEffort?: string;

The reasoning effort for this attempt (e.g. “low”/“medium”/“high”; valid values are decided by the adapter/model).

`flags`

readonly flags: Readonly<Record<string, JsonValue>>;

The experiment flags in effect for this attempt (a read-only view of experiment.flags; an experimental condition, not a command-line switch).

`progress`

progress(update: ProgressUpdate): void;

Scoped feedback: reports a long-running step the eval itself is performing (uploading a Fixture, running a build, …). A short-lived status with scope fixed to eval.run; it only reports, it does not assert (see docs/feature/eval/library/context.md, “Reporting long steps back to the run”).

`diagnostic`

diagnostic(input: DiagnosticInput): void;

Scoped feedback: reports an issue that should be kept around after the run ends (a permanent event, recorded in the attempt’s diagnostics). Even when level is “error” this does not automatically change the verdict — the test’s conclusion is still decided by assertions.

`log`

log(msg: string): void;

An alias for progress({ message: msg }) (debug logging); does not appear in the final result.

`skip`

skip(reason: string): never;

Immediately aborts this eval and forms a skipped Verdict in the niceeval.verdict channel; reason cannot be empty.

`check`

check(value: unknown, assertion: ValueAssertion): AssertionHandle;

Runs a ValueAssertion against any value, returning an AssertionHandle that can be chained with .gate() / .atLeast(). Assertion evaluation is deferred until finalize at the end of the eval; the call itself is synchronous and never throws — a failure just records a failed assertion, it does not abort the rest of the code. Use require for “abort the eval immediately if this isn’t met.”

`require`

require<T>(value: T, assertion: ValueAssertion): Promise<T>;

Runs a ValueAssertion against any value, evaluated immediately (when awaited); if it isn’t met, throws and aborts the rest of the eval’s steps (the assertion is still recorded in the report, and does not affect other already-recorded assertions). The difference from check: check only records and never throws, with assertion evaluation left to the end; require evaluates on the spot and aborts on failure — suited to “if this prerequisite isn’t met, nothing written after it matters.” On success it returns the original value with its type preserved.

`group`

group<T>(title: string, fn: () => Promise<T> | T): Promise<T>;

Groups a set of assertions under a titled section (analogous to vitest’s test('title', ...)). Purely for organization/reporting — it does not change assertion evaluation: every assertion inside the group is still evaluated independently. Can be nested (titles joined with ›).

`succeeded`

succeeded(): AssertionHandle;

Asserts the default session’s cumulative status is “completed” (across every turn on that session, not just the last one).

`parked`

parked(): AssertionHandle;

Asserts the default session is currently stuck on an HITL input request.

`messageIncludes`

messageIncludes(token: string | RegExp): AssertionHandle;

Asserts the default session’s cumulative assistant replies contain token (across every turn on that session, not just the last one).

`calledTool`

calledTool(name: string, match?: ToolMatch): AssertionHandle;

Asserts the default session has cumulatively called the named tool; match can constrain arguments / count / status.

`notCalledTool`

notCalledTool(name: string, match?: ToolMatch): AssertionHandle;

Asserts the default session has not cumulatively called the named tool (or not under the match condition).

`toolOrder`

toolOrder(names: string[]): AssertionHandle;

Asserts the default session’s cumulative tool calls appear in the given order (other calls may be interleaved).

`usedNoTools`

usedNoTools(): AssertionHandle;

Asserts the default session has not called any tool so far.

`maxToolCalls`

maxToolCalls(max: number): AssertionHandle;

Asserts the default session’s cumulative tool call count does not exceed max.

`loadedSkill`

loadedSkill(skill: string): AssertionHandle;

Asserts the default session has cumulatively loaded the named skill.

`noFailedActions`

noFailedActions(): AssertionHandle;

Asserts the default session has no cumulative failed tool calls / commands.

`event`

event(type: StreamEvent["type"], opts?: { count?: number }): AssertionHandle;

Asserts the default session has cumulatively seen an event of the given type; opts.count can constrain the occurrence count.

`notEvent`

notEvent(type: StreamEvent["type"]): AssertionHandle;

Asserts the default session has not cumulatively seen an event of the given type.

`calledSubagent`

calledSubagent(name: string, match?: SubagentMatch): AssertionHandle;

Asserts the default session has cumulatively called the named subagent; match can constrain count / status / remoteUrl.

`eventOrder`

eventOrder(types: StreamEvent["type"][]): AssertionHandle;

Asserts the default session’s cumulative events appear in the given type order (other events may be interleaved).

`eventsSatisfy`

eventsSatisfy(label: string, predicate: (events: readonly StreamEvent[]) => boolean): AssertionHandle;

Asserts against the default session’s whole cumulative event stream using a custom predicate; label is required and becomes the assertion’s title.

`sandbox`

readonly sandbox: SandboxHandle;

A restricted Sandbox view: can run commands / read and write files / view the final diff, but cannot stop the Sandbox itself (see SandboxHandle).

`usage`

readonly usage: Usage;

The default session’s cumulative token usage and estimated cost.

`maxTokens`

maxTokens(max: number): AssertionHandle;

Asserts the default session’s cumulative token usage does not exceed max.

`maxCost`

maxCost(usd: number): AssertionHandle;

Asserts the default session’s cumulative spend (USD) does not exceed usd.

`judge`

readonly judge: JudgeNamespace;

The available judge namespace (t.judge.autoevals.*).

Judge assertions

t.judge.autoevals.factuality(expected, { on: t.reply }).atLeast(0.8);
t.judge.autoevals.closedQA(question, { on: t.reply }).atLeast(0.7);
t.judge.autoevals.closedQA(rubric, { on: t.reply }).atLeast(0.75);
t.judge.autoevals.summarizes(sourceText, { on: t.reply }).atLeast(0.7);

t.judge.autoevals currently has three methods: factuality (whether the reply is consistent with the expected facts), closedQA (whether the reply satisfies the given question or rubric), and summarizes (whether the reply is a valid summary of the given source text source). All three directly use scorers from the autoevals (braintrust) library; { on: t.reply } specifies the text being scored.

`Turn` return type

t.send(...) returns a TurnHandle: convenience fields derived from the event stream, plus a full set of turn-scoped assertions.

`events`

readonly events: StreamEvent[];

This turn’s raw event stream (tool calls, message deltas, etc.); all the derived fields below are computed from it.

`toolCalls`

readonly toolCalls: readonly ToolCall[];

List of tools called within this turn, derived from events.

`status`

readonly status: "completed" | "failed" | "waiting";

This turn’s end status: “completed” ended normally, “failed” errored, “waiting” stuck on an HITL input request.

`message`

readonly message: string;

The assistant’s final text reply for this turn (the concatenation of the message deltas in events).

`data`

readonly data?: JsonValue;

Structured output attached by the adapter (if any), for outputEquals / outputMatches to compare against.

`usage`

readonly usage?: Usage;

This turn’s token usage and estimated cost (present only for agents that reported usage).

`outputEquals`

outputEquals(value: unknown): AssertionHandle;

Asserts data deep-equals the given value.

`outputMatches`

outputMatches(schema: unknown): AssertionHandle;

Asserts data satisfies the given schema (e.g. a zod schema).

`messageIncludes`

messageIncludes(token: string | RegExp): AssertionHandle;

Asserts this turn’s assistant reply contains token (limited to this turn’s event stream, not across turns).

`succeeded`

succeeded(): AssertionHandle;

Asserts this turn’s status is “completed”.

`parked`

parked(): AssertionHandle;

Asserts this turn is stuck on an HITL input request (status is “waiting”).

`calledTool`

calledTool(name: string, match?: ToolMatch): AssertionHandle;

Asserts this turn called the named tool; match can further constrain arguments / count / status.

`notCalledTool`

notCalledTool(name: string, match?: ToolMatch): AssertionHandle;

Asserts this turn did not call the named tool (or not under the match condition).

`toolOrder`

toolOrder(names: string[]): AssertionHandle;

Asserts this turn’s tool calls appear in the given order (other calls may be interleaved).

`usedNoTools`

usedNoTools(): AssertionHandle;

Asserts this turn did not call any tool.

`maxToolCalls`

maxToolCalls(max: number): AssertionHandle;

Asserts this turn’s total tool call count does not exceed max.

`loadedSkill`

loadedSkill(skill: string): AssertionHandle;

Asserts this turn loaded the named skill.

`noFailedActions`

noFailedActions(): AssertionHandle;

Asserts this turn has no failed tool calls / commands.

`event`

event(type: StreamEvent["type"], opts?: { count?: number }): AssertionHandle;

Asserts this turn saw an event of the given type; opts.count can constrain the occurrence count.

`notEvent`

notEvent(type: StreamEvent["type"]): AssertionHandle;

Asserts this turn did not see an event of the given type.

`calledSubagent`

calledSubagent(name: string, match?: SubagentMatch): AssertionHandle;

Asserts this turn called the named subagent; match can constrain count / status / remoteUrl.

`eventOrder`

eventOrder(types: StreamEvent["type"][]): AssertionHandle;

Asserts this turn’s events appear in the given type order (other events may be interleaved).

`eventsSatisfy`

eventsSatisfy(label: string, predicate: (events: readonly StreamEvent[]) => boolean): AssertionHandle;

Asserts against this turn’s whole event stream using a custom predicate; label is required and becomes the assertion’s title (the predicate itself is opaque, so the burden of explanation falls on label).

`maxTokens`

maxTokens(max: number): AssertionHandle;

Asserts this turn’s token usage does not exceed max.

`maxCost`

maxCost(usd: number): AssertionHandle;

Asserts this turn’s spend (USD) does not exceed usd.

`judge`

readonly judge: JudgeNamespace;

The judge namespace available for this turn (t.judge.autoevals.*).

Dataset export

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
    },
  }),
);

Array exports generate stable IDs: file/0000, file/0001, and so on. When you already have a stable business key, you can instead default-export a Record<string, EvalDef>. For example, the key 15193 in swelancer.eval.ts generates swelancer/15193. A key must be a non-empty single path segment: it cannot be . or .., and cannot contain /, \, or control characters; discovery order is fixed by lexicographic order over the keys.

​defineEval options

​id

​description

​tags

​environment

​judge

​reporters

​timeoutMs

​metadata

​diff

​setup

​teardown

​test

​Test context: t

​send

​sendFile

​requireInputRequest

​respond

​respondAll

​reply

​sessionId

​events

​newSession

​signal

​model

​reasoningEffort

​flags

​progress

​diagnostic

​log

​skip

​check

​require

​group

​succeeded

​parked

​messageIncludes

​calledTool

​notCalledTool

​toolOrder

​usedNoTools

​maxToolCalls

​loadedSkill

​noFailedActions

​event

​notEvent

​calledSubagent

​eventOrder

​eventsSatisfy

​sandbox

​usage

​maxTokens

​maxCost

​judge

​Judge assertions

​Turn return type

​events

​toolCalls

​status

​message

​data

​usage

​outputEquals

​outputMatches

​messageIncludes

​succeeded

​parked

​calledTool

​notCalledTool

​toolOrder

​usedNoTools

​maxToolCalls

​loadedSkill

​noFailedActions

​event

​notEvent

​calledSubagent

​eventOrder

​eventsSatisfy

​maxTokens

`defineEval` options

`id`

`description`

`tags`

`environment`

`judge`

`reporters`

`timeoutMs`

`metadata`

`diff`

`setup`

`teardown`

`test`

Test context: `t`

`send`

`sendFile`

`requireInputRequest`

`respond`

`respondAll`

`reply`

`sessionId`

`events`

`newSession`

`signal`

`model`

`reasoningEffort`

`flags`

`progress`

`diagnostic`

`log`

`skip`

`check`

`require`

`group`

`succeeded`

`parked`

`messageIncludes`

`calledTool`

`notCalledTool`

`toolOrder`

`usedNoTools`

`maxToolCalls`

`loadedSkill`

`noFailedActions`

`event`

`notEvent`

`calledSubagent`

`eventOrder`

`eventsSatisfy`

`sandbox`

`usage`

`maxTokens`

`maxCost`

`judge`

Judge assertions

`Turn` return type

`events`

`toolCalls`

`status`

`message`

`data`

`usage`

`outputEquals`

`outputMatches`

`messageIncludes`

`succeeded`

`parked`

`calledTool`

`notCalledTool`

`toolOrder`

`usedNoTools`

`maxToolCalls`

`loadedSkill`

`noFailedActions`

`event`

`notEvent`

`calledSubagent`

`eventOrder`

`eventsSatisfy`

`maxTokens`