> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# defineEval：声明、配置并运行 NiceEval 评估用例

> defineEval 参考：选项、test context t、Turn 返回值、Sandbox 辅助函数，以及数组和 keyed record 测试集导出。

`defineEval` 是编写评估用例的主要入口。每个评估用例文件调用一次，传入描述和 `test(t)`，并默认导出结果。

```ts theme={null}
import { defineEval } from "niceeval";

export default defineEval({
  description: "Brooklyn weather query",
  async test(t) {
    const turn = await t.send("What's the weather like in Brooklyn today?");
    turn.succeeded().label("Turn 完成");
  },
});
```

<Note>
  不要提供 `id` 或 `name`。[NiceEval](https://niceeval.com/) 从文件路径推导评估用例 ID。
</Note>

## `defineEval` 选项

#### `description`

```ts theme={null}
description?: string;
```

一句话描述,展示在 `niceeval list` 和 view 里;纯说明,不影响调度或打分。

#### `tags`

```ts theme={null}
tags?: string[];
```

标签,供 CLI `--tag` 过滤和 view 分类;与 id 前缀过滤是两套独立的筛选维度。

#### `sandbox`

```ts theme={null}
sandbox?: SandboxLayer;
```

这道题贡献的 Sandbox 声明层。省略等价于空 command-only layer，不提供隐式 template。
每个实际 Eval x Experiment 配对必须恰好一方提供 template-bearing layer。

#### `plugins`

```ts theme={null}
plugins?: readonly PluginInstance<"eval">[];
```

Explicit, immutable Eval Plugin occurrences; no directory inheritance exists.

#### `judge`

```ts theme={null}
judge?: JudgeDeclaration;
```

声明 Judge capability；true 继承 Experiment/Config，对象同时声明并覆盖它们。

#### `reporters`

```ts theme={null}
reporters?: Reporter[];
```

覆盖 / 追加项目级 Config.reporters,只对这一条评估用例生效。

#### `timeoutMs`

```ts theme={null}
timeoutMs?: number;
```

覆盖项目级 / CLI 的单次 attempt 超时(毫秒),只对这一条评估用例生效。

#### `metadata`

```ts theme={null}
metadata?: globalThis.Record<string, JsonValue>;
```

任意附加元数据,作为 Attempt Provenance 保存,不参与调度或打分;供自定义 reporter 消费。

#### `diff`

```ts theme={null}
diff?: { include?: string[]; ignore?: string[] };
```

调整 agent diff 的归因排除清单(仅 Sandbox 型;见 docs/feature/eval/README.md):两个数组都是
gitignore 风格 glob(workdir 相对)。默认排除 .git/node\_modules/构建产物/包管理器缓存;
`ignore` 在默认清单上追加排除;`include` 优先级最高,把匹配路径显式加回。
合成规则固定为「默认 ∪ ignore,再被 include 打洞」,清单在分类账锚点时冻结。

#### `test`

```ts theme={null}
test(t: TestContext): EvalTestReturn;
```

## Test context: `t`

`t`（`TestContext`）是评估用例作者拿到的高层上下文。每一次作者入口调用都直接登记一条 Assertion。`t.check(value, match)` 严格接收两个参数。`t.succeeded()`、`session.succeeded()`、`turn.succeeded()`、`calledTool(...)`、`notCalledTool(...)` 与 Judge recipe 也直接登记 Assertion。handle 只配置同一条 entry 的 `key`、`label`、`.atLeast(n)` 与 `.score(n)`。`.orStop()` 也只作用于此 entry，绝不登记第二条 Assertion。

Pass Eval 用 Boolean condition 折叠 Attempt Verdict。measurement 必须 `.atLeast(n)`；`await handle.orStop()` 在 mismatch 或 below 时停止当前 continuation。`defineScoreEval` 的 `ScoreTestContext` 额外提供 `t.score(n)` 直接登记 contribution，`n` 必须 finite 且不小于零；已有 Assertion 用 `.score(n)` 贡献分数，`n` 必须 finite 且大于零。`calledTool` 与 `notCalledTool` 的完整契约见 [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md)。全部成员：

#### `evaluationKind`

```ts theme={null}
readonly evaluationKind: Kind;
```

#### `send`

```ts theme={null}
send(input: string | { readonly text: string; readonly files?: readonly InputFile[] }): Promise<TurnHandle<Kind>>;
```

#### `sendFile`

```ts theme={null}
sendFile(path: string, text?: string): Promise<TurnHandle<Kind>>;
```

#### `requireInputRequest`

```ts theme={null}
requireInputRequest(filter?: InputRequestFilter): InputRequest;
```

#### `respond`

```ts theme={null}
respond(...responses: readonly (string | InputAnswer)[]): Promise<TurnHandle<Kind>>;
```

#### `respondAll`

```ts theme={null}
respondAll(optionId: string): Promise<TurnHandle<Kind>>;
```

#### `reply`

```ts theme={null}
readonly reply: string;
```

#### `sessionId`

```ts theme={null}
readonly sessionId: string | undefined;
```

#### `events`

```ts theme={null}
readonly events: readonly StreamEvent[];
```

#### `newSession`

```ts theme={null}
newSession(): SessionHandle<Kind>;
```

#### `signal`

```ts theme={null}
readonly signal: AbortSignal;
```

#### `model`

```ts theme={null}
readonly model?: string;
```

#### `reasoningEffort`

```ts theme={null}
readonly reasoningEffort?: string;
```

#### `flags`

```ts theme={null}
readonly flags: Readonly<globalThis.Record<string, JsonValue>>;
```

#### `progress`

```ts theme={null}
progress(update: import("../types.ts").ProgressUpdate): void;
```

#### `diagnostic`

```ts theme={null}
diagnostic(input: import("../types.ts").DiagnosticInput): void;
```

#### `log`

```ts theme={null}
log(message: string): void;
```

#### `skip`

```ts theme={null}
skip(reason: string): never;
```

#### `group`

```ts theme={null}
group<Value>(
  title: string,
  body: () => Value | PromiseLike<Value>,
): Promise<Awaited<Value>>;
```

#### `check`

```ts theme={null}
check: AssertionsRuntime<Kind>["t"]["check"];
```

#### `sandbox`

```ts theme={null}
readonly sandbox: Sandbox<Kind>;
```

#### `o11y`

```ts theme={null}
readonly o11y: import("../o11y/types.ts").O11ySummary;
```

#### `usage`

```ts theme={null}
readonly usage: Usage;
```

#### `succeeded`

```ts theme={null}
succeeded(): BooleanAssertionHandle<Kind, void>;
```

#### `usedNoTools`

```ts theme={null}
usedNoTools(): BooleanAssertionHandle<Kind, void>;
```

#### `maxToolCalls`

```ts theme={null}
maxToolCalls(max: number): BooleanAssertionHandle<Kind, void>;
```

#### `noFailedActions`

```ts theme={null}
noFailedActions(): BooleanAssertionHandle<Kind, void>;
```

#### `event`

```ts theme={null}
event(match: EventMatch, options?: EventOptions): BooleanAssertionHandle<Kind, void>;
```

#### `notEvent`

```ts theme={null}
notEvent(match: EventMatch): BooleanAssertionHandle<Kind, void>;
```

#### `maxTokens`

```ts theme={null}
maxTokens(max: number): BooleanAssertionHandle<Kind, void>;
```

#### `maxCost`

```ts theme={null}
maxCost(usd: number): BooleanAssertionHandle<Kind, void>;
```

#### `judge`

```ts theme={null}
readonly judge: RootJudge<Kind>;
```

## Judge measurement

先在 `defineEval` 或 `defineScoreEval` 声明 `judge: true`，再直接登记 Judge Assertion：

```ts theme={null}
const turn = await t.send("总结需求。");
turn.judge.autoevals.factuality(expected).atLeast(0.8).label("事实一致");
turn.judge.autoevals.closedQA(question).atLeast(0.7).label("回答质量");
turn.judge.autoevals.summarizes(sourceText).score(20).label("摘要质量");
```

`t.judge.autoevals` 的根级 recipe 额外接收 `{ input, output }` 字符串材料。`turn.judge` 已绑定 immutable Turn 材料。模型配置按 Eval、Experiment、项目配置解析一次；没有单次模型覆盖或 `{ on }` 选项。Pass Eval 必须 `.atLeast(n)`；Score Eval 直接 `.score(n)`。

## `Turn` 返回类型

`t.send(...)` 返回一个 `TurnHandle`：从事件流派生的便利字段，加上一整套本轮作用域断言。

`calledTool` 与 `notCalledTool` 也属于 Turn 的作用域断言；它们的签名和 Match 规则只在 [Scoped assertions](https://github.com/NiceEval/NiceEval/blob/main/docs/feature/assertions/library/scoped-assertions.md) 定义。

#### `events`

```ts theme={null}
readonly events: readonly StreamEvent[];
```

#### `toolCalls`

```ts theme={null}
readonly toolCalls: readonly import("../o11y/types.ts").ToolCall[];
```

#### `status`

```ts theme={null}
readonly status: "completed" | "failed" | "waiting";
```

#### `message`

```ts theme={null}
readonly message: string;
```

#### `data`

```ts theme={null}
readonly data?: JsonValue;
```

#### `usage`

```ts theme={null}
readonly usage?: Usage;
```

#### `succeeded`

```ts theme={null}
succeeded(): BooleanAssertionHandle<Kind, void>;
```

#### `toolOrder`

```ts theme={null}
toolOrder(matches: readonly [ToolMatch, ToolMatch, ...ToolMatch[]]): BooleanAssertionHandle<Kind, void>;
```

#### `usedNoTools`

```ts theme={null}
usedNoTools(): BooleanAssertionHandle<Kind, void>;
```

#### `maxToolCalls`

```ts theme={null}
maxToolCalls(max: number): BooleanAssertionHandle<Kind, void>;
```

#### `noFailedActions`

```ts theme={null}
noFailedActions(): BooleanAssertionHandle<Kind, void>;
```

#### `event`

```ts theme={null}
event(match: EventMatch, options?: EventOptions): BooleanAssertionHandle<Kind, void>;
```

#### `notEvent`

```ts theme={null}
notEvent(match: EventMatch): BooleanAssertionHandle<Kind, void>;
```

#### `eventOrder`

```ts theme={null}
eventOrder(matches: readonly [EventMatch, EventMatch, ...EventMatch[]]): BooleanAssertionHandle<Kind, void>;
```

#### `maxTokens`

```ts theme={null}
maxTokens(max: number): BooleanAssertionHandle<Kind, void>;
```

#### `maxCost`

```ts theme={null}
maxCost(usd: number): BooleanAssertionHandle<Kind, void>;
```

#### `judge`

```ts theme={null}
readonly judge: TurnJudge<Kind>;
```

## 测试集导出

```ts theme={null}
export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      await t.send(row.prompt);
    },
  }),
);
```

数组导出会生成稳定 ID：`file/0000`、`file/0001` 等。

已有稳定业务 key 时也可以默认导出 `Record<string, EvalDef>`。例如 key `15193` 在 `swelancer.eval.ts` 中生成 `swelancer/15193`。key 必须是非空单一路径片段，不能是 `.` / `..`，不能含 `/`、`\\` 或控制字符。发现顺序按 key 字典序固定。