> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM-as-judge in NiceEval: assessing open-ended output

> How t.judge and turn.judge use a separate judge model to score factuality, closed-ended quality, and summarization faithfulness — plus model resolution and severity.

Judge assertions are the fifth assertion mechanism, alongside the four covered in [Assert](/docs/explanation/assert). Use them when correctness cannot be expressed as a rule — for open-ended prose, tone, factual consistency, or summarization quality. The Judge model is **entirely separate** from the agent under test, which prevents self-evaluation bias: the same model grading its own output would be prone to rating itself generously.

```ts theme={null}
t.judge.autoevals.factuality(reference).atLeast(0.8);   // factual consistency with a reference text
t.judge.autoevals.closedQA("Is the response appropriate for a 10-year-old?");
t.judge.autoevals.summarizes(sourceDocument);            // faithful summarization check
turn.judge.autoevals.closedQA("Custom evaluation rubric description");  // bound to the turn's material
```

## The three judge shapes

`t.judge.autoevals` exposes exactly three fixed forms, whatever you're evaluating maps onto one of them:

* **`closedQA(question, opts?)`** — a closed-ended judgment: does the output satisfy this criterion? Best for rubric-style checks ("is the tone polite and specific", "does this avoid inventing policy details").
* **`factuality(reference, opts?)`** — factual consistency between the output and a reference text you provide.
* **`summarizes(source, opts?)`** — whether the output is a faithful summary of a source document.

Judge material is either bound by the receiver or passed explicitly: `turn.judge` defaults to that turn's material, while root-level `t.judge` must receive explicit `{ input, output }` strings (see below).

## Where judge lives: `t` and `turn`

Like scoped assertions, judge calls follow the rule that **the receiver decides the default material, not the call site**:

```ts theme={null}
turn.judge.autoevals.closedQA("...")     // (from t.send()'s return value) defaults to turn.message only
t.judge.autoevals.closedQA("...", {      // root level: no implicit material — pass both strings
  input: "The original request",
  output: "The material to judge",
});
```

`turn.judge` is turn-level: it only sees the message from that one turn, which is what you want when different turns in a multi-turn eval need different rubrics. Root-level `t.judge` carries no implicit material, so pass `{ input, output }` explicitly for cross-turn, file, or other custom material:

```ts theme={null}
const turn1 = await t.send("What's in this image?");                  // turn 1: the image
const turn2 = await t.send("What color is the background?");          // turns 2–3: text-only follow-ups,
const turn3 = await t.send("What color is the shape in the middle?"); // testing cross-turn memory

t.judge.autoevals
  .closedQA("Does the assistant stay grounded in the image from turn 1 throughout?", {
    input: "Identify the object in the image from turn 1 and describe it.",
    output: "turn 2 and turn 3 answers",
  })
  .atLeast(0.7);

turn3.judge.autoevals.closedQA("Does this turn answer the shape's color?").gate();
```

To judge material that isn't the conversation itself — a file's current content or anything else — read it through the public Sandbox API and pass both strings explicitly:

```ts theme={null}
const src = await t.sandbox.readText("src/handler.ts");
t.judge.autoevals.closedQA("Is the generated code idiomatic TypeScript?", {
  input: "Rewrite src/handler.ts to idiomatic TypeScript.",
  output: src,
}).atLeast(0.7);

t.sandbox.fileChanged("src/handler.ts");   // attribution: did the Agent change this file?
```

Judge material is content you read; agent attribution is a separate claim made by `fileChanged`, not by what you pass to the Judge.

## Judge model resolution

The judge model is resolved from most- to least-specific:

```
Per-eval judge.model
  ↓
Global config judge.model
```

```ts theme={null}
// niceeval.config.ts — global default
defineConfig({ judge: { model: "anthropic/claude-haiku-4-5" } });

// A specific eval that needs a more capable judge
defineEval({
  judge: { model: "anthropic/claude-opus-4-8" },
  async test(t) {
    t.judge.autoevals.factuality(reference);  // uses claude-opus-4-8 for this eval
  },
});
```

If neither level is set, the judge call fails at the call site as a configuration error. There is no built-in default judge model, and no environment variable for it — the model is configuration, so it comes from code only.

## Judge endpoint and key: OpenAI-compatible protocol

Judge calls use an **OpenAI-compatible `/chat/completions`** protocol. OpenAI, DeepSeek, and any gateway compatible with that protocol can serve as the judge endpoint. Point at it in the `judge` config:

```ts theme={null}
// niceeval.config.ts
defineConfig({
  judge: {
    model: "deepseek-chat",
    baseUrl: "https://api.deepseek.com/v1",  // defaults to the official OpenAI endpoint
    apiKeyEnv: "DEEPSEEK_API_KEY",           // the key is read from this environment variable
  },
});
```

The endpoint is configuration and the key is a credential, so they come from different places:

| Item     | Resolution order                                          |
| -------- | --------------------------------------------------------- |
| Endpoint | `judge.baseUrl` → `https://api.openai.com/v1`             |
| Key      | env var named by `judge.apiKeyEnv` → `NICEEVAL_JUDGE_KEY` |

Point `judge.baseUrl` explicitly at your gateway or OpenAI-compatible proxy — NiceEval does not go looking for an endpoint in variables like `OPENAI_BASE_URL`. The key works the same way: exactly one name, either the one `judge.apiKeyEnv` names or the default `NICEEVAL_JUDGE_KEY`. The judge has its own credential and never borrows the one belonging to the app under test or to an agent.

<Warning>
  When the model or key does not resolve, the judge assertion is recorded as `unavailable` (with a reason) rather than vanishing: unless you explicitly chain `.optional()`, an assertion that cannot be evaluated makes the attempt `errored` — a verdict nobody could reach is neither a pass nor the agent's fault. After configuring keys, run one eval with `t.judge` and confirm the judge score appears in `niceeval view`.
</Warning>

## Severity: judge defaults to soft

Judge calls are assertions, so they follow the same mechanics described in [Assert · Gate vs soft severity](/docs/explanation/assert#gate-vs-soft-severity) — but their **default** is different from most value matchers:

```ts theme={null}
turn.judge.autoevals.closedQA("Is the tone polite?");              // no threshold → soft, pure score, never fails the eval
turn.judge.autoevals.closedQA("Is the tone polite?").atLeast(0.7); // soft with a threshold → fails only under --strict
turn.judge.autoevals.closedQA("Is the tone polite?").gate();       // promoted to gate → fails immediately if below threshold
```

With no `.atLeast()` and no `.gate()`, a bare judge call is recorded purely as a quality score — it shows up on the eval's score chip but can never fail the run on its own. This is the right default for judge: it's a probabilistic model assessing another probabilistic model's output, and treating every rubric miss as a hard failure would make suites brittle and constantly red. Reach for `.gate()` only when the judged property is truly load-bearing (for example, "the response must not fabricate account details").

## Related pages

* [Assert](/docs/explanation/assert) — gate vs soft severity in full, plus the verdict rules judge scores feed into.
* [Drive](/docs/explanation/drive) — `t.send()`, `t.newSession()`, and which handle `t.judge` and `turn.judge` each hang off of.
* [Evals](/docs/explanation/evals) — how judge scores fold into the eval lifecycle and verdict types.