LLM-as-judge in NiceEval: assessing open-ended output

Judge assertions are the fifth assertion mechanism, alongside the four covered in Assert. Use them when correctness cannot be expressed as a rule — for open-ended prose, tone, factual consistency, or summarization quality. The Judge model is entirely separate from the agent under test, which prevents self-evaluation bias: the same model grading its own output would be prone to rating itself generously.

t.judge.autoevals.factuality(reference).atLeast(0.8);   // factual consistency with a reference text
t.judge.autoevals.closedQA("Is the response appropriate for a 10-year-old?");
t.judge.autoevals.summarizes(sourceDocument);            // faithful summarization check
turn.judge.autoevals.closedQA("Custom evaluation rubric description");  // bound to the turn's material

The three judge shapes

t.judge.autoevals exposes exactly three fixed forms, whatever you’re evaluating maps onto one of them:

closedQA(question, opts?) — a closed-ended judgment: does the output satisfy this criterion? Best for rubric-style checks (“is the tone polite and specific”, “does this avoid inventing policy details”).
factuality(reference, opts?) — factual consistency between the output and a reference text you provide.
summarizes(source, opts?) — whether the output is a faithful summary of a source document.

Judge material is either bound by the receiver or passed explicitly: turn.judge defaults to that turn’s material, while root-level t.judge must receive explicit { input, output } strings (see below).

Where judge lives: `t` and `turn`

Like scoped assertions, judge calls follow the rule that the receiver decides the default material, not the call site:

turn.judge.autoevals.closedQA("...")     // (from t.send()'s return value) defaults to turn.message only
t.judge.autoevals.closedQA("...", {      // root level: no implicit material — pass both strings
  input: "The original request",
  output: "The material to judge",
});

turn.judge is turn-level: it only sees the message from that one turn, which is what you want when different turns in a multi-turn eval need different rubrics. Root-level t.judge carries no implicit material, so pass { input, output } explicitly for cross-turn, file, or other custom material:

const turn1 = await t.send("What's in this image?");                  // turn 1: the image
const turn2 = await t.send("What color is the background?");          // turns 2–3: text-only follow-ups,
const turn3 = await t.send("What color is the shape in the middle?"); // testing cross-turn memory

t.judge.autoevals
  .closedQA("Does the assistant stay grounded in the image from turn 1 throughout?", {
    input: "Identify the object in the image from turn 1 and describe it.",
    output: "turn 2 and turn 3 answers",
  })
  .atLeast(0.7);

turn3.judge.autoevals.closedQA("Does this turn answer the shape's color?").gate();

To judge material that isn’t the conversation itself — a file’s current content or anything else — read it through the public Sandbox API and pass both strings explicitly:

const src = await t.sandbox.readText("src/handler.ts");
t.judge.autoevals.closedQA("Is the generated code idiomatic TypeScript?", {
  input: "Rewrite src/handler.ts to idiomatic TypeScript.",
  output: src,
}).atLeast(0.7);

t.sandbox.fileChanged("src/handler.ts");   // attribution: did the Agent change this file?

Judge material is content you read; agent attribution is a separate claim made by fileChanged, not by what you pass to the Judge.

Judge model resolution

The judge model is resolved from most- to least-specific:

Per-eval judge.model
  ↓
Global config judge.model

// niceeval.config.ts — global default
defineConfig({ judge: { model: "anthropic/claude-haiku-4-5" } });

// A specific eval that needs a more capable judge
defineEval({
  judge: { model: "anthropic/claude-opus-4-8" },
  async test(t) {
    t.judge.autoevals.factuality(reference);  // uses claude-opus-4-8 for this eval
  },
});

If neither level is set, the judge call fails at the call site as a configuration error. There is no built-in default judge model, and no environment variable for it — the model is configuration, so it comes from code only.

Judge endpoint and key: OpenAI-compatible protocol

Judge calls use an OpenAI-compatible /chat/completions protocol. OpenAI, DeepSeek, and any gateway compatible with that protocol can serve as the judge endpoint. Point at it in the judge config:

// niceeval.config.ts
defineConfig({
  judge: {
    model: "deepseek-chat",
    baseUrl: "https://api.deepseek.com/v1",  // defaults to the official OpenAI endpoint
    apiKeyEnv: "DEEPSEEK_API_KEY",           // the key is read from this environment variable
  },
});

The endpoint is configuration and the key is a credential, so they come from different places:

Item	Resolution order
Endpoint	`judge.baseUrl` → `https://api.openai.com/v1`
Key	env var named by `judge.apiKeyEnv` → `NICEEVAL_JUDGE_KEY`

Point judge.baseUrl explicitly at your gateway or OpenAI-compatible proxy — NiceEval does not go looking for an endpoint in variables like OPENAI_BASE_URL. The key works the same way: exactly one name, either the one judge.apiKeyEnv names or the default NICEEVAL_JUDGE_KEY. The judge has its own credential and never borrows the one belonging to the app under test or to an agent.

When the model or key does not resolve, the judge assertion is recorded as unavailable (with a reason) rather than vanishing: unless you explicitly chain .optional(), an assertion that cannot be evaluated makes the attempt errored — a verdict nobody could reach is neither a pass nor the agent’s fault. After configuring keys, run one eval with t.judge and confirm the judge score appears in niceeval view.

Severity: judge defaults to soft

Judge calls are assertions, so they follow the same mechanics described in Assert · Gate vs soft severity — but their default is different from most value matchers:

turn.judge.autoevals.closedQA("Is the tone polite?");              // no threshold → soft, pure score, never fails the eval
turn.judge.autoevals.closedQA("Is the tone polite?").atLeast(0.7); // soft with a threshold → fails only under --strict
turn.judge.autoevals.closedQA("Is the tone polite?").gate();       // promoted to gate → fails immediately if below threshold

With no .atLeast() and no .gate(), a bare judge call is recorded purely as a quality score — it shows up on the eval’s score chip but can never fail the run on its own. This is the right default for judge: it’s a probabilistic model assessing another probabilistic model’s output, and treating every rubric miss as a hard failure would make suites brittle and constantly red. Reach for .gate() only when the judged property is truly load-bearing (for example, “the response must not fabricate account details”).

Assert — gate vs soft severity in full, plus the verdict rules judge scores feed into.
Drive — t.send(), t.newSession(), and which handle t.judge and turn.judge each hang off of.
Evals — how judge scores fold into the eval lifecycle and verdict types.

Core Model

Interaction & Assertions

Execution Model

LLM-as-judge in NiceEval: assessing open-ended output

The three judge shapes

Where judge lives: `t` and `turn`

Judge model resolution

Judge endpoint and key: OpenAI-compatible protocol

Severity: judge defaults to soft

​The three judge shapes

​Where judge lives: t and turn

​Judge model resolution

​Judge endpoint and key: OpenAI-compatible protocol

​Severity: judge defaults to soft

​Related pages

The three judge shapes

Where judge lives: `t` and `turn`

Judge model resolution

Judge endpoint and key: OpenAI-compatible protocol

Severity: judge defaults to soft

Related pages