Judge in NiceEval: assess open-ended output

Judge uses a separate grading model to assess factual consistency, summary faithfulness, and explanation quality. It is a managed evaluator. Ordinary custom ScoreMatch values stay pure functions; defineJudge, the built-in factories, and the advanced managed defineScoreMatch all take the same Assertion path.

Declare a scoring definition

Write the criterion at module scope. rubric describes the dimensions to evaluate, and anchors give calibration points on a continuous quality scale:

import { defineJudge } from "niceeval/expect";

const explainsRisk = defineJudge({ rubric: "Based on change, evaluate whether explanation covers compatibility, rollback, and data risks.",
  anchors: [
    { measurement: 0, description: "Does not explain a concrete risk or rollback plan" },
    { measurement: 0.5, description: "Explains some risks but omits one key point" },
    { measurement: 1, description: "Fully explains compatibility, rollback, and data risks" },
  ],
});

When anchors is omitted, NiceEval uses 0 and 1 as the two default endpoints. A measurement can be any finite value in [0,1]; the endpoints do not limit the output to two classes, and they are not model confidence.

Select evaluation material

Pass the values needed for the decision directly to t.judge. The eval’s judge only selects the model configuration for this eval:

export default defineEval({
  judge: "judge-model",
  async test(t) {
    const change = "Modify a persistent field and provide a rollback plan";
    const turn = await t.send(`Explain this change: ${change}`);

    t.judge({ change, explanation: turn.reply }, explainsRisk)
      .gate(0.8)
      .label("Risk explanation quality");
  },
});

Material uses strings, finite numbers, booleans, null, arrays, and plain objects. Field names should express application meaning, such as change and explanation; you do not need to join the object into text first. NiceEval creates a bounded JSON snapshot when the Assertion is registered, so later changes to the original object do not change the request. When you need to inspect image content, put judgeImage({ body, mediaType }) in the material. It freezes PNG or JPEG bytes without creating another Judge or sending an extra request. The image and text are sent in the same Judge call and saved in the same Assertion’s evidence. For the setup steps, see Add image material. t.judge(material, explainsRisk) and t.check(material, explainsRisk) each register one measurement Assertion. They share the material snapshot, budget, evaluation, audit, result saving, and handle; there is no Match allowlist.

Use built-in Judges directly

Root t, Session, and Turn all provide built-in methods that return an Assertion’s MeasurementAssertionHandle<Kind>. For example:

import { eventMatch } from "niceeval/expect";
const answer = await t.send("List the risks and rollback steps.");
answer.closeQA("Does this turn explain the risks and actionable rollback steps?")
  .gate(0.8).label("Rollback explanation");

// For a special check, select all assistant messages explicitly.
answer.closeQA(
  eventMatch("message", { role: "assistant" }),
  "Do all messages clearly explain the risks?",
).gate(0.8);

closeQA(selector, question, options?) sends all matched material to one Judge call in its original order. The question is an acceptance question answered only from the material, not a preset answer. An Agent’s closeQA(question, options?) uses the complete recorded history of that Turn, Session, or Attempt by default, including messages and tool facts. It does not pick only the last reply or successful operations. An explicit EventMatch selects from the three event views. For questions about tool inputs and receipts, use a ToolMatch or a custom MaterialMatch. All these forms compose a scoring Match and pass it to the same check. Application assertions such as usedNoTools() wrap a fixed criterion and do not accept another Match. The result is 1 when the condition holds, 0 when it does not, and unavailable when evidence is insufficient. A complete empty set gives 0 without calling the model; missing or oversized material is not filled with zero or truncated before scoring. An Adapter provides the application ctx through create, and the synchronous reader in defineMaterialMatch reads the current read-only ctx when the Assertion is registered. A selector’s predicate judges one item only; and constrains the same item. Use t.check(selector).gate() for ordinary existence checks and t.closeQA(selector, question).gate(0.8) for a question about the whole set. factuality, faithfulness, instructionFollowing, and pairwisePreference take explicit material, build a Match, and pass it to the same check.

Configure the Judge Provider

Project configuration selects the service explicitly with a named Provider factory. This example connects to OpenAI or an explicitly compatible gateway:

import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({
    model: "judge-model",
    baseUrl: "https://gateway.example.com/v1",
    apiKeyEnv: "JUDGE_GATEWAY_KEY",
    timeoutMs: 180_000,
    maxOutputTokens: 512,
  }),
});

Factory	Service	Default key variable
`OpenAIProvider`	OpenAI or an explicitly compatible gateway	`OPENAI_API_KEY`
`VercelProvider`	Vercel AI Gateway	`AI_GATEWAY_API_KEY`
`OpenRouterProvider`	OpenRouter	`OPENROUTER_API_KEY`
`TypesafeProvider`	TypeSafe System One	`TYPESAFE_API_KEY`

All four factories are exported from niceeval/judge. The Vercel factory means AI Gateway, not Vercel Sandbox or a general AI SDK model object. When one eval only needs a different model, write just this field:

export default defineEval({
  judge: "another-model",
  async test(t) {
    const answer = await t.send("What is the capital of France?");
    t.factuality({ input: "What is the capital of France?", output: answer.message, expected: "Paris" }).gate(1);
  },
});

This keeps the global Provider’s endpoint, credential source, and execution limits, and replaces only model. Evals and Experiments can also provide a complete Provider, which replaces the service, default model, and limits as a whole. There is no rule that guesses the service from the hostname, model prefix, or an existing key. NiceEval uses the selected Provider’s managed protocol request to get a finite [0,1] measurement and a public rationale. Chat services use forced-function requests, and TypeSafe uses /systemone. With no model or key configured, no network request is sent and the Assertion is unavailable. Transport failures or timeouts are unavailable; HTTP 400, protocol incompatibility, or invalid responses are errored. Static configuration validation does not probe the network. TypeSafe uses the /systemone protocol. It supports score, classify, and batch classify, but not extract. So faithfulness() on TypesafeProvider gives unavailable; it is not converted to an overall score or a different denominator.

Thresholds and scores

Both pass/fail and scored evals call .gate(minimum) on the registered handle to form a quality gate. Scored evals can also call .score(points), and the contribution is the measurement times points. Gate and score can be combined in either order, and both run the Judge only once. Successful requests save the fixed material actually sent. A path, URL, alt text, or ordinary base64 string is not visual input; only explicit image material becomes the image part of the model API request.

Core Model

Interaction and Scoring

Execution Model