Configure the Judge and Confirm It Can Really Score

Questions such as “is this explanation suitable for elementary school students” or “did the answer make up facts” cannot be written as a regex; they go to the Judge model to score. The Judge itself can go wrong too: a wrong API key, a misspelled endpoint, a misspelled model name. If you only find these problems after 200 evals have run, none of the Judge scores along the way can be obtained. So configure it first, then run one real scoring pass with a minimal eval, and use it at scale only after confirming you get a score.

Configure which service the Judge uses

In niceeval.config.ts, spell out the service, model, and endpoint with a named Provider. The example below uses an OpenAI-compatible gateway:

niceeval.config.ts

import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({
    model: "judge-model",
    baseUrl: "https://gateway.example.com/v1",
    apiKeyEnv: "JUDGE_GATEWAY_KEY",
    timeoutMs: 180_000,
    maxOutputTokens: 512,
  }),
});

apiKeyEnv holds only the variable name. Put the key itself in an environment variable, or in the project’s .env:

export JUDGE_GATEWAY_KEY="..."

Write a minimal eval

Write an eval with only one Judge assertion to confirm the configuration can get a score:

evals/judge-smoke.eval.ts

import { defineEval } from "niceeval";

export default defineEval({
  judge: "judge-model",
  async test(t) {
    const operation = "Complete the data import";
    const turn = await t.send(operation);
    turn.closeQA("Does this turn clearly report the execution status and data import result?")
      .gate(0.8)
      .label("Reports success clearly");
  },
});

turn.closeQA(question) hands this Turn’s full conversation to the Judge, which answers your question and gives a score from 0 to 1. .gate(0.8) means this eval fails when the score is below 0.8. When the material is empty, this assertion scores 0 directly without calling the Judge. When the material is incomplete and cannot be judged, the result is unavailable, which is not counted as 0. To have the Judge look only at specific messages or tool calls, pass the matching Match to closeQA; for the application’s own business material, use a custom MaterialMatch. See NiceEval Taste for how to write them.

Attach image material

Attach images only when the criterion really needs to see them. Keep the same defineJudge and t.judge; you do not need a second Judge or score gate. First set supportsImages: true in the existing chat Provider’s configuration, and switch model to a model that supports both images and tool calls. TypesafeProvider does not support images. The eval below reads the screenshot.png the application has already saved, and does not take a new screenshot for the eval:

import { readFile } from "node:fs/promises";
import { defineEval } from "niceeval";
import { defineJudge } from "niceeval/expect";
import { judgeImage } from "niceeval/judge";

const actionQuality = defineJudge({ rubric: "Rate whether the action is reasonable based on known facts; the image is only supplementary, and do not infer actions outside the frame.",
});

export default defineEval({
  async test(t) {
    const screenshot = judgeImage({
      body: await readFile("screenshot.png"),
      mediaType: "image/png",
    });
    t.judge({ action: "The character finished carrying the item", screenshot }, actionQuality).gate(0.8);
  },
});

Images must be PNG or JPEG, at most 4 MiB each. When an image is too large or has an invalid file header, NiceEval rejects it before sending. Do not pass a path, URL, or plain base64 string in place of an image. After the run, use query’s attempt.assertion.detail to view this assertion; the result includes the image’s imageId and digest. To get the original image back, use attempt.assertion.image with the same Attempt’s entryId and imageId, reading in segments: offset is the byte offset, and limit is at most 256 KiB. It reads the saved image and never rereads the file or calls the model.

Run once and confirm you get a score

Make sure an existing Experiment’s evals includes judge-smoke, then run:

pnpm exec niceeval exp local judge-smoke

NiceEval does not contact the Judge endpoint when loading configuration; it only sends a request when a Judge assertion really executes. So a configuration mistake only shows up after a run. After the run, copy the Run ID from the terminal’s NEXT panel and open this run:

pnpm exec niceeval show --run <run-id>

Copy the locator of the judge-smoke Attempt and view the assertion result:

pnpm exec niceeval show @<attempt-locator>

When the configuration is correct, this assertion shows a score and whether it reached the threshold. You can also view it in the browser with pnpm exec niceeval view --run <run-id>. When you need the reasons the Judge gave and the material actually sent to it, use query’s attempt.assertion.detail; see the example in View Run Results. When you get no score, decide what to check from the result:

Result you see	Cause	Check first
`unavailable`, reason `judge-provider-unresolved`	No Provider configured; no request was sent	`judgeProvider` in `niceeval.config.ts`
`unavailable`, reason `judge-key-unresolved`	The key could not be read; no request was sent	Whether the variable named in `apiKeyEnv` is set
`unavailable`	The request could not be sent or timed out	Network, `baseUrl`, `timeoutMs`
`errored`	The endpoint returned HTTP 400, the protocol is incompatible, or the response is not in the required format	Whether `baseUrl` points to a compatible gateway, and whether the `model` name is correct
`errored`	The Judge gave no judgment, used the wrong format, or returned an out-of-range score	Whether the model supports tool calls

To compare different Judges, have several Experiments use different model strings or named Providers, and mark the dimension being compared with labels.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Configure the Judge and Confirm It Can Really Score

Configure which service the Judge uses

Write a minimal eval

Attach image material

Run once and confirm you get a score