Configure and Verify a Judge

Use a Judge for open-ended quality judgments that rules cannot express as an exact match. First use one minimal Eval to confirm that the model, key, and endpoint can score for real, then add the Judge to production tasks.

Configure the model and credential source

niceeval.config.ts

import { defineConfig } from "niceeval";

export default defineConfig({
  judge: {
    model: "judge-model",
    baseUrl: "https://gateway.example.com/v1",
    apiKeyEnv: "JUDGE_GATEWAY_KEY",
    timeoutMs: 180_000,
  },
});

Keep the key only in the process environment:

export JUDGE_GATEWAY_KEY="..."

The configuration file stores the environment-variable name, never the key value. To compare Judge models, override judge.model in different Experiments. Model selection comes from the resolved Eval configuration, not an individual factory call.

Write one minimal real scoring Eval

evals/judge-smoke.eval.ts

import { defineEval } from "niceeval";
import { closedQA } from "niceeval/expect";

export default defineEval({
  judge: true,
  async test(t) {
    t.check(
      {
        input: "operation completed successfully",
        output: "operation completed successfully",
      },
      closedQA("Does the text clearly state that the operation succeeded?").atLeast(0.8),
    )
      .gate()
      .label("Clearly states success");
  },
});

judge: true declares that this Eval needs a Judge and inherits its fields from the Experiment and project configuration. .atLeast(0.8) forms the threshold before registration, and parameterless .gate() makes the resulting condition a pass/fail requirement.

Run only this verification Eval

First ensure that an existing Experiment’s evals scope includes judge-smoke, then use the second positional argument to select only that Eval. This example uses the Experiment ID local:

pnpm exec niceeval exp local judge-smoke

Use pnpm exec niceeval exp list to find the actual Experiment ID. Do not treat an Eval ID as an Experiment ID too. When both the model and key are present, the Runner preflights the Judge endpoint before dispatching. A failed preflight is a setup error and never produces a fabricated Judge score.

Inspect the scoring result

Copy the Run ID from the completion feedback:

pnpm exec niceeval view --run <run-id>

The result should contain the Judge measurement, threshold, condition, reason, and trimmed material. Distinguish these three failure types:

Result	Meaning
`judge-model-unresolved` / `judge-key-unresolved`	Configuration or the environment variable is missing; no network preflight began.
`unavailable`	The network or Provider became unavailable after the request started. It is not a zero score.
evaluator `errored`	The response is invalid, its score is not finite, or it lies outside `[0, 1]`.

For production tasks, first calibrate the rubric and threshold with representative results. To compare two Judge models, have two Experiments select the same Evals and use different labels to identify the Judge conditions. See the full matrix pattern in Experiment Matrices.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Configure and Verify a Judge

Configure the model and credential source

Write one minimal real scoring Eval

Run only this verification Eval

Inspect the scoring result

​Configure the model and credential source

​Write one minimal real scoring Eval

​Run only this verification Eval

​Inspect the scoring result

Configure the model and credential source

Write one minimal real scoring Eval

Run only this verification Eval

Inspect the scoring result