> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure and Verify a Judge

> Connect an OpenAI-compatible Judge gateway, use one minimal Eval to preflight and score for real, then distinguish missing configuration, network failure, and low-quality output in the results.

Use a Judge for open-ended quality judgments that rules cannot express as an exact match. First use one minimal Eval to confirm that the model, key, and endpoint can score for real, then add the Judge to production tasks.

## Configure the model and credential source

```ts title="niceeval.config.ts" theme={null}
import { defineConfig } from "niceeval";

export default defineConfig({
  judge: {
    model: "judge-model",
    baseUrl: "https://gateway.example.com/v1",
    apiKeyEnv: "JUDGE_GATEWAY_KEY",
    timeoutMs: 180_000,
  },
});
```

Keep the key only in the process environment:

```sh theme={null}
export JUDGE_GATEWAY_KEY="..."
```

The configuration file stores the environment-variable name, never the key value. To compare Judge models, override `judge.model` in different Experiments. Model selection comes from the resolved Eval configuration, not an individual factory call.

## Write one minimal real scoring Eval

```ts title="evals/judge-smoke.eval.ts" theme={null}
import { defineEval } from "niceeval";
import { closedQA } from "niceeval/expect";

export default defineEval({
  judge: true,
  async test(t) {
    t.check(
      {
        input: "operation completed successfully",
        output: "operation completed successfully",
      },
      closedQA("Does the text clearly state that the operation succeeded?").atLeast(0.8),
    )
      .gate()
      .label("Clearly states success");
  },
});
```

`judge: true` declares that this Eval needs a Judge and inherits its fields from the Experiment and project configuration. `.atLeast(0.8)` forms the threshold before registration, and parameterless `.gate()` makes the resulting condition a pass/fail requirement.

## Run only this verification Eval

First ensure that an existing Experiment's `evals` scope includes `judge-smoke`, then use the second positional argument to select only that Eval. This example uses the Experiment ID `local`:

```sh theme={null}
pnpm exec niceeval exp local judge-smoke
```

Use `pnpm exec niceeval exp list` to find the actual Experiment ID. Do not treat an Eval ID as an Experiment ID too. When both the model and key are present, the Runner preflights the Judge endpoint before dispatching. A failed preflight is a setup error and never produces a fabricated Judge score.

## Inspect the scoring result

Copy the Run ID from the completion feedback:

```sh theme={null}
pnpm exec niceeval view --run <run-id>
```

The result should contain the Judge measurement, threshold, condition, reason, and trimmed material. Distinguish these three failure types:

| Result                                            | Meaning                                                                                       |
| ------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `judge-model-unresolved` / `judge-key-unresolved` | Configuration or the environment variable is missing; no network preflight began.             |
| `unavailable`                                     | The network or Provider became unavailable after the request started. It is not a zero score. |
| evaluator `errored`                               | The response is invalid, its score is not finite, or it lies outside `[0, 1]`.                |

For production tasks, first calibrate the rubric and threshold with representative results. To compare two Judge models, have two Experiments select the same Evals and use different `labels` to identify the Judge conditions. See the full matrix pattern in [Experiment Matrices](/docs/tutorials/experiments).