Score by Completion: Pass/Fail and Scored Evals

You ask an Agent to install an open source project and get its health check passing, in five steps. It completes three and gets stuck on the fourth. Looking only at pass or fail, this run is a “failure”, no different from doing nothing at all. But what you want to know is how far the Agent is from finishing, and whether a prompt change moved it from three steps to four or left it where it was. So NiceEval has two kinds of evals:

defineEval is pass/fail. It answers “was it done this time”, and you read the pass rate. It suits tasks that are only meaningful when everything holds, such as tests that must pass or keys that must not leak.
defineScoreEval is scored. It answers “how much was done”, and you read the accumulated score. It suits tasks with partial progress, such as step-by-step installation or several quality requirements.

An Experiment or eval group can contain only one kind. niceeval check, --dry, and normal runs all reject mixing them before starting the Agent or Sandbox; when you need both readings, split them into two Experiments.

Use `defineEval` for what must be done

import { defineEval } from "niceeval";
import { commandSucceeded, includes } from "niceeval/expect";

export default defineEval({
  async test(t) {
    const turn = await t.send("Fix the tests and explain the cause.");

    t.check(turn.reply, includes("cause")).label("Explains the failure cause");

    const tests = await t.sandbox.runCommand("pnpm", ["test"]);
    await t.check(tests, commandSucceeded()).label("Tests pass").orStop();
  },
});

A Boolean matched goes into the Verdict; mismatched makes the final Verdict failed, but does not stop other Assertions from being registered and settled. When later code must depend on this result, await handle.orStop().

Continuous quality thresholds

Matches such as similarity(...) produce a measurement in [0, 1]. It computes character-level Levenshtein edit similarity and does not judge meaning. A pass/fail eval sets a quality gate on the handle with .gate(minimum):

t.check(turn.reply, similarity(expected))
  .gate(0.8)
  .label("Answer is close to the expected content");

When the value is below the threshold formed on the Match, this requirement fails. When you only want to leave a note that does not affect the verdict, use t.diagnostic(...); do not register a measurement nobody consumes.

Three steps done should still earn three points

Step-by-step tasks suit defineScoreEval. An Assertion only saves its evaluation by default and does not score; only .score(n) makes it contribute points. A Boolean matched contributes n and mismatched contributes 0; a measurement m contributes m * n.

import { defineScoreEval } from "niceeval";
import { commandSucceeded, includes } from "niceeval/expect";

export default defineScoreEval({
  description: "Install and start DB-GPT",
  async test(t) {
    const turn = await t.send("Install DB-GPT and get the health check passing.");

    t.sandbox.fileChanged("db-gpt/.env").score(1).label("Configure the runtime environment");

    const health = await t.sandbox.runCommand("curl", ["-s", "localhost:5670/health"]);
    t.check(health, commandSucceeded()).score(1).label("Health check reachable");
    t.check(health.stdout, includes("ok")).score(1).label("Health check content correct");

    t.check(turn.reply, includes("verify")).score(2).label("Reply explains how it was verified");

    t.score(1).label("Concise code");
  },
});

t.score(n) registers a contribution directly. n must be finite and not less than zero, and the returned handle can only configure key and label. A Judge or other measurement can call .gate(minimum) and .score(n) on the same handle in either order, and the evaluator runs only once. When test returns normally, NiceEval computes the final score automatically. Having no scoring items is also a valid result and yields a formal score: 0; it means the eval successfully produced a zero, not an execution failure or missing evidence.

Results of the two kinds

Kind	Success state	Explicit requirement fails	Evidence unavailable	Execution or evaluator error
`defineEval`	`passed`	`failed`	`errored`	`errored`
`defineScoreEval`	`passed` with a complete score; can enter the success ranking	`failed`; keeps the earned score, does not enter the success ranking	`errored` and not rankable when an item with a configured score or control is unavailable	`errored`; keeps only an auditable partial score

A complete score in a scored eval can be 0. passed + complete 0 means the eval successfully produced a zero, which is not an execution failure or missing evidence; failed + complete must still show as a failure and cannot be rewritten as success just because the value is complete.

The Judge is a measurement too

defineJudge declares a managed Judge Match. The author assembles the values needed for the judgment into material with semantic fields. The Judge can be passed straight to t.judge without registering it on the eval. The model defaults to the project’s judgeProvider; to switch only the model for one eval, set judge: "judge-model".

const explainsMotive = defineJudge({ rubric: "Given the change, rate whether the explanation states the motive for the change.",
});
const explainsRisk = defineJudge({ rubric: "Given the change, rate whether the explanation identifies specific risks.",
});

export default defineEval({
  async test(t) {
    const change = "Add the model version to the cache key";
    const turn = await t.send(`Explain the motive and risks of this change: ${change}`);
    const material = { change, explanation: turn.reply };
    t.judge(material, explainsMotive).gate(0.8).label("Motive for the change");
    t.judge(material, explainsRisk).gate(0.8).label("Specific risks");
  },
});

Requirements that can fail, be weighted, or show reasons independently should each register their own Judge. A scored eval can combine .gate(minimum) and .score(points) on the same Judge; the measurement times points gives the contribution, and the gate decides the Verdict. The same Judge runs only once.

Accept code tasks with real commands

await t.check(
  await t.sandbox.runCommand("pnpm", ["test"]),
  commandSucceeded(),
).label("Project tests pass");

When the scoring standard is itself a file, read hidden tests, reference implementations, or test scripts with loadText. Changing the file triggers a rerun of the matching evals; see Hidden Test Scoring.

Practical advice

Register each condition that must hold as a Boolean Assertion, so one Attempt collects complete failure information.
Split open-ended standards that can fail independently, be weighted independently, or need separate diagnosis into several Judges.
Use one Judge’s measurement times a single weight only for an indivisible overall quality dimension.
When later steps depend on a result, await handle.orStop().
Use defineScoreEval when a task has meaningful partial completion; it wraps up automatically when test returns normally.
Use the Judge for open-ended meaning; prefer deterministic Matches for files, commands, and structured output that can be checked exactly.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Score by Completion: Pass/Fail and Scored Evals

Use `defineEval` for what must be done

Continuous quality thresholds

Three steps done should still earn three points

Results of the two kinds

The Judge is a measurement too

Accept code tasks with real commands

Practical advice

Use defineEval for what must be done

Continuous quality thresholds

Three steps done should still earn three points

Results of the two kinds

The Judge is a measurement too

Accept code tasks with real commands

Practical advice

Use `defineEval` for what must be done