Cut Losses When Credentials Expire or a Service Goes Down

A batch of 200 evals reaches the 30th, and the API key expires. The remaining 170 are bound to fail, yet each one would still create a Sandbox, call the model, and consume budget. NiceEval handles two cases:

For transient failures such as rate limits and network blips, as long as it can confirm the Agent has not received the request yet, NiceEval retries automatically, and you do not need to do anything.
For faults such as expired credentials or a disconnected shared service, retrying does not help. You can declare in code which scope it affects, and NiceEval stops dispatching the remaining related Attempts. After the fix, rerun only what did not finish; results already reached are kept as usual.

First tell a run error from a wrong answer

Before fixing anything, check which kind the failed Attempt is:

errored: the run itself did not complete and produced no result that can be scored. For example, the target service did not start, the protocol does not match, credentials are invalid, a dependency is missing, or the Adapter threw. Fix the run environment and rerun.
failed: the Attempt completed, but some business assertion was not met. It may really expose a problem in the system under test, or it may mean a criterion is too strict or written wrong. First look at the same Attempt’s input, output, assertions, and evidence, then decide whether to change the system under test or the eval.

Do not attribute a real business failure to infrastructure, and do not change criteria just to turn everything green. A scored eval finishing only means scoring is over. How well it did depends on the score and each assertion; a low-score result does not count as full marks just because it finished. To see an Attempt’s error text, copy the locator the terminal gives you:

pnpm exec niceeval show @<attempt-locator>

Understand automatic retries

Automatic retries need no configuration. While waiting to retry, the Attempt’s progress line shows something like:

turn retry 2/4 (rate_limit) — waiting 8s

After a successful retry, scoring and the conversation record keep only the successful Turn. When retries run out, the Attempt becomes errored, and the error contains a summary:

retries exhausted (4 attempts, rate_limit)

Without this summary, NiceEval did not retry the error. Requests the Agent has already received, or that cannot be confirmed as not received, are never retried automatically, to avoid repeating operations with side effects.

Declare the affected scope when you find a fault

When you check a shared service or Fixture yourself, you can throw an error that carries the affected scope:

ExperimentFatalError stops the Attempts of the same experiment that have not started. It suits shared services, shared credentials, and experiment-level configuration faults.
EvalFatalError stops only the current eval’s Attempts that have not started. It suits a Fixture that is deterministically missing, or prerequisite resources dedicated to that eval.

The eval below stops its own remaining Attempts when the Fixture is missing:

// evals/coding/fix-button.eval.ts
import { defineEval, EvalFatalError } from "niceeval";

export default defineEval({
  async test(t) {
    const fixture = await t.sandbox.runCommand("test", ["-f", "src/Button.tsx"]);
    if (fixture.exitCode !== 0) {
      throw new EvalFatalError(
        "Fixture is missing src/Button.tsx. Run pnpm fixtures:sync first, then rerun the eval.",
      );
    }

    await t.send("Fix the Button keyboard focus issue.");
  },
});

When a shared service is unreachable, put the check in the test(t) callback of a real eval. First replace serviceHealthUrl with the service’s health check address, then use ExperimentFatalError to stop the remaining Attempts of the same experiment:

// evals/coding/shared-service-health.eval.ts
import { defineEval, ExperimentFatalError } from "niceeval";

const serviceHealthUrl = "https://service.example.test/health";

export default defineEval({
  async test(t) {
    const health = await t.sandbox.runCommand("curl", ["-fsS", serviceHealthUrl]);
    if (health.exitCode !== 0) {
      throw new ExperimentFatalError(
        "The shared service is unreachable. Check the service and tunnel, then rerun the experiment.",
        { cause: health.stderr },
      );
    }
  },
});

The error message appears in the terminal and in the run record. State what happened and how to fix it next.

Handle a disconnected shared service

At the call boundary where you can establish the affected scope, catch the SDK error and throw ExperimentFatalError or EvalFatalError with the original error as cause. Experiment definitions do not accept classifyFailure. Built-in retry rules and the Adapter classifySendFailure hook remain available.

After fixing, rerun what did not finish

Once loss-cutting takes effect, Attempts in that scope that have not started are no longer dispatched and are recorded as unstarted, and the run status shows incomplete. Attempts already running finish and record their results honestly; those that triggered the fault are recorded as errored. First look at this run’s results and confirm that the stopped scope matches what you expected. The Run ID is in the terminal’s NEXT panel:

pnpm exec niceeval show --run <run-id>

You can also view it in the browser with pnpm exec niceeval view --run <run-id>. To see an Attempt’s raw error and retry summary, copy its locator:

pnpm exec niceeval show @<attempt-locator> --execution

After fixing the service, credentials, or Fixture, run the original command again:

pnpm exec niceeval exp compare evals/coding

errored and unstarted Attempts run again, and passed results are reused as usual. Add --dry before running to confirm which will rerun; for the rules, see Rerun Only the Evals Affected by a Change. If a custom Adapter can clearly recognize a protocol signal meaning “the request was rejected and not yet accepted”, it can provide classifySendFailure to mark such failures as retryable. For field types and boundaries, see the defineSandboxAgent reference.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting