> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cut Losses When Credentials Expire or a Service Goes Down

> Learn which failures NiceEval retries automatically; when you find faults that are bound to fail, such as expired credentials or a disconnected shared service, stop the remaining Attempts and, after fixing them, rerun only what did not finish.

A batch of 200 evals reaches the 30th, and the API key expires. The remaining 170 are bound to fail, yet each one would still create a Sandbox, call the model, and consume budget.

NiceEval handles two cases:

* For transient failures such as rate limits and network blips, as long as it can confirm the Agent has not received the request yet, NiceEval retries automatically, and you do not need to do anything.
* For faults such as expired credentials or a disconnected shared service, retrying does not help. You can declare in code which scope it affects, and NiceEval stops dispatching the remaining related Attempts. After the fix, rerun only what did not finish; results already reached are kept as usual.

## First tell a run error from a wrong answer

Before fixing anything, check which kind the failed Attempt is:

* `errored`: the run itself did not complete and produced no result that can be scored. For example, the target service did not start, the protocol does not match, credentials are invalid, a dependency is missing, or the Adapter threw. Fix the run environment and rerun.
* `failed`: the Attempt completed, but some business assertion was not met. It may really expose a problem in the system under test, or it may mean a criterion is too strict or written wrong. First look at the same Attempt's input, output, assertions, and evidence, then decide whether to change the system under test or the eval.

Do not attribute a real business failure to infrastructure, and do not change criteria just to turn everything green.

A scored eval finishing only means scoring is over. How well it did depends on the score and each assertion; a low-score result does not count as full marks just because it finished.

To see an Attempt's error text, copy the locator the terminal gives you:

```sh theme={null}
pnpm exec niceeval show @<attempt-locator>
```

## Understand automatic retries

Automatic retries need no configuration. While waiting to retry, the Attempt's progress line shows something like:

```text theme={null}
turn retry 2/4 (rate_limit) — waiting 8s
```

After a successful retry, scoring and the conversation record keep only the successful Turn. When retries run out, the Attempt becomes `errored`, and the error contains a summary:

```text theme={null}
retries exhausted (4 attempts, rate_limit)
```

Without this summary, NiceEval did not retry the error. Requests the Agent has already received, or that cannot be confirmed as not received, are never retried automatically, to avoid repeating operations with side effects.

## Declare the affected scope when you find a fault

When you check a shared service or Fixture yourself, you can throw an error that carries the affected scope:

* `ExperimentFatalError` stops the Attempts of the same experiment that have not started. It suits shared services, shared credentials, and experiment-level configuration faults.
* `EvalFatalError` stops only the current eval's Attempts that have not started. It suits a Fixture that is deterministically missing, or prerequisite resources dedicated to that eval.

The eval below stops its own remaining Attempts when the Fixture is missing:

```ts theme={null}
// evals/coding/fix-button.eval.ts
import { defineEval, EvalFatalError } from "niceeval";

export default defineEval({
  async test(t) {
    const fixture = await t.sandbox.runCommand("test", ["-f", "src/Button.tsx"]);
    if (fixture.exitCode !== 0) {
      throw new EvalFatalError(
        "Fixture is missing src/Button.tsx. Run pnpm fixtures:sync first, then rerun the eval.",
      );
    }

    await t.send("Fix the Button keyboard focus issue.");
  },
});
```

When a shared service is unreachable, put the check in the `test(t)` callback of a real eval. First replace
`serviceHealthUrl` with the service's health check address, then use `ExperimentFatalError` to stop the remaining Attempts of the same experiment:

```ts theme={null}
// evals/coding/shared-service-health.eval.ts
import { defineEval, ExperimentFatalError } from "niceeval";

const serviceHealthUrl = "https://service.example.test/health";

export default defineEval({
  async test(t) {
    const health = await t.sandbox.runCommand("curl", ["-fsS", serviceHealthUrl]);
    if (health.exitCode !== 0) {
      throw new ExperimentFatalError(
        "The shared service is unreachable. Check the service and tunnel, then rerun the experiment.",
        { cause: health.stderr },
      );
    }
  },
});
```

The error message appears in the terminal and in the run record. State what happened and how to fix it next.

## Recognize a shared service that disconnects mid-run

Some shared services are fine when the experiment starts but disconnect during the run. Such failures are usually thrown by an SDK, CLI, or network library. You can recognize your own service address with the experiment's `classifyFailure`:

```ts theme={null}
// experiments/compare.ts
import { defineExperiment } from "niceeval";

const serviceHost = "memory.internal.example";

export default defineExperiment({
  // agent and sandbox configuration omitted
  classifyFailure({ text }) {
    const isOurService = text.includes(serviceHost);
    const isConnectionFailure = /ECONNREFUSED|ENOTFOUND|connection refused/i.test(text);

    if (isOurService && isConnectionFailure) {
      return {
        retryable: false,
        scope: "experiment",
        reason: "shared_service_unavailable",
      };
    }
    return undefined;
  },
});
```

Only recognize services whose affected scope you are sure of. Do not treat every `ECONNREFUSED` as an experiment-level fault, because the Agent may hit the same error when visiting other sites.

## After fixing, rerun what did not finish

Once loss-cutting takes effect, Attempts in that scope that have not started are no longer dispatched and are recorded as `unstarted`, and the run status shows `incomplete`. Attempts already running finish and record their results honestly; those that triggered the fault are recorded as `errored`.

First look at this run's results and confirm that the stopped scope matches what you expected. The Run ID is in the terminal's `NEXT` panel:

```sh theme={null}
pnpm exec niceeval show --run <run-id>
```

You can also view it in the browser with `pnpm exec niceeval view --run <run-id>`. To see an Attempt's raw error and retry summary, copy its locator:

```sh theme={null}
pnpm exec niceeval show @<attempt-locator> --execution
```

After fixing the service, credentials, or Fixture, run the original command again:

```sh theme={null}
pnpm exec niceeval exp compare evals/coding
```

`errored` and `unstarted` Attempts run again, and passed results are reused as usual. Add `--dry` before running to confirm which will rerun; for the rules, see [Rerun Only the Evals Affected by a Change](/docs/tutorials/rerun-and-cache).

If a custom Adapter can clearly recognize a protocol signal meaning "the request was rejected and not yet accepted", it can provide `classifySendFailure` to mark such failures as retryable. For field types and boundaries, see the [`defineSandboxAgent` reference](/docs/reference/define-agent#definesandboxagent).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.