Skip to main content
A full eval run calls the model hundreds of times, plus Sandbox costs, so the bill easily exceeds expectations. Before controlling cost, decide what you want to save: the number of repeated runs, a ceiling for when things go wrong, or the resources used at the same time. Each setting controls one thing, and they cannot replace each other.

See how much will run before running

Before spending money, confirm how many evals and Experiments this run selects, and how many times each eval runs:
--dry calls no model and creates no Sandbox. The first line gives the total number of Attempts in this plan, and each following line is one eval, marking which will reuse last results and which will run again. --dry does not predict the final cost. Its job is to catch a wrong scope or too many repetitions before dispatch. When the total is much larger than expected, check the Experiment’s evals selection and attempts first.

Stop at the first pass when you only care whether it can succeed

When you only want to know “can this task be done” and do not care about the pass rate distribution, turn on early exit:
After one Attempt of an eval gets passed, that eval’s Attempts that have not started are no longer dispatched and are recorded as not-dispatched. Attempts of the same eval that are already running receive a cancellation signal through ctx.signal; how quickly they end depends on whether the Adapter responds to that signal promptly. Attempts of the same eval that return after early exit triggers do not enter the results. failed, errored, or skipped do not trigger early exit. Do not turn this on when you need the full pass rate distribution. Under default concurrency, several Attempts of the same eval may already be running at once, so nothing is saved; to make it really save money, see Make early exit actually save money.

Set a protective spending ceiling

When you worry that a run might get out of hand and spend more than expected, set a dollar ceiling:
The budget is calculated separately for each Experiment ID. As each Attempt completes, NiceEval estimates its cost (estimatedCostUSD) from the Experiment’s model, the reported token usage, and the price table, then adds it to the budget. The actual amount usage.costUSD reported by a Provider or Adapter is only for display and does not take part in the budget decision. When the cumulative estimate reaches the ceiling, NiceEval stops dispatching new Attempts; those already running are not stopped midway. So --budget is a protective ceiling, not an exact bill: while several concurrent tasks have not settled, the final cost can briefly exceed the setting. When the budget runs out, you see:
  • a budget_exhausted diagnostic;
  • Attempts that were not dispatched in time recorded as not-dispatched;
  • the command exiting with a non-zero status.
After raising the budget, rerun the same scope. Attempts that already completed and qualify for reuse are reused directly, and only the remainder runs. When model, token usage, or a matching price is missing, NiceEval cannot compute an estimated cost, and usage.costUSD cannot fill the gap. The run then gives a warning instead of pretending the budget is still in effect. When you see this warning, first confirm the Experiment sets model and the Adapter reports token usage.

Concurrency only controls resource use

When the local machine or the Provider cannot keep up, limit the number of Attempts running at once:
A concurrency limit is not a cost limit; cost protection still uses --budget. Sandbox build concurrency is controlled by a separate flag, --max-build-concurrency. To also spread out the creation and preparation time of each Sandbox, see Reuse Sandboxes. To choose between rerunning only failures and revalidating everything, see Rerun and Reuse Results.

See how much was spent

After the run, view the cost of one Attempt or a whole experiment in the terminal:
To browse in the browser, run pnpm exec niceeval view. The Attempt page first shows the combined cost of the application and the Judge:
  • When the cost is complete, it shows the total.
  • When receipts, prices, or complete call records are missing, it shows the known subtotal and which parts are missing.
  • An explicitly recorded zero amount counts as a known cost. Money already spent on failures and retries is kept too.
Model purpose, each actual call, and Judge costs are in the breakdown below. The experiment page sums the latest result of each Attempt of each eval, counting the same original Attempt only once. Old results replaced by a rerun are not added, and results that were not rerun are kept. So it shows the cost of the current batch of results, not a bill for all historical runs. Omitting a few rows in the breakdown does not reduce the total; different currencies are shown separately without exchange-rate conversion.

Have a custom Adapter report cost

For costs from your own application Adapter to appear on the pages above, report the usage of each call:
  • Use ctx.recordUsage(call) to report the final snapshot of each physical call.
  • After all records are obtained, call ctx.sealUsage({ state: "complete" }) to tell NiceEval this record is complete.
  • When only part of the records could be read, call ctx.sealUsage({ state: "partial", reason: "source-incomplete" }).
An empty record that was not declared complete shows as an unknown cost, never as zero.

Check cache read rate by model purpose

A Headless Adapter registers real calls with ctx.recordUsage and calls ctx.sealUsage after collection ends. You can then set a cache read rate threshold directly with t.usage.cacheRead:
The read rate is total cache read tokens / total input tokens including cache across calls in the same group, including failed calls and retries. It is not an average of per-call percentages. Each call uses inputTotalTokens first; without a separate total, it must provide all three mutually exclusive buckets: inputTokens, cacheReadTokens, and cacheWriteTokens. cacheRead.total gives the result for the whole Attempt, roles groups by purpose, and models further groups by the actual provider and model. Unknown identities stay null and are not filled in from configured values. coveredCalls / recordedCalls shows the input coverage of registered calls; collectionComplete separately shows whether collection is complete. Missing input, no seal, a partial seal, an unknown call state, an empty group, or a zero denominator all return unavailable, with the causes listed in reasons. Missing data is not treated as zero; only a complete record with a positive input total and no cache reads yields the number 0. An old ledger without an explicit producer completeness declaration also returns unavailable. Use niceeval show <Attempt locator> --usage or View to see the read rate, coverage, and causes; scripts read the same report from usage.cacheRead in attempt.usage. Existing recordUsage calls need no renaming, but they must register cache tokens and seal honestly after collection completes. Historical result files need no migration. Slot names come from the Experiment’s models keys, not a built-in list. The single-model model shorthand maps to default. Passing an undeclared slot throws at the call site and lists the available names. A declared slot with no calls in a complete collection returns role-has-no-calls; an incomplete collection still returns collection-incomplete. An Eval can be reused across Experiments, so this API validates roles inside the running callback rather than checking dynamic strings at definition time.

Record usage from a command-line Agent

Record actual calls from external receipts and set modelRole to a purpose declared by the Experiment, such as mind. Do not record both streaming increments and the final receipt for the same request. Merge them by request ID first, then record the final receipt.
Claude’s input_tokens is the uncached input bucket. Total input includes the cache read and write buckets; see Claude cache usage. When Codex usage provides input_tokens inclusive of cached tokens, map it to inputTotalTokens and map cached_input_tokens to cacheReadTokens. Do not add the cached tokens to the total again. A successful command exit does not prove that every model call was recorded. Check whether your CLI version reports individual requests, Turns, or session totals. When only session totals are available, do not represent several physical calls as one call or record both totals and details. recordUsage currently records physical calls. Save session totals in a statistics attachment and seal as partial; cache criteria will remain unavailable. When no usage is exposed, also seal as partial, without filling in zeros or claiming a complete empty ledger:
Call ctx.sealUsage({ state: "complete" }) only after obtaining the final records for all physical calls.

Type-check model purpose names

Starting with 0.18, shared declarations let you reuse an eval across experiments while checking purpose names at compile time:
models.ts
Declare requirements on the factories bound to your existing Adapter. You can also destructure adapter.defineScoreEval, but the factory with the same name at the package root does not accept requires:
evals/cache.eval.ts
Use defineExperiment({ adapter, models: roles, evals: ["cache"] }) in the experiment. Other experiments can choose different models for these purposes. requires requires the purposes to exist, without requiring identical model values. All evals selected by the experiment are checked when it loads. CLI tag or prefix filters cannot hide missing purposes. Pass requires: { models: roles } to defineAdapter as well to narrow both ctx.models and the modelRole accepted by ctx.recordUsage. A null or omitted modelRole still represents a call with no assigned purpose. When a declared purpose has no calls in this run, rate returns unavailable material with the reason role-has-no-calls. The 0.17.x syntax that omits requires remains compatible with 0.18. Names are still validated at the call site. To migrate, first extract a shared declaration, then add requires to the Adapter and its bound evals separately. Do not annotate the declaration as Record<string, ModelRoleSelection> first. That loses the specific keys, so TypeScript cannot detect typos.