Skip to main content
The number of Attempts that run at once is determined by two concurrency limits: global concurrency slots control what this machine and Provider can sustain, while an Experiment’s own maxConcurrency controls how many of its Attempts it is willing to run in parallel. Work starts only after it passes both gates. When a run feels slow, concurrency is only one control. First identify where the time goes:

​
What the two concurrency limits control

Global precedence is --max-concurrency → maxConcurrency in config → the current Sandbox Provider’s recommended default (docker 10, e2b 20, vercel 1). The recommendation reflects Provider-side constraints. Use --max-concurrency to cap the rate of your own Agent interface.

​
How slots are allocated

This batch mixes work: there are three global slots, fast is a regular Experiment, and slow declares maxConcurrency: 1. Three things follow from this diagram:
  • slow always occupies one lane. Its second Attempt starts only after the first one’s teardown and Sandbox destruction finish. Other Experiments in the batch are not held back; they use the remaining slots normally.
  • Slots favor the Experiments that need the most rounds to finish. Mixing fast and slow Experiments in one command is safe. You do not need to make waves manually or reserve slots for fast tasks. Fast work fills gaps as they open.
  • An Attempt in backoff releases a global slot, but not its Experiment’s own slot. When rate-limited, the live panel’s running count can exceed the limit by exactly the number of Attempts in backoff. At any instant, the work actually executing still does not exceed the limit. This is not failed concurrency control.
Attempts waiting for Experiment-level setup—for example, while starting a tunnel or shared service—neither hold nor reserve a concurrency slot and remain queued. A slow-starting tunnel does not turn a long period of “0 running, N queued” into a concurrency configuration problem.

​
Confirm applied concurrency in the live panel

The PLAN panel first tells you how many lanes this execution will open and where that number came from. Experiments that declare maxConcurrency follow with their own caps:
(from flag) means 19 came from --max-concurrency. When neither the flag nor config supplies a value, you see a Provider recommendation such as (from vercel default): one lane is not a scheduler failure; vercel recommends one. slow ≤1 means this Experiment is bounded by its own cap, so increasing --max-concurrency will not make it open more lanes. During execution, inspect the top-line count:
When running stays at the limit while queued steadily drains, concurrency slots are the bottleneck, so increase the limit if capacity remains. When running remains below it, the bottleneck is elsewhere: an Experiment’s own cap (listed in PLAN), Provider-exclusive serialization, or waiting for Experiment-level setup.

​
Serialize work that shares state across Attempts

When several evals load, modify, and write back the same host file or central service state, reduce this Experiment to one lane:
maxConcurrency: 1 serializes only Attempts within this Invocation. sharedState.key protects the same checkpoint across Invocations in the same project Coordination domain. The lease is acquired before Experiment and Sandbox setup, then released only after Sandbox teardown, the Provider finalizer, and Experiment teardown finish. A waiting Invocation does not create a Sandbox early. After it acquires the lease, it continues its own plan; it does not read or carry results from another Run. sharedState provides mutual exclusion only. It does not store the checkpoint, make write-back atomic, roll changes back, or repair a partial business write after a forced kill. Write back atomically when possible; otherwise use a new key and rebuild a clean cohort. Coordination across machines still belongs to external orchestration. NiceEval does not automatically take over a lease based on a timeout, PID check, or heartbeat age. A paused owner—even a process stopped with SIGSTOP—continues to hold its lease. After a forced kill or cleanup failure, the lease remains held and waiters remain blocked. This prevents two Invocations from modifying the same state. After you confirm that the original owner has terminated and external state is quiescent, use the public recovery inspection command to inspect the exact owner token. Then recover explicitly with that token and both confirmation flags. When only one Agent service rate-limits frequently, set maxConcurrency: N on that Experiment instead of lowering the global cap. Lowering the global cap harms other Experiments in the batch. During backoff, an Experiment cap does not admit the N+1 Attempt, so a rate-limited service is not put under even more pressure.

​
Store Hook state by Sandbox

Under concurrency, the same module serves more than one Sandbox at once. A handle obtained in setup cannot live in a regular module variable because a later concurrent Attempt would overwrite it. Store it by Sandbox instance:
If those Attempts really must read and write the same business state in order, do not use a WeakMap to hide that meaning. Set the Experiment to maxConcurrency: 1 directly. When only a few eval groups need a shared environment, use defineEvalGroup() to list compatible members explicitly. Each group dispatches only one Attempt at a time and reuses at most one Sandbox. Other groups and ungrouped evals keep competing for the remaining concurrency slots. You do not need to set the entire Experiment to maxConcurrency: 1. The evals array declares members, not business order. The Runner serializes normalized eval IDs stably. When attempts > 1, the Attempts that must actually dispatch for one member enter the group lane consecutively. See Reuse Sandboxes with Eval Groups for the complete directory layout, Experiment shape, and --dry output. Eval groups are not task dependency graphs. Result carry and CLI filtering can leave an earlier item unexecuted, while the reset between tasks can remove its workdir changes. Put steps that require a later step to read an earlier step’s file into the same eval; do not rely on group dispatch order for correctness.

​
You do not need serialization just for an ordered display

Results always appear in discovery order, independent of the current concurrency. Terminal and Report rows are stable and easy to diff as a result. If ordered output is all you want, stop there: naming prefixes still work and you do not need to sacrifice throughput.

​
Make early exit actually save money

earlyExit stops Attempts that have not been dispatched yet. With normal concurrency, several Attempts for the same eval can already be dispatched together. By the time the first one passes, the later ones are already running, so no money is saved. To get the one-after-another behavior—run once, stop after a pass, and only run the next attempt after a failure—use:

​
Open another terminal against the same project Record

One project’s .niceeval/record.sqlite allows multiple Invocations to publish results, but it does not let two Invocations divide work automatically. When the machine, Provider, and Agent service all have remaining capacity, both terminals can run in the same project:
Each side plans its complete selection independently. They can execute the same Evals twice because neither side reads the other’s unpublished Attempts. Each Attempt becomes readable as soon as it is published, even while its source Run is active. To isolate results, run in another project directory; NiceEval does not merge two Records. There are four boundaries:
  • The two CLI caps add together; Provider capacity does not. Lower both when capacity is tight.
  • An Experiment’s maxConcurrency also counts separately per terminal: if both commands declare 3, up to six Attempts run in total.
  • The two Sandbox pools are independent too. Declare sharedState.key only when an external checkpoint is genuinely shared; it does not merge Records.
  • Concurrent Invocations can share a project Record. Do not copy .niceeval/record.sqlite while runs are active; copy or archive that one file only after a command completes its portable gate successfully.

​
Boundaries

  • A concurrency cap controls resource use, not spending. Use --budget to cap cost.
  • Providers that declare exclusive serialization have a Provider-level serial limit. --max-concurrency does not remove it; it is a correctness constraint, not a scheduling option.
  • For Agent services that bill or rate-limit by the number of concurrent runs, setting a global cap exactly at the service limit is unstable: a slot released for backoff is immediately filled by a new Attempt, so service-side concurrency stays at the limit. In one terminal, leave headroom with an Experiment-level maxConcurrency. Across terminals, values add together, so lower each side for an account-level quota or use external orchestration.
  • Experiments that declare sandboxReuse: true run serially inside each Sandbox and in parallel across Sandboxes. See Reuse Sandboxes.
  • Eval groups serialize stably within a group while different groups still run in parallel. See Eval Groups.

​
Continue reading

Rerun and Carry Results

See which results do not cost money again in this run.

Reuse Sandboxes

Another path to speed: pay common preparation once.

Eval Groups

Reuse a Sandbox with stable serial order within a group while other groups keep running in parallel.

Write Experiments

Where maxConcurrency and Experiment-level lifecycle belong.

Runner

The full mechanism for discovery, dispatch, retries, and budgets.