Skip to main content
200 evals took two hours, and you want them to go faster. The first instinct is to raise concurrency, but concurrency is only one possible cause. Unchanged tasks may be running again, every eval may be creating a new Sandbox, or the Provider’s quota may hit its limit first. When you do decide to tune concurrency, remember that the number of Attempts running at once has two levels of limits: global concurrency controls how much this machine and the Provider can handle, and the Experiment’s own maxConcurrency controls how many Attempts this Experiment runs in parallel at most. An Attempt starts only when both levels have available capacity. First find where the time goes, then change the matching setting:

What each concurrency limit controls

The number of Attempts that can run at once is decided by the following together. If any of them has no available capacity, the Attempt queues: Global concurrency takes its value in this order: --max-concurrency, then maxConcurrency in the config, and finally the current Sandbox Provider’s recommended value (docker 10, e2b 20, vercel 1). The recommended value reflects limits on the Provider side. When your own Agent interface is rate-limited, lower it with --max-concurrency:

How concurrency capacity is shared

Below is a mixed batch: a global concurrency limit of 3, fast is an ordinary Experiment, and slow declares maxConcurrency: 1. This diagram shows three things:
  • slow always occupies one lane. Its second Attempt waits until the first one’s teardown and Sandbox destruction finish. Other Experiments in the batch are not affected and keep using the remaining capacity.
  • Capacity goes first to the Experiment that still has the most rounds to run. Running fast and slow Experiments in one command is safe. You do not need to split them into batches by hand or reserve capacity for fast tasks. Fast tasks fill available capacity automatically.
  • An Attempt backing off to retry gives up its global capacity, but not its Experiment’s own capacity. So when you hit rate limits, running in the live panel can exceed the limit, by exactly the number of Attempts backing off. At any moment, the work actually executing still does not exceed the limit. Concurrency control has not failed.
Attempts waiting for Experiment-level setup (starting a tunnel, starting shared services) do not consume concurrency capacity and show as queued in the counts. So when a tunnel is slow to start, the panel shows “0 running, N queued” for a long time; the problem is in setup, not in the concurrency configuration. Attempts waiting for Docker profile capacity also show as queued, with the queue reason provider-capacity. They do not show creating sandbox and do not consume ordinary concurrency capacity. Once the profile has room, the Attempt acquires concurrency capacity again and enters running only after it is assigned resources. Attempts in the same batch that use other Providers can keep starting and are not held up by Docker tasks ahead of them.

Confirm in the live panel that concurrency took effect

When a run starts, the PLAN panel tells you how many lanes this run opens and where that number came from. Experiments that declare maxConcurrency follow, each with its own cap:
(from flag) means 19 came from --max-concurrency. When no flag was passed and the config does not set one, this shows a Provider recommendation such as (from vercel default). Opening only 1 lane is not a scheduling problem; the vercel recommendation is 1. slow ≤1 means this Experiment is bounded by its own cap, and raising --max-concurrency will not make it open more. During the run, watch the top-line counts:
  • running stays at the limit while queued drops steadily: concurrency is the bottleneck. If the machine still has capacity, raise it.
  • running stays below the limit for a long time: the bottleneck is elsewhere. It may be the Experiment’s own cap, Provider capacity, or waiting for Experiment-level setup. Attempts waiting for capacity show waiting for provider capacity on their activity line.

Run serially when Attempts share state

When several evals all need to read, modify, and write back the same host file or server-side state, reduce this Experiment to one lane:
maxConcurrency: 1 only makes the Attempts within this one niceeval exp command serial. When another terminal runs at the same time, sharedState.key ensures the two sides never modify the same checkpoint at once:
  • The lease is acquired before the Experiment’s and Sandbox’s .before(), and released only after Sandbox cleanup, Provider finalization, and Experiment teardown all finish.
  • The waiting side does not create a Sandbox early. After it gets the lease, it continues with its own plan and does not read or reuse the other side’s results.
sharedState only prevents simultaneous modification. It does not save the checkpoint, and it does not handle atomic commit, rollback, or recovery from a half-finished write after a forced kill. Write back atomically when you can. When you cannot, switch to a new key and rebuild from a clean state. Coordination across machines still needs external orchestration. NiceEval never hands a lease to someone else because of a timeout, process ID, or heartbeat. A paused holder still holds the lease. After the holder is force-killed or fails to clean up, the waiting side keeps waiting, so the two sides never modify the same state at once. After you confirm that the holder has exited and the external state no longer changes, use the commands in Recover After a Forced Kill to inspect the holder and recover manually. When only one Agent service is often rate-limited, also set maxConcurrency: N on that Experiment instead of lowering the global limit, which would slow down the other Experiments in the batch. During backoff, the Experiment-level cap does not let an N+1th Attempt in, so a rate-limited service is not put under more pressure.

Store Hook state by Sandbox

Under concurrency, the same module serves several Sandboxes at once. A handle obtained in setup cannot be stored in an ordinary module variable, or the next concurrent Attempt overwrites it. Store and look it up keyed by the Sandbox instance:
If these Attempts really must read and write the same business state in order, do not hide that with a WeakMap. Reduce the Experiment to maxConcurrency: 1 directly. When only a few groups of evals need a shared environment, list the members of each group with defineEvalGroup(). Each group runs only one Attempt at a time and reuses at most one Sandbox. Other groups and ungrouped evals keep using the remaining concurrency capacity. You do not need to set the whole Experiment to maxConcurrency: 1. The evals array only lists members; it does not decide the order. Within a group, evals run sorted by eval ID. When attempts > 1, the Attempts of one eval run back to back before the next eval’s turn. For the full directory layout, Experiment shape, and --dry check, see Reuse Sandboxes with Eval Groups. An eval group is not a task dependency graph. Reusing last results and command-line filtering can leave an earlier eval unexecuted, and the reset between tasks also undoes an earlier eval’s changes to workdir. When a later step must read a file from an earlier step, put them in the same eval. Do not depend on the order within a group.

Ordered output does not require serial runs

Results are always output in discovery order, regardless of how much concurrency this run used. Row order in the terminal and in reports is therefore stable, and you can diff them directly. If ordered output is all you want, you are done here: naming prefixes still work, and you do not have to give up throughput.

Make early exit actually save money

earlyExit stops Attempts that have not been dispatched yet. Under default concurrency, several Attempts of the same eval may already be running at the same time. When the first one passes, the later ones are already running, so no money is saved. To get “run once, stop if it passes, run the next only if it did not”:

Share the result file across terminals

Two terminals can run in the same project at the same time, and both write results into .niceeval/record.sqlite. But they do not divide the work automatically. After confirming that the machine, Provider, and Agent service have spare capacity, run directly in both terminals:
Each side plans the complete selection on its own and may execute the same evals twice, because neither can see the other’s unfinished Attempts. An Attempt can be viewed as soon as it completes, even while its Run is still running. When you need to keep results apart, run in another project directory; NiceEval does not merge results from two projects. When running several terminals:
  • The --max-concurrency values of both sides add up; Provider capacity does not. When quota is tight, lower both.
  • The Experiment’s own maxConcurrency is also counted per terminal: when both commands declare 3, up to 6 run in total.
  • The two terminals’ Sandboxes are independent as well. Declare sharedState.key only when you really share an external checkpoint; it has nothing to do with the result file.
  • Do not copy .niceeval/record.sqlite while runs are in progress. Copy or archive the file after the command finishes successfully.
After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view.

Boundaries

  • Concurrency limits control resource use, not spending. To limit spending, use --budget; see Control Run Cost.
  • Some Providers declare that they can only run one at a time, and no --max-concurrency value changes that. It is a correctness requirement, not a scheduling parameter.
  • When an Agent service bills or limits by “number of tasks running at once”, setting the global limit right at the service limit does not hold steady: capacity released during backoff is immediately filled by a new Attempt, so concurrency on the service side stays at the limit. In a single terminal, leave headroom with an Experiment-level maxConcurrency. With several terminals, the values add up, so lower each side separately for an account-level quota, or hand it to external orchestration.
  • An Experiment that declares sandboxReuse: true runs one after another inside each Sandbox, and Sandboxes run in parallel; see Reuse Sandboxes.
  • An eval group runs in order internally, and different groups can still run in parallel; see Eval Groups.

Rerun and Reuse

Which results do not cost money again this time.

Reuse Sandboxes

Another path to speed: pay for shared preparation only once.

Eval Groups

Reuse a Sandbox serially and stably within a group while other groups keep running in parallel.

Write Experiments

Where maxConcurrency and Experiment-level lifecycle go.

Runner

The full mechanism of discovery, dispatch, retries, and budgets.