Skip to main content
You wrote .before() on the Experiment, the eval group, and the eval, and configured installation steps for the Agent. Before actually running, you want to confirm the final order of these steps and each step’s parameters, instead of finding out the order is wrong after the Sandbox is built and dependencies took five minutes to install. exp --dry tells you which evals run and how many times. niceeval debug goes further: it lists one eval’s complete preparation, installation, execution, and cleanup steps in the Sandbox, without creating any Sandbox or calling any model.

Pick one Experiment and one eval

debug looks at one eval under one Experiment at a time:
Each argument must match exactly one item. An exact ID wins; when a prefix matches several, the CLI lists the candidates and stops, and you copy the full ID from them and run again. The eval is looked up only within the scope this Experiment already selects. When you pick one eval from an eval group, the plan does not add the other evals in the group, but it still shows the preparation steps the eval group contributes.

Read the plan in order

The output uses nested boxes: the outermost is the Experiment, and inside it come each Attempt and its lifecycle steps. Each sandbox.before step shows:
  • who declared this step (owner) and its ID (action);
  • its final execution number, and which eval it belongs to;
  • the changeFrequency value you set, where it came from, and the matching preset;
  • which steps it depends on, why it sits in this position, and the level it runs at (occurrence, currently always attempt);
  • whether the Provider can cache this step, and the cache status not yet checked at run time;
  • whether this step can be restored from cache (declared, cumulative, providerCoverage, and barrier in cache.state).
execution order is the real run-time order, not the order you wrote in code. The rules: dependencies always run first; when several steps can run, the one with the smaller changeFrequency value runs first; when the values are equal too, the declarer and declaration order decide. So a rarely changing step in an eval group can come before a frequently changing step in the Experiment.

Tell which steps can be restored from cache

  • declared is the state this step itself declares it will change.
  • cumulative includes the state changed by this step and every preparation step before it.
  • providerCoverage describes what the Provider can actually restore.
  • When barrier is not none, the output also gives barrierActionId, pointing to the first step that cannot be cached. The reason may be that its state declaration is the default all, that it is opaque, or that the Provider does not support it. This step and every step after it really execute.

Confirm commands and parameters

Hard-coded command steps show the shape of the command. Ordinary callbacks are marked OPAQUE, because they can only run at run time and debug cannot see what they do. Environment variables show only the variable name and a digest, and stdin shows only a digest and byte count. Credential values, raw stdin, and unsafe remote addresses never appear in the terminal or JSON output.

Let tools read the plan

The JSON is the same plan as the terminal output, without the box lines. Each step carries declarationOrder, executionOrder, changeFrequency, dependencies, occurrence, and cache. cache.state always gives declared, cumulative, providerCoverage, and barrier, plus barrierActionId when there is a barrier; steps that can be determined also give cache.prefixIdentity. debug does not read the run-time cache, so cache.lookup and cache.runtime.finalKey are always not-probed, and cache.runtime.status is always pending. debug accepts only --json. Run parameters such as budget, rerun, and concurrency do not apply to it.

It creates no resources

debug does not run before callbacks, test, after, ensure, or Provider cleanup functions, and does not create Runs, locks, Sandboxes, or build jobs, or write to the result file. It does load your Experiment definition and run the evals selection function. Providers may also read local files, call read-only CLIs, or query remote services during planning, so still run it only in trusted projects. To confirm which evals run and how many times each, see Confirm What Runs and How Many Times Before Running. When a real run fails and you need to get into the scene, see Keep the Sandbox to Debug Problems.