Install the CLI for a Custom Sandbox Agent

You are evaluating a Coding Agent CLI that the built-in adapters do not support yet, or you need to pin a specific version. Installing it into the image by hand every time is error-prone: when the version silently upgrades, two eval runs are no longer comparing the same Agent. Instead, declare the CLI’s name, version, and installation method on a custom Sandbox Agent. Before each run, NiceEval checks the CLI in the Sandbox; when the version is wrong, it installs the version you specified, checks again after installing, and starts the eval only when everything is correct.

Declare the CLI’s version and installation method

First spell out on the Agent which CLI, which version, and how to install it. For a Node.js CLI, use createNpmCliInstaller. The code below connects the npm package my-agent@1.0.0 to a custom Adapter:

// agents/my-agent.ts
import {
  createNpmCliInstaller,
  defineSandboxAgent,
  agentBin,
  responsesEvidenceCoverage,
  turnFromResponses,
  type ResponseLike,
} from "niceeval/adapter";

const { ensure, installer } = createNpmCliInstaller({
  identity: {
    agent: "my-agent",
    version: "1.0.0",
    revision: "1",
  },
  packageName: "my-agent",
  bin: "my-agent",
  progress: {
    checking: "Checking my-agent CLI 1.0.0",
    installing: "Installing my-agent CLI 1.0.0",
    ready: "my-agent CLI 1.0.0 is ready",
  },
});

export default defineSandboxAgent({
  name: "my-agent",
  evidenceCoverage: responsesEvidenceCoverage,
  ensure,
  installers: [installer],

  async setup(sandbox) {
    await sandbox.writeText(".my-agent/config.json", JSON.stringify({ telemetry: true }));
  },

  async send(input, ctx) {
    const result = await ctx.sandbox.runCommand("sh", [
      "-lc",
      `exec ${agentBin("my-agent")} run --json "$1"`,
      "my-agent",
      input.text,
    ]);
    if (result.exitCode !== 0) {
      throw new Error(`my-agent exit code ${result.exitCode}: ${result.stderr}`);
    }
    return turnFromResponses(JSON.parse(result.stdout) as ResponseLike);
  },
});

This example assumes my-agent run --json outputs a JSON object compatible with OpenAI Responses. turnFromResponses converts its output and usage into a NiceEval Turn. When the CLI outputs another format, replace this line with your own mapping to events, status, and usage; see Write send.

Write the CLI’s exact version in identity.version, not latest. When the version changes silently, two eval runs are not comparing the same Agent.
When the version stays the same but the installation steps change, raise identity.revision as well.
setup only writes auth and run configuration. Leave installation to installer, so the version is checked before every Attempt starts.
The text in progress is shown in the terminal while checking, installing, and when ready.

Run it in an Experiment

Put the Agent and the Sandbox Provider in an Experiment:

// experiments/my-agent.ts
import { defineExperiment } from "niceeval";
import { dockerSandbox } from "niceeval/sandbox";
import myAgent from "../agents/my-agent.ts";

export default defineExperiment({
  agent: myAgent,
  model: "acme-model-v2",
  sandbox: dockerSandbox({ source: { type: "image", image: "node:22-slim" } }),
});

pnpm exec niceeval exp my-agent fixtures/button

During the run, the terminal shows the checking, installing, and ready text from progress in turn. When the Sandbox already has the correct version, NiceEval uses it directly and skips installation. After the run, view results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view. An ordinary npm package requires Node.js and npm in the Sandbox. createNpmCliInstaller packs on the host, uploads, then runs npm install -g in the Sandbox. Only a platformPackage that bundles its own runtime does not need Node.js and npm.

Check installation failures

When installation or the post-install recheck fails, the Attempt becomes errored in the agent.ensure phase, and the terminal gives its locator:

pnpm exec niceeval show @<attempt-locator>

First check the package name, version, and target platform in the error, then confirm the image has Node.js and npm. After correcting identity, the npm package, or the installation steps, run the original command again.

Choose another installation mode

When packing on the host and uploading does not suit your case, AgentInstaller has two other modes:

sandbox-network: let the Sandbox install the CLI over its own network. Use it only when the task explicitly allows this network access.
verify-only: accept only an exact version preinstalled in the image. When the version is wrong, the Attempt reports an error directly without trying to install.

Both modes require you to provide your own AgentInstaller. For the full fields, see the defineSandboxAgent reference.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Install the CLI for a Custom Sandbox Agent

Declare the CLI’s version and installation method

Run it in an Experiment

Check installation failures

Choose another installation mode