Sandbox Fixture: evaluate coding agents with tasks

When you evaluate a coding agent, checking a single reply is not enough. You usually need to give it a real project, let it read and write files, run commands, and commit changes, then check whether the output is correct. NiceEval’s current recommended shape is a normal .eval.ts file that explicitly prepares the Sandbox workspace, sends the task, and asserts on the result. See coding-agent-skill (a separate repository) for a runnable reference.

Recommended directory structure

evals/
├─ api-validation.eval.ts
├─ config-schema.eval.ts
└─ ponytail-safe-path.eval.ts

workspaces/
└─ ts-starter/
   ├─ package.json
   ├─ tsconfig.json
   └─ src/

skills/
└─ zod.md

experiments/
├─ baseline.ts
└─ with-skill.ts

workspaces/ holds the starting project the Agent can see. .eval.ts controls when the workspace is uploaded, the task prompt, and what to verify. experiments/ selects the Agent, the model, and whether to inject a Skill or plugin.

Prepare the workspace

Inside the eval, upload the starting project to the Sandbox:

import { defineEval } from "niceeval";

export default defineEval({
  description: "Add request-body validation to an Express route",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");

    await t.send("Implement POST /users in src/routes/users.ts.").then((turn) => turn.succeeded());

    t.sandbox.fileChanged("src/routes/users.ts");
  },
});

You can also use t.sandbox.writeText() to add seed files within a single eval — a good fit for small tasks or security probes.

Write the task prompt

The prompt should read like a real work order. Describe the goal and constraints, but do not leak the verification answer.

await t
  .send(
    `Implement environment validation in src/config/env.ts.
Requirements:
- Define EnvSchema with Zod
- Convert PORT to a number
- DATABASE_URL must be a valid URL
- JWT_SECRET must be at least 32 characters
- Export the env object and Env type`,
  )
  .then((turn) => turn.succeeded());

Leave implicit quality requirements, such as whether the code defends against path traversal or reuses existing tools, for the verification phase. That is how you find out whether a Skill or plugin actually helps the agent fill in missing context on its own.

Verify files and code

import { includes, excludes } from "niceeval/expect";

const code = await t.sandbox.readText("src/config/env.ts");

t.check(code, includes(/z\.object\s*\(/));
t.check(code, includes(/EnvSchema\.parse\s*\(\s*process\.env/));
t.check(code, excludes(/process\.env\.\w+\s*\?\?/));
t.sandbox.fileChanged("src/config/env.ts");

t.sandbox.fileChanged() is a scoped assertion: after the run ends, it uses the agent-attributed diff to determine whether the target file actually changed.

Run project tests or probe scripts

For coding-agent tasks, real commands are usually more reliable than text matching:

await t.sandbox.writeText(
  "_test_traversal.py",
  [
    "import sys, os",
    "sys.path.insert(0, '.')",
    "from uploads import safe_upload_path",
    "base = '/var/uploads'",
    "try:",
    "    p = safe_upload_path(base, '../../etc/passwd')",
    "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
    "except (ValueError, PermissionError, AssertionError):",
    "    pass",
    "print('ok')",
  ].join("\n"),
);

const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
t.check(result.stdout.trim(), includes(/ok/));

You can also run the project’s own test, lint, or build scripts:

import { commandSucceeded } from "niceeval/expect";

const result = await t.sandbox.runCommand("npm", ["test"]);
t.check(result, commandSucceeded());

Use experiments for baselines

Do not put Agent names, plugin names, or URLs into positionals. The Experiment selects the Agent and pins the run configuration; the CLI’s eval positionals only filter eval IDs.

pnpm exec niceeval exp compare
pnpm exec niceeval exp compare api-validation

A typical A/B setup:

experiments/baseline.ts: a plain agent.
experiments/with-skill.ts: the same agent plus a setup hook that writes CLAUDE.md.
Both groups use the same eval set, model, attempt count, and budget.

When to use Sandbox Fixture

The agent needs to modify real files.
You need to run the project’s tests, build, or lint.
You need to compare diffs.
You are evaluating a coding agent such as Claude Code or Codex.
You are comparing whether a Skill, plugin, or Hook improves real tasks.

If you are only evaluating a direct agent application, a plain direct adapter is lighter.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Sandbox Fixture: evaluate coding agents with tasks

Recommended directory structure

Prepare the workspace

Write the task prompt

Verify files and code

Run project tests or probe scripts

Use experiments for baselines

When to use Sandbox Fixture

​Recommended directory structure

​Prepare the workspace

​Write the task prompt

​Verify files and code

​Run project tests or probe scripts

​Use experiments for baselines

​When to use Sandbox Fixture

Recommended directory structure

Prepare the workspace

Write the task prompt

Verify files and code

Run project tests or probe scripts

Use experiments for baselines

When to use Sandbox Fixture