Sandbox Fixtures: Evaluate Coding Agents with Tasks

When you evaluate a Coding Agent, checking one reply is not enough. You normally give it a real project, let it read and write files, run commands, and commit changes, then check whether the artifact is correct. NiceEval’s current recommended approach is an ordinary .eval.ts file that explicitly prepares a Sandbox workspace, sends the task, and asserts the result. For a runnable reference, see coding-agent-skill (a separate repository).

Recommended directory structure

evals/
├─ api-validation.eval.ts
├─ config-schema.eval.ts
└─ ponytail-safe-path.eval.ts

workspaces/
└─ ts-starter/
   ├─ package.json
   ├─ tsconfig.json
   └─ src/

skills/
└─ zod.md

experiments/
├─ baseline.ts
└─ with-skill.ts

workspaces/ holds the starting project visible to the Agent. .eval.ts controls when the Workspace uploads, the task prompt, and result checks. experiments/ select the Agent, model, and whether to inject a Skill or plugin.

Prepare the workspace

Upload the starting project to the Sandbox in the eval:

import { defineEval } from "niceeval";

export default defineEval({
  description: "Add request-body validation to an Express route",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");

    await t.send("Implement POST /users in src/routes/users.ts.").then((turn) => turn.succeeded());

    t.sandbox.fileChanged("src/routes/users.ts");
  },
});

You can also use t.sandbox.writeText() to add seed files in one eval, which suits small tasks or security probes.

Write the task prompt

A prompt should read like a real work order. Describe the goal and constraints, but do not disclose the verification answer.

await t
  .send(
    `Implement environment variable validation in src/config/env.ts.
Requirements:
- Define EnvSchema with Zod
- Convert PORT to a number
- DATABASE_URL must be a valid URL
- JWT_SECRET must be at least 32 characters
- Export the env object and Env type`,
  )
  .then((turn) => turn.succeeded());

Leave implicit quality requirements such as path-traversal protection or reuse of existing tools for the verification phase. That is how you test whether a Skill or plugin actually helps the Agent fill in context on its own.

Verify files and code

import { not, pattern } from "niceeval/expect";

const code = await t.sandbox.readText("src/config/env.ts");

t.check(code, pattern(/z\.object\s*\(/));
t.check(code, pattern(/EnvSchema\.parse\s*\(\s*process\.env/));
t.check(code, not(pattern(/process\.env\.\w+\s*\?\?/)));
t.sandbox.fileChanged("src/config/env.ts");

t.sandbox.fileChanged() is a scoped assertion: after the run ends, it uses the Agent-attributed diff to determine whether the target file truly changed.

Run project tests or probe scripts

For Coding Agent tasks, real commands are usually more reliable than text matching:

await t.sandbox.writeText(
  "_test_traversal.py",
  [
    "import sys, os",
    "sys.path.insert(0, '.')",
    "from uploads import safe_upload_path",
    "base = '/var/uploads'",
    "try:",
    "    p = safe_upload_path(base, '../../etc/passwd')",
    "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
    "except (ValueError, PermissionError, AssertionError):",
    "    pass",
    "print('ok')",
  ].join("\n"),
);

const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
t.check(result.stdout.trim(), pattern(/ok/));

You can also run the project’s own test, lint, or build scripts:

import { commandSucceeded } from "niceeval/expect";

const result = await t.sandbox.runCommand("npm", ["test"]);
t.check(result, commandSucceeded());

Use Experiments for a baseline

Do not put an Agent name, plugin name, or URL in positional arguments. The Experiment selects the Agent and fixes run configuration. CLI eval positionals filter eval IDs only.

pnpm exec niceeval exp compare
pnpm exec niceeval exp compare api-validation

A typical A/B setup:

experiments/baseline.ts: an Agent with no extra configuration.
experiments/with-skill.ts: the same Agent with a setup Hook that injects CLAUDE.md.
Both groups use the same eval set, model, attempts, and budget.

When to use a Sandbox Fixture

The Agent needs to modify real files.
You need to run the project’s tests, build, or lint.
You need to compare diffs.
You need to evaluate a Coding Agent such as Claude Code or Codex.
You need to compare whether a Skill, plugin, or Hook improves real tasks.

If you only evaluate a direct Agent application, a normal direct Adapter is lighter.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Sandbox Fixtures: Evaluate Coding Agents with Tasks

Recommended directory structure

Prepare the workspace

Write the task prompt

Verify files and code

Run project tests or probe scripts

Use Experiments for a baseline

When to use a Sandbox Fixture

​Recommended directory structure

​Prepare the workspace

​Write the task prompt

​Verify files and code

​Run project tests or probe scripts

​Use Experiments for a baseline

​When to use a Sandbox Fixture

Recommended directory structure

Prepare the workspace

Write the task prompt

Verify files and code

Run project tests or probe scripts

Use Experiments for a baseline

When to use a Sandbox Fixture