Prepare Tasks for a Coding Agent: Workspace, Prompt, and Acceptance

When evaluating a Coding Agent, looking only at what it replies is not enough. You give it a real project, let it read and write files and run commands, then check whether what it changed is right. All of this is written in an ordinary .eval.ts, in three steps:

Put the starting project into the Sandbox; this is the Fixture.
Give the Agent a specific task.
Accept the work with file contents, command results, and diffs, and hand open-ended quality to the Judge.

For a runnable reference, see coding-agent-skill (a separate repository).

Recommended directory layout

Keep the starting project and the evals apart. The Agent can only see what you upload into the Sandbox; eval code and verification materials stay outside:

evals/
├─ api-validation.eval.ts
├─ config-schema.eval.ts
└─ ponytail-safe-path.eval.ts

workspaces/
└─ ts-starter/
   ├─ package.json
   ├─ tsconfig.json
   └─ src/

skills/
└─ zod.md

experiments/
├─ baseline.ts
└─ with-skill.ts

workspaces/ holds the starting projects the Agent can see. The .eval.ts decides when to upload, what task to send, and how to check the result. experiments/ chooses the Agent and model, and whether to load a Skill or plugin.

Put the starting project into the Sandbox

For the Agent to work in a real project, the first step is uploading the project into the Sandbox:

import { defineEval } from "niceeval";

export default defineEval({
  description: "Add request body validation to an Express route",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");

    await t.send("Implement the POST /users route in src/routes/users.ts.").then((turn) => turn.succeeded());

    t.sandbox.fileChanged("src/routes/users.ts");
  },
});

The path is relative to the eval file. Uploaded files go under the Sandbox’s working directory, and the Agent can see them from the start. When only one or two files are missing, write them directly with t.sandbox.writeText(), which suits small tasks or safety check scripts. When fileChanged fails, first use pnpm exec niceeval show @<locator> --diff to see which files the Agent actually changed.

Write the task prompt

The prompt should read like a real ticket: state the goal and constraints clearly, but do not give away the verification answer.

await t
  .send(
    `Implement environment variable validation in src/config/env.ts.
Requirements:
- Define EnvSchema with Zod
- Convert PORT to a number
- DATABASE_URL must be a valid URL
- JWT_SECRET must be at least 32 characters
- Export the env object and the Env type`,
  )
  .then((turn) => turn.succeeded());

Quality requirements not written into the request, such as “does it prevent path traversal” or “does it reuse existing tools”, are checked later during verification. That is the only way to measure whether a Skill or plugin really made the Agent add those considerations on its own.

Verify files and code

After the Agent runs, read the files it changed and check that the key code is in place:

import { not, pattern } from "niceeval/expect";

const code = await t.sandbox.readText("src/config/env.ts");

t.check(code, pattern(/z\.object\s*\(/));
t.check(code, pattern(/EnvSchema\.parse\s*\(\s*process\.env/));
t.check(code, not(pattern(/process\.env\.\w+\s*\?\?/)));
t.sandbox.fileChanged("src/config/env.ts");

t.sandbox.fileChanged() only looks at the Agent’s changes during t.send() to decide whether it changed the target file. Files you uploaded or wrote yourself do not count. For the recording rules, see Sandbox Agent · Make file changes recordable.

Run project tests or check scripts

Text matching only shows what the code looks like; running it shows whether it is right. For Coding Agent tasks, real commands are usually more reliable. The example below writes a check script after the Agent finishes and runs it:

await t.sandbox.writeText(
  "_test_traversal.py",
  [
    "import sys, os",
    "sys.path.insert(0, '.')",
    "from uploads import safe_upload_path",
    "base = '/var/uploads'",
    "try:",
    "    p = safe_upload_path(base, '../../etc/passwd')",
    "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
    "except (ValueError, PermissionError, AssertionError):",
    "    pass",
    "print('ok')",
  ].join("\n"),
);

const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
t.check(result.stdout.trim(), pattern(/ok/));

You can also run the project’s own test, lint, or build scripts directly:

import { commandSucceeded } from "niceeval/expect";

const result = await t.sandbox.runCommand("npm", ["test"]);
t.check(result, commandSucceeded());

When commandSucceeded() fails, use pnpm exec niceeval show @<locator> to see the command’s exit code and output.

Use Experiments as control groups

To know whether a Skill or plugin helps, run the same tasks with and without it. The conditions go in the Experiment: it chooses the Agent and fixes the run configuration. The first command-line argument selects the Experiment, and later arguments filter evals by ID prefix. Do not put Agent names, plugin names, or URLs in command-line arguments.

pnpm exec niceeval exp compare
pnpm exec niceeval exp compare api-validation

A typical A/B comparison:

experiments/baseline.ts: the Agent with no extra configuration.
experiments/with-skill.ts: the same Agent, writing CLAUDE.md with a lifecycle Hook or a Plugin.
Both groups use the same evals, the same model, the same attempts, and the same budget.

After both groups run, compare them in the terminal with pnpm exec niceeval show --experiment baseline --experiment with-skill, or view them in the browser with pnpm exec niceeval view. When the two groups score almost the same, first confirm that CLAUDE.md really exists in the with-skill Sandbox.

When to use a Fixture

The Agent needs to modify real files.
You need to run the project’s tests, build, or lint.
You need to compare code changes.
You are evaluating a Coding Agent such as Claude Code or Codex.
You want to compare whether a Skill, plugin, or Hook improves results on real tasks.

If the system under test is an ordinary conversational application that needs no files or commands, connecting it with a conversational Adapter is lighter; see Connect Your Agent.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting