> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Sandbox Fixture: evaluate coding agents with tasks

> Use .eval.ts to give a coding agent an isolated workspace, send it a real task, and verify the result with Sandbox files, commands, diffs, and a Judge.

When you evaluate a coding agent, checking a single reply is not enough. You usually need to give it a real project, let it read and write files, run commands, and commit changes, then check whether the output is correct. [NiceEval](https://niceeval.com/)'s current recommended shape is a normal `.eval.ts` file that explicitly prepares the Sandbox workspace, sends the task, and asserts on the result.

See [coding-agent-skill](https://github.com/CorrectRoadH/coding-agent-skill) (a separate repository) for a runnable reference.

## Recommended directory structure

```text theme={null}
evals/
├─ api-validation.eval.ts
├─ config-schema.eval.ts
└─ ponytail-safe-path.eval.ts

workspaces/
└─ ts-starter/
   ├─ package.json
   ├─ tsconfig.json
   └─ src/

skills/
└─ zod.md

experiments/
├─ baseline.ts
└─ with-skill.ts
```

`workspaces/` holds the starting project the Agent can see. `.eval.ts` controls when the workspace is uploaded, the task prompt, and what to verify. `experiments/` selects the Agent, the model, and whether to inject a Skill or plugin.

## Prepare the workspace

Inside the eval, upload the starting project to the Sandbox:

```ts theme={null}
import { defineEval } from "niceeval";

export default defineEval({
  description: "Add request-body validation to an Express route",
  async test(t) {
    await t.sandbox.uploadDirectory("../workspaces/ts-starter");

    await t.send("Implement POST /users in src/routes/users.ts.").then((turn) => turn.succeeded());

    t.sandbox.fileChanged("src/routes/users.ts");
  },
});
```

You can also use `t.sandbox.writeText()` to add seed files within a single eval — a good fit for small tasks or security probes.

## Write the task prompt

The prompt should read like a real work order. Describe the goal and constraints, but do not leak the verification answer.

```ts theme={null}
await t
  .send(
    `Implement environment validation in src/config/env.ts.
Requirements:
- Define EnvSchema with Zod
- Convert PORT to a number
- DATABASE_URL must be a valid URL
- JWT_SECRET must be at least 32 characters
- Export the env object and Env type`,
  )
  .then((turn) => turn.succeeded());
```

Leave implicit quality requirements, such as whether the code defends against path traversal or reuses existing tools, for the verification phase. That is how you find out whether a Skill or plugin actually helps the agent fill in missing context on its own.

## Verify files and code

```ts theme={null}
import { includes, excludes } from "niceeval/expect";

const code = await t.sandbox.readText("src/config/env.ts");

t.check(code, includes(/z\.object\s*\(/));
t.check(code, includes(/EnvSchema\.parse\s*\(\s*process\.env/));
t.check(code, excludes(/process\.env\.\w+\s*\?\?/));
t.sandbox.fileChanged("src/config/env.ts");
```

`t.sandbox.fileChanged()` is a scoped assertion: after the run ends, it uses the agent-attributed diff to determine whether the target file actually changed.

## Run project tests or probe scripts

For coding-agent tasks, real commands are usually more reliable than text matching:

```ts theme={null}
await t.sandbox.writeText(
  "_test_traversal.py",
  [
    "import sys, os",
    "sys.path.insert(0, '.')",
    "from uploads import safe_upload_path",
    "base = '/var/uploads'",
    "try:",
    "    p = safe_upload_path(base, '../../etc/passwd')",
    "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
    "except (ValueError, PermissionError, AssertionError):",
    "    pass",
    "print('ok')",
  ].join("\n"),
);

const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
t.check(result.stdout.trim(), includes(/ok/));
```

You can also run the project's own test, lint, or build scripts:

```ts theme={null}
import { commandSucceeded } from "niceeval/expect";

const result = await t.sandbox.runCommand("npm", ["test"]);
t.check(result, commandSucceeded());
```

## Use experiments for baselines

Do not put Agent names, plugin names, or URLs into positionals. The Experiment selects the Agent and pins the run configuration; the CLI's eval positionals only filter eval IDs.

```bash theme={null}
pnpm exec niceeval exp compare
pnpm exec niceeval exp compare api-validation
```

A typical A/B setup:

* `experiments/baseline.ts`: a plain agent.
* `experiments/with-skill.ts`: the same agent plus a setup hook that writes `CLAUDE.md`.
* Both groups use the same eval set, model, attempt count, and budget.

## When to use Sandbox Fixture

* The agent needs to modify real files.
* You need to run the project's tests, build, or lint.
* You need to compare diffs.
* You are evaluating a coding agent such as Claude Code or Codex.
* You are comparing whether a Skill, plugin, or Hook improves real tasks.

If you are only evaluating a direct agent application, a plain direct adapter is lighter.