AI-Native Eval for Agents.

Docs

Read the quickstart guide, then write an eval and run it across targets without building a bespoke harness.

Start GitHub Blog

your-project/

agents/web-agent.tsadapter
evals/
weather-tool.eval.ts
image-understanding.eval.ts
experiments/compare-models/
niceeval.config.tsconfig

$ niceeval

weatherpassed

fixtures/button91.7%

compare-models

gpt-5.4100%
deepseek-v4-pro60%

Connect

Connect your agent — or CC/Codex — via an adapter plus o11y.

Define

Write evals and experiments the way you'd write unit tests.

Evaluate

Evaluate in parallel.

Eval examples

eval chats, tool calls, and coding agents

Each card is a runnable defineEval file. Click a highlighted line to peek at replies and assertion notes.

multimodalMulti-turn image Q&A

passed

1import { defineEval } from "niceeval";
2
3export default defineEval({
4  description: "Evaluate an agent's multimodal ability across a multi-turn conversation",
5
6  async test(t) {
    const first = await t.sendFile("evals/sample.png", "What is in this image?");
    t.succeeded();
    first.usedNoTools();
    const second = await t.send("What color is the background?");
    second.messageIncludes(/blue|white|square/i);
    await t.send("What color is the shape in the middle?");
13
14    await t.group("follow-ups stay grounded in the image context", () => {
      t.messageIncludes(/white/i);
16    });
17
18    t.judge.autoevals
19      .closedQA("Does the assistant keep grounding every answer in the turn-one image, across all three turns, instead of making things up?")
      .gate(0.7);1/0.7
21  },
22});

tool callsWeather via tool calls

passed

1import { defineEval } from "niceeval";
2
3export default defineEval({
4  description: "Live weather must go through get_weather — no making it up",
5
6  async test(t) {
7    const turn = await t.send("What's the weather in Beijing today?");
8    turn.expectOk();
9
10    t.calledTool("get_weather", { input: { city: "Beijing" } });
11    t.notCalledTool("web_search");
12    t.eventOrder(["action.called", "action.result", "message"]);
13    t.messageIncludes(/°C|sunny|cloudy|rain/i);
14
15    await t.group("stays within budget", () => {
16      t.maxToolCalls(2);
17      t.maxCost(0.05);
18    });
19
20    t.judge.autoevals
21      .closedQA("Did the assistant give concrete weather data instead of hedging?")
22      .atLeast(0.7);
23  },
24});

sandboxCoding agent in a sandbox

passed

1import { defineEval } from "niceeval";
2import { includes } from "niceeval/expect";
3
4export default defineEval({
5  description: "Ask a coding agent to sum a CSV column, then verify the artifact",
6
7  async test(t) {
8    await t.sandbox.writeFiles({ "sales.csv": "id,amount\n1,100.5\n2,200.0\n3,50.5" });
9
10    const turn = await t.send("Write sum_sales.py that prints the total of the amount column.");
11    turn.expectOk();
12
13    t.fileChanged("sum_sales.py");
14
15    const run = await t.sandbox.runCommand("python3", ["sum_sales.py"]);
16    t.check(run.stdout.trim(), includes(/^351(\.0)?$/));
17
18    const code = (await t.sandbox.readSourceFiles({ extensions: ["py"] })).text();
19    t.judge.autoevals
20      .closedQA("Does the code use the csv stdlib instead of pandas, and stay concise?", { on: code })
21      .atLeast(0.7);
22  },
23});

Agents are users too

turn evals into a loop

The NiceEval CLI is designed for agents as much as for humans — not just an evaluation tool, but a framework that loops: build evals, run them, improve the agent system. Every output has an agent-readable face, so a coding agent drives the whole loop over bash.

terminalniceeval exp local --output agent --force

$ niceeval exp local --output agent --force
NICEEVAL RESULT failed
summary: 14 passed, 1 failed, 0 errored
failures:
  - @1k2m9qtr weather/brooklyn [local]
      gate: tool was never called
next:
  niceeval show @1k2m9qtr
  niceeval show @1k2m9qtr --execution