Quickstart: Run Your First Eval

This page walks you through running your first eval against your own application. You write three files:

Adapter: tells NiceEval how to call your application.
Experiment: says which Adapter to run against and with which model.
Eval: says which message to send and what counts as correct.

There are two ways to do it. Having a Coding Agent write them is fastest; writing them yourself shows more clearly what each file is for.

Let a Coding Agent integrate it for you

Send the instruction to your Agent

Open the Coding Agent you use in the current project, such as Claude Code or Codex, and send this message:

Read https://niceeval.com/INIT.md, install and integrate niceeval into the current repository, and get the first eval running end to end. When done, list the Experiment ID you created, the run command, and the command to view results.

The Agent first explores your project, confirms which interface to test, then writes the three files and runs them once.

Run it again yourself

Run with the Experiment ID the Agent listed at the end. If it did not list one, first see which Experiments exist with pnpm exec niceeval exp list:

pnpm exec niceeval exp <experiment-id>

See the results

pnpm exec niceeval show

The terminal lists each eval’s result, with failures first. For how to read the results, see Read the results below.

Integrate it yourself

Prepare the application under test

First start your application and confirm it has an interface you can call. The example below assumes:

the interface is POST http://localhost:3000/chat;
the request body is { "message": "...", "model": "..." };
the response body is { "reply": "..." }.

If your interface differs, that is fine; you only change the address, the request body, and the line that reads the reply in the Adapter. If you do not have an application you can start yet, pick a runnable project from Examples first.

Install

pnpm add -D niceeval
pnpm exec niceeval init

init creates niceeval.config.ts. It also writes a section into AGENTS.md or CLAUDE.md so your Coding Agent reads the docs that match the installed version.

Write the Adapter

The Adapter does one thing: send the eval’s messages to your application, then hand the reply back to NiceEval.

// agents/my-agent.ts
import { completeEvidenceCoverage, defineAgent } from "niceeval/adapter";

export default defineAgent({
  name: "my-agent",
  evidenceCoverage: completeEvidenceCoverage,
  async send(input, ctx) {
    // Replace with your own application's address
    const r = await fetch("http://localhost:3000/chat", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ message: input.text, model: ctx.model }), // ctx.model comes from the Experiment
      signal: ctx.signal,
    });
    const body = await r.json();
    return {
      status: r.ok ? "completed" : "failed",
      events: [{ type: "message", role: "assistant", text: body.reply }],
    };
  },
});

events is how NiceEval understands the reply. Here it hands back only one assistant message; later you can add events for tool calls, usage, and more.

Write the Experiment

// experiments/my-agent.ts
import { defineExperiment } from "niceeval";
import myAgent from "../agents/my-agent.ts";

export default defineExperiment({
  description: "my-agent baseline",
  agent: myAgent,
  model: "gpt-4o",
  attempts: 1,
});

The file name is the Experiment ID, here my-agent. To compare another model later, copy this file and change model; the evals stay the same.

Write the eval

// evals/refund-policy.eval.ts
import { defineEval } from "niceeval";
import { pattern } from "niceeval/expect";

export default defineEval({
  description: "Answers the refund policy correctly",
  async test(t) {
    await t.send("What is your refund policy?");
    t.succeeded();
    t.check(t.reply, pattern(/30\s*days|one month/i));
  },
});

The model’s wording changes every time, so this uses a regex to match “30 days” or “one month” instead of requiring an exact match.

Run

pnpm exec niceeval exp my-agent

The terminal shows progress while it runs. When it finishes, you see output like this:

╭─ PASSED ─────────────────────────────────────────────────────── 12s ─╮
│ 1 passed · 0 failed · 0 errored  (0 reused)                          │
╰──────────────────────────────────────────────────────────────────────╯

╭─ RESULTS ───────────────────────────────────── 1 run configuration ─╮
│ my-agent                                                             │
│   refund-policy  1/1 passed                                          │
╰──────────────────────────────────────────────────────────────────────╯

╭─ NEXT ───────────────────────────────────────────────────────────────╮
│ my-agent                                                             │
│   show: niceeval show --run 8f3d6f62-1d34-4cf3-99c7-84ba3c483706     │
│   view: niceeval view --run 8f3d6f62-1d34-4cf3-99c7-84ba3c483706     │
╰──────────────────────────────────────────────────────────────────────╯

Your first eval is now running.

Read the results

The title tells you the conclusion of this run:

Title	Meaning	What to do first
`PASSED`	All assertions were met.	Keep adding evals.
`FAILED`	The application replied normally, but at least one assertion was not met.	See what the Agent actually replied, and decide whether the application is wrong or the assertion is too strict.
`ERRORED`	No result could be obtained, for example the application was not running, the interface returned an error, or the Adapter threw an exception.	Look at the `error:` line in the `FAILURES` panel, which usually shows the cause directly; for more detail, run the `details:` command below it.

Each failed Attempt has a line details: niceeval show @<locator> below it. Copy it to see that run’s conversation and assertions in the terminal. To see the whole run, copy the show: command in the NEXT panel; to browse the conversation and durations in the browser, copy the view: line. For more ways to view results, see View Run Results.

Next steps

This minimal Adapter only supports single-turn conversations: the second turn does not carry the first turn’s history, and tool calls are not visible. Choose based on what you need next:

To test multi-turn conversations, tool calls, or usage: extend the Adapter following Write send; existing evals stay unchanged.
To write more evals that tell results apart better: read Write Evals.
To compare two models or two prompt versions: read Organize Experiments.
To run automatically on PRs: read CI Integration.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Quickstart: Run Your First Eval

Let a Coding Agent integrate it for you

Integrate it yourself

Prepare the application under test

Install

Write the Adapter

Write the Experiment

Write the eval

Run

Read the results

Next steps

​Let a Coding Agent integrate it for you

​Integrate it yourself

​Prepare the application under test

​Install

​Write the Adapter

​Write the Experiment

​Write the eval

​Run

​Read the results

​Next steps

Let a Coding Agent integrate it for you

Integrate it yourself

Prepare the application under test

Install

Write the Adapter

Write the Experiment

Write the eval

Run

Read the results

Next steps