> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# NiceEval: an eval tool for agents
> NiceEval helps teams measure, evaluate, and improve AI in production: compare models, iterate on agents, catch regressions, and improve with real data.
[NiceEval](https://niceeval.com/) is an agent eval tool that helps teams measure, evaluate, and improve AI in production. With NiceEval, teams can compare models, iterate on agents, catch regressions, and keep improving their AI applications using real user data.
NiceEval is local-first at its core: your evals run in your own environment. When your team needs to share eval results or track regressions, you can push a Report to platforms like BrainTrust, or export a custom report.
It can evaluate Plugins, Hooks, and Skills for Claude Code / Codex, and it can also evaluate your own AI agent application directly. Whether your agent is built on AI SDK, LangGraph, Pi, or a custom agent SDK, you can connect it through an Adapter into the same eval framework.
## Build an eval
```ts theme={null}
// evals/eval-tool-call.eval.ts
import { defineEval } from "niceeval";
export default defineEval({
description: "Verify the agent calls the weather tool correctly for a live weather question and answers from its result",
async test(t) {
const turn = await t.send("What's the weather in Beijing today?");
t.succeeded();
await t.group("calls get_weather with the right city", () => {
t.calledTool("get_weather", { input: { city: "Beijing" } });
t.messageIncludes(/°C|sunny|cloudy|rain/);
});
const second = await t.send("What about Shanghai tomorrow?");
second.messageIncludes("Shanghai");
t.judge.autoevals
.closedQA("Does the assistant answer from the tool's weather data instead of making up a temperature?")
.atLeast(0.7);
},
});
```
```sh theme={null}
pnpm exec niceeval exp local eval-tool-call # run only eval-tool-call under the local experiment
pnpm exec niceeval show # read current-project results in the terminal
pnpm exec niceeval view # browse the same Sample in the web viewer
```
## Why NiceEval when DeepEval, LangFuse, and BrainTrust already exist
NiceEval is an Agent-Native eval tool. The Dataset / golden pattern of building an Input and an Expected Output doesn't fit real agent evaluation — today's agents need to be checked at a finer grain: multi-turn conversations, multi-agent collaboration, tool calls, Skill loading. NiceEval is designed for exactly this kind of scenario: assertions land directly on tool calls, message content, structured output, and usage, instead of requiring you to hand-build a golden dataset first.
Tools like LangFuse and BrainTrust lean more toward tracing and monitoring, which is too heavy for the path of "write evals, run evals, inspect results." NiceEval is built from the start with a developer experience friendly to that path. The two aren't mutually exclusive either: NiceEval coexists with LangFuse and BrainTrust — use them for tracing, or upload eval results to either.
## What you can evaluate
Put Claude Code, Codex, or bub into a Sandbox, give them a task, then verify the result with real tests and file assertions.
Connect directly to your app's interface and assert on replies, tool calls, and structured outputs. Switch between different prompt versions with a Flag.
## Two integration modes
Fits coding agents like Codex and Claude Code that must edit code and run commands on a real file system.
```text theme={null}
evals/*.eval.ts
│
▼
┌────────────┐
│ niceeval │
└────────────┘
│
│ Agent adapter (official)
▼
┌───────────────────────────────┐
│ Docker Sandbox │
│ ┌───────────────────────┐ │
│ │ Codex / Claude Code / │ │
│ │ apps that need an │ │
│ │ isolated workspace │ │
│ └───────────────────────┘ │
└───────────────────────────────┘
```
Fits your own AI agent; no Docker required.
```text theme={null}
evals/*.eval.ts
│
▼
┌────────────┐
│ niceeval │
└────────────┘
│
│ Agent adapter (official, or your own)
▼
┌────────────────────────────┐
│ Your own AI agent app │
│ (AI SDK · LangGraph · Pi · │
│ custom agent loop, etc.) │
└────────────────────────────┘
```
## Core concepts at a glance
| Concept | In one line |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Eval | A test case: written in `evals/*.eval.ts`, describing what to check. |
| Experiment | A checked-in run configuration: which Adapter, which model, which flags. |
| Adapter | The layer that connects to the system under test: implement one `send`, and it translates the response into a standard event stream. |
| Sandbox | Only needed for coding agents that require an isolated workspace; connecting directly to a web agent doesn't need one. |
| Tier | Three levels of investment for integrating an Adapter: Tier 1 wires up `send` only, Tier 2 adds OTel for a call waterfall, Tier 3 makes invasive changes for feature A/B testing. |
See the full glossary in the [architecture overview](/docs/explanation/overview).
## Start from your scenario
Connect your agent by writing an Adapter that translates messages, tool calls, and structured outputs into the standard event stream.
Measure the effect of Skills, prompts, and Plugin benchmarks with a real workspace and comparison experiments.
Learn how to write evals, assertions, experiment configs, and reports.
## What to read next
[Quickstart](/docs/tutorials/quickstart) walks you through installing [NiceEval](https://niceeval.com/), writing an eval, and getting an eval report.