> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# NiceEval

> Write evals like unit tests and compare Agents through experiments. Assert tool calls, replies, code changes, and business outcomes in TypeScript, run locally, and investigate every failure.

# NiceEval

**Write evals like unit tests. Compare Agents through experiments.**

You changed a prompt, switched models, wrote a new Skill for a coding Agent, or tuned an NPC in an LLM game. Now you want to know whether it actually improved. With NiceEval, you define success in TypeScript, then run, score, and compare repeatedly. When something fails, you can inspect what the Agent actually did.

Everything runs on your own machine and in your CI. No account required.

## Check numbers and ratios directly

Inside `defineScoreEval`'s `test(t)`, score a precomputed ratio from 0 to 1 directly, without a custom fraction matcher:

```ts theme={null}
import { atMost, isTrue, measured, quantile } from "niceeval/expect";

t.score(completionRate, { weight: 20 });
t.score(measured(optionalRate, { unavailable: "no-samples" }), { weight: 10 });
t.check(ready, isTrue()).gate();
t.check(measured(latencyMs, { unavailable: "no-actions" }), atMost(1200)).gate();
t.check(quantile(intervalsMs, 0.9), atMost(3000)).gate();
```

`measured` preserves null as unavailable with a reason; it never treats it as zero. `t.score` requires a ratio between 0 and 1 and does not clamp it. Apply an explicit bound if your domain requires one.
`quantile` uses linear interpolation by default; pass `{ method: "nearest-rank" }` to select that method. Its `statistics` contains the sample count, probability, and method. Empty samples return `no-samples`; invalid samples return `non-finite-sample`.
For statistics that only need displaying, an Adapter can save a Markdown or JSON attachment with `ctx.attach`, without creating an always-true check.

## Start here

> [card] [What is NiceEval](https://niceeval.com/docs/introduction.md)
>
> What you can evaluate, what an eval looks like, and how NiceEval differs from other tools.

> [card] [Quickstart](https://niceeval.com/docs/tutorials/quickstart.md)
>
> Run your first eval and see the results in about ten minutes.

> [card] [Runnable examples](https://niceeval.com/docs/examples.md)
>
> Complete integration projects for AI SDK, Claude SDK, Codex SDK, Pi, and LangGraph.

## Choose what to evaluate

> [card] [Your own AI Agent](https://niceeval.com/docs/tutorials/connect-your-agent.md)
>
> Write an Adapter to connect to your Agent's API and assert replies, tool calls, and structured output.

> [card] [Coding Agents and their extensions](https://niceeval.com/docs/examples/coding-agent-extensions.md)
>
> Give an Agent a real repository in a Sandbox and use controlled experiments to measure Skills and Plugins.

> [card] [Any AI application, including LLM games](https://niceeval.com/docs/tutorials/custom-application.md)
>
> Expose operations such as posting, replying, or NPC actions directly to evals, then check structured results and world state.

## Common questions

> [accordion] Do I need an account?
>
> No. Evals and result inspection run locally. You only need a Sandbox Provider such as Docker or E2B when the subject under test needs an isolated environment.

> [accordion] Can I connect an Agent written in another language?
>
> Yes. Evals and Adapters use TypeScript, but your application can use any language. An Adapter calls your existing API over HTTP, gRPC, or WebSocket. See [Adapter](https://niceeval.com/docs/explanation/adapter.md).

> [accordion] Can I use it in CI?
>
> Yes. Run the same `niceeval exp` command in CI. Its exit code indicates whether the evals passed. See [CI integration](https://niceeval.com/docs/tutorials/ci-integration.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.