Skip to main content

NiceEval

Write evals like unit tests. Compare Agents through experiments.
You changed a prompt, switched models, wrote a new Skill for a coding Agent, or tuned an NPC in an LLM game. Now you want to know whether it actually improved. With NiceEval, you define success in TypeScript, then run, score, and compare repeatedly. When something fails, you can inspect what the Agent actually did. Everything runs on your own machine and in your CI. No account required.

Check numbers and ratios directly

Inside defineScoreEval’s test(t), score a precomputed ratio from 0 to 1 directly, without a custom fraction matcher:
measured preserves null as unavailable with a reason; it never treats it as zero. t.score requires a ratio between 0 and 1 and does not clamp it. Apply an explicit bound if your domain requires one. quantile uses linear interpolation by default; pass { method: "nearest-rank" } to select that method. Its statistics contains the sample count, probability, and method. Empty samples return no-samples; invalid samples return non-finite-sample. For statistics that only need displaying, an Adapter can save a Markdown or JSON attachment with ctx.attach, without creating an always-true check.

Start here

What is NiceEval

What you can evaluate, what an eval looks like, and how NiceEval differs from other tools.

Quickstart

Run your first eval and see the results in about ten minutes.

Runnable examples

Complete integration projects for AI SDK, Claude SDK, Codex SDK, Pi, and LangGraph.

Choose what to evaluate

Your own AI Agent

Write an Adapter to connect to your Agent’s API and assert replies, tool calls, and structured output.

Coding Agents and their extensions

Give an Agent a real repository in a Sandbox and use controlled experiments to measure Skills and Plugins.

Any AI application, including LLM games

Expose operations such as posting, replying, or NPC actions directly to evals, then check structured results and world state.

Common questions

No. Evals and result inspection run locally. You only need a Sandbox Provider such as Docker or E2B when the subject under test needs an isolated environment.
Yes. Evals and Adapters use TypeScript, but your application can use any language. An Adapter calls your existing API over HTTP, gRPC, or WebSocket. See Adapter.
Yes. Run the same niceeval exp command in CI. Its exit code indicates whether the evals passed. See CI integration.