What is NiceEval

You revised a prompt, switched models, wrote a new Skill for Claude Code, or tuned an NPC’s prompt in a game. Did it get better? A few manual trials cannot reliably answer that. An Agent’s responses vary, and one lucky answer proves little. Fixing one case can quietly break another. NiceEval helps answer that question. It focuses on Agents and also evaluates any LLM-powered application. You define success in TypeScript, and NiceEval handles the rest:

Connect to the subject under test and send it tasks.
Repeat runs as needed and score each one.
Save conversations, tool calls, file changes, duration, and cost for every run.
Compare configurations in the terminal or browser, then inspect a specific failure to see what happened.

Everything runs on your own machine and in your CI. No account required.

What an eval looks like

This eval checks that a weather assistant actually calls get_weather when asked about the weather, instead of making up an answer.

// evals/weather-tool.eval.ts
import { defineEval, defineJudge } from "niceeval";
import { includes } from "niceeval/expect";

const groundedAnswer = defineJudge({
  name: "grounded-weather-answer",
  rubric: "Assess whether the answer gives specific weather information based on the question instead of refusing or answering vaguely.",
});

export default defineEval({
  description: "Call the weather tool and answer based on its result",
  judge: groundedAnswer,

  async test(t) {
    const turn = await t.send("What's the weather like in Beijing today?");
    turn.succeeded().label("Turn succeeded");

    t.calledTool("get_weather").label("Calls the weather tool");
    t.check(turn.message, includes("weather")).label("Describes the weather");

    const second = await t.send("And Shanghai tomorrow?");
    t.check(second.message, includes("Shanghai")).label("Second turn follows the context");

    t.judge({ question: turn.input, answer: turn.message }, groundedAnswer).gate(0.7)
      .label("Answer is grounded");
  },
});

Use deterministic rules for facts such as which tool was called or whether a reply contains a word. For open-ended quality, such as whether an answer is grounded, use a Judge model to score it. A score below 0.7 fails this check. The Agent and model to run belong in an Experiment, separate from the eval. You can therefore use the same evals to compare two models or two prompt versions.

What evals are for

Evals look like unit tests when you write them and work like experiments when you run them. The same set of evals can serve several purposes:

Prevent regressions: rerun after changing prompts, models, or dependencies and check whether the pass rate drops. See Rerun affected evals after changes.
Compare alternatives: run the same evals against two models, two prompts, or with and without a Skill, then compare each case. See Write an experiment and Compare runs.
Gate CI: make a PR fail when its evals fail. See CI integration.
Build coverage over time: turn failures from real use into new evals so every future change checks them.

What you can evaluate

Your own AI Agent

Use AI SDK, LangGraph, Pi, or your own Agent loop, in any language. If it exposes a callable API, you can connect it with an Adapter.

Coding Agents and their extensions

Run Claude Code or Codex in a Sandbox with a real repository and task, then score it with the project’s own tests. Measure the effect of Skills, Plugins, or memory.

Any AI application, including LLM games

If your application exposes operations such as posting, replying, or NPC actions, call them directly from evals and check structured results and world state.

For applications that are not conversational Agents, evals call the application’s own methods. This eval comes from the AI social application in examples/zh/llm-x:

// evals/social-journey.eval.ts — evaluates an AI-powered social application
export default x.defineEval({
  description: "AI characters respond to posts and replies, and the social world stays consistent",
  async test(t) {
    const initial = await t.visitDiscoveryPage();
    const post = await t.post({ intent: "Invite everyone to photograph the city at night", withImage: true });
    t.check(post, authoredPost(initial.viewerId)).label("Post belongs to the current player");

    const reply = await t.reply({ postId: post.id, intent: "Add that we meet at the riverside path entrance" });
    const responses = await t.waitForReplies(reply.id);
    t.check(responses.length, greaterThan(0)).label("AI characters respond");
    t.check(worldMaterial(await t.refreshFeed()), coherentSocialWorld()).label("Social relationships survive a refresh");
  },
});

There are two ways to connect the subject under test:

Connect directly to your application
Run in a Sandbox

NiceEval calls your Agent or application API through an Adapter. Docker is not required. Use this for conversational Agents and applications that expose business operations.

How NiceEval differs from DeepEval, LangFuse, and Braintrust

DeepEval is a mature Python evaluation framework with a broad library of metrics. NiceEval makes different choices:

Native TypeScript: evals, Adapters, and Experiments are typed TypeScript code.
Assert what happens along the way: an Agent may need several conversation turns, tool calls, file reads, and code changes to complete a task. The final answer is only part of it. NiceEval asserts these facts directly, without requiring a golden dataset first.
Score in a real environment: coding Agents run in a Sandbox and are scored using the project’s own tests and file changes, beyond the reply alone.
Built for controlled experiments: separate evals from Experiments to compare models, prompts, and extensions with the same cases.

Observability platforms such as LangFuse and Braintrust answer “What happened in production?” Evals answer “Is this behavior good enough?” NiceEval focuses on the latter and the local cycle of writing evals, running them, reviewing results, and improving the Agent. They can coexist: keep using those platforms for production traces and send NiceEval results to Braintrust.

Terms you will see often

Term	Meaning
Eval	A test in `evals/*.eval.ts` that defines what to send and what counts as success.
Experiment	A run configuration that chooses the Agent, model, and number of runs.
Adapter	Code that connects to the subject under test, sends requests, and translates replies into events NiceEval can read.
Sandbox	An isolated runtime environment. Coding Agents need it; directly connected applications do not.
Attempt	One execution of an eval. Running the same eval five times produces five Attempts.

Other terms are explained where they first appear.

The Quickstart walks you through running your first eval and seeing results in the terminal and browser in about ten minutes.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

What an eval looks like

What evals are for

What you can evaluate

Your own AI Agent

Coding Agents and their extensions

Any AI application, including LLM games

How NiceEval differs from DeepEval, LangFuse, and Braintrust

Terms you will see often

Next

​What an eval looks like

​What evals are for

​What you can evaluate

Your own AI Agent

Coding Agents and their extensions

Any AI application, including LLM games

​How NiceEval differs from DeepEval, LangFuse, and Braintrust

​Terms you will see often

​Next

What an eval looks like

What evals are for

What you can evaluate

How NiceEval differs from DeepEval, LangFuse, and Braintrust

Terms you will see often

Next