# niceeval ## Tutorials ### Get Started - [NiceEval](https://niceeval.com/docs/index.md): Write evals like unit tests and compare Agents through experiments. Assert tool calls, replies, code changes, and business outcomes in TypeScript, run locally, and investigate every failure. - [What is NiceEval](https://niceeval.com/docs/introduction.md): Write TypeScript evals for AI Agents and applications: assert tool calls, replies, code changes, and business outcomes, compare configurations, and investigate failures. - [Quickstart: Run Your First Eval](https://niceeval.com/docs/tutorials/quickstart.md): Install NiceEval, write an Adapter, an Experiment, and an eval, get your first eval running against your own application in about ten minutes, and see the results in the terminal and the browser. - [Onboard a Project from Scratch with a Coding Agent](https://niceeval.com/docs/tutorials/agent-onboarding.md): The full onboarding flow for a Coding Agent: explore the project, confirm the path with the user, configure the Judge, write the Adapter, Experiment, and evals, and get the first experiment running. - [Put Configuration and Secrets Where They Belong](https://niceeval.com/docs/tutorials/configuration.md): Which layer of code holds values such as attempts, timeouts, concurrency, and the Judge, and which environment variables hold API keys and provider tokens. ### Connect Your Application - [Connect Your Agent: Choose an Integration and Pass Configuration](https://niceeval.com/docs/tutorials/connect-your-agent.md): Choose a built-in adapter or a handwritten Adapter for the system under test, put the address and auth in the Adapter factory, pass model and flags to the application through ctx, and add tool calls, multi-turn, and tracing as needed. - [Write Send](https://niceeval.com/docs/tutorials/write-send.md): Write an Adapter's send function in seven steps: send messages, continue sessions, record usage, map tool events, handle human-in-the-loop (HITL), connect OTel traces, and pass through Experiment flags. - [Evaluate a Custom Application: Business Operations Such as Posting and Image Generation](https://niceeval.com/docs/tutorials/custom-application.md): When the application under test is not a conversational Agent, use defineAdapter to expose its business methods directly to evals, and check structured results and business operations. - [Connect OTel: See a Per-Turn Call Waterfall in the Browser](https://niceeval.com/docs/tutorials/connect-otel.md): When your application already sends OTel traces, have it send a copy of its spans to NiceEval as well, and see each turn's model calls, tool executions, and durations in niceeval view. Assertions are unaffected. ### Write Evals - [Write Evals: Single-Turn, Multi-Turn, and Datasets](https://niceeval.com/docs/tutorials/authoring.md): Write evals with defineEval: send messages, check replies and tool calls, run multi-turn conversations, generate many evals from one dataset, and prepare a workspace and clean up external resources for Coding Agents. - [Score by Completion: Pass/Fail and Scored Evals](https://niceeval.com/docs/tutorials/evaluation-kinds.md): When a task has only two outcomes, done or not done, use defineEval and read the pass rate; when it has partial progress, use defineScoreEval and accumulate points for completed steps. - [Evaluating RAG and Knowledge Base Agents](https://niceeval.com/docs/tutorials/rag-application.md): Evaluate Retrieval-Augmented Generation (RAG) and knowledge base agents: assert retrieval tool calls, check citation formats, verify grounding with a Judge, and test refusal on unknown queries. - [Data-Driven Tests: Generate Many Evals from One Dataset](https://niceeval.com/docs/tutorials/dataset-fanout.md): When dozens of evals share the same logic and differ only in input, export an array or object from one .eval.ts and generate many evals from one set of logic. You can also read an external dataset with loadYaml or loadJson and give each case a stable ID. - [Configure the Judge and Confirm It Can Really Score](https://niceeval.com/docs/tutorials/verify-judge.md): Choose the Provider, model, and API key for the Judge model, then run one real scoring pass with a minimal eval, and confirm the configuration is correct before using it at scale. - [NiceEval Taste: Make Eval Criteria Clear at a Glance](https://niceeval.com/docs/tutorials/niceeval-taste.md): Let a colleague read an eval and explain what it tests, what passes, why the evidence is sufficient, and how it scores. Use existing facts, simple conditions, and ordinary arithmetic. - [Eval Criteria for Custom Applications: Business Facts, Domain Matches, and Plain Arithmetic](https://niceeval.com/docs/tutorials/niceeval-taste-application.md): When an application supplies support records or game state, let its Adapter provide facts, compose small conditions with domain Matches, calculate scores in the eval, and preserve unknown states. ### Evaluate Coding Agents - [Sandbox Agent: Evaluate Claude Code, Codex, and bub](https://niceeval.com/docs/tutorials/sandbox-agent.md): Put Coding Agents such as Claude Code, Codex, and bub into a Docker or cloud Sandbox, give them a real repository and task, then score with file changes and project tests. - [Prepare Tasks for a Coding Agent: Workspace, Prompt, and Acceptance](https://niceeval.com/docs/tutorials/fixtures.md): Put a real project into the Sandbox in an eval, give the Coding Agent a task, then check whether it did it right with file contents, command results, diffs, and the Judge. - [Score with Hidden Tests and Rerun Automatically When Tests Change](https://niceeval.com/docs/tutorials/criteria-files.md): Use loadText to read scoring files such as hidden tests and test scripts. NiceEval remembers the file content, so when a test changes, the evals that use it rerun automatically next time. - [Install the CLI for a Custom Sandbox Agent](https://niceeval.com/docs/tutorials/install-custom-sandbox-agent.md): Use ensure and installer to check, install, and recheck the exact version of a custom coding-agent CLI. - [Choose and Configure a Sandbox Provider](https://niceeval.com/docs/tutorials/sandbox-providers.md): Choose, prepare, and cache environments with Docker, Incus, Vercel, E2B, or OpenSandbox. - [Prebuilt Sandbox Environments: Start Each Eval Faster](https://niceeval.com/docs/tutorials/prebuilt-sandbox.md): Bake system packages, Agent CLIs, and large caches into a Docker image, Vercel snapshot, or E2B template derived from an official baseline; carry run-time caches between Attempts with checkpoints. ### Experiments and Comparisons - [Organize Experiments: Compare Models, Prompts, and Agents](https://niceeval.com/docs/tutorials/write-experiment.md): Use defineExperiment to fix the Agent under test, model, flags, and number of runs. Write two Experiments that differ in only one variable over the same evals to compare two models or two prompt versions. - [Confirm What Runs and How Many Times Before Running](https://niceeval.com/docs/tutorials/select-and-preview.md): List which Experiments exist, narrow the scope with ID prefixes and tags, then use --dry to see which evals this run executes and how many times, and confirm before spending money. - [Start Shared Experiment Services](https://niceeval.com/docs/tutorials/experiment-services.md): Use an Experiment's before / after to start and stop tunnels, mock servers, or leases shared by the whole experiment, and hand run-time addresses to the Agent and Sandbox without invalidating existing results. - [Control the Cost of a Run](https://niceeval.com/docs/tutorials/control-run-cost.md): One eval run can cost tens of dollars. Depending on what you want to control, use attempts, early exit, budget, concurrency, and result reuse. ### Migration - [Upgrade to 0.20](https://niceeval.com/docs/tutorials/upgrade-0-20.md): Update configuration and fixed-version protocols. - [Upgrade to 0.19](https://niceeval.com/docs/tutorials/upgrade-0-19.md): Rename model purposes and replace removed configuration fields. - [Upgrade to 0.18](https://niceeval.com/docs/tutorials/upgrade-0-18.md): Shared model purposes, native numeric material, and public entry point changes ### Review Results and Iterate - [View Run Results: Find the Cause of Failure from the Overview](https://niceeval.com/docs/tutorials/viewing-results.md): After evals run, use niceeval show in the terminal to drill down from the overview to the failed run and see its conversation, durations, usage, and file changes; browse with view in the browser; and have scripts and CI read JSON with show --json. - [Compare Two Runs: Is the Baseline or the Candidate Better](https://niceeval.com/docs/tutorials/compare-runs.md): Pin the Run IDs of two runs, review them side by side in the browser, or use runs.compare to give scripts and CI a per-eval paired comparison. - [Rerun Only the Evals Affected by a Change](https://niceeval.com/docs/tutorials/rerun-and-cache.md): After changing a prompt, an eval, or the model, use --dry to see which results will be reused and which will rerun, then control the scope with --rerun and accept. - [Confirm Old Results Still Hold and Skip a Rerun](https://niceeval.com/docs/tutorials/accept-results.md): When configuration changes and NiceEval will not reuse old results, but you have confirmed they are unaffected, accept them explicitly with niceeval accept and save a paid rerun. - [Let a Coding Agent Iterate on Results](https://niceeval.com/docs/tutorials/agent-feedback-loop.md): Let a Coding Agent run evals, read results, decide whether to change the application or the assertions, and verify again, without you copying terminal output back and forth. ### Concurrency and Reuse - [Tune Concurrency: Throughput, Shared State, and a Fixed Execution Order](https://niceeval.com/docs/tutorials/concurrency.md): When a batch of evals runs too slowly, first find where the time goes, then tune global concurrency and the Experiment's maxConcurrency. When Attempts need to share state, have them run serially in a fixed order. - [Reuse Sandboxes Across a Batch of Evals](https://niceeval.com/docs/tutorials/sandbox-reuse.md): When creating a new Sandbox for every Attempt is too slow, use sandboxReuse to let a batch of Attempts share the same Sandbox in turn. Learn what it saves, what it leaves behind, and how to write preparation code that replays safely. - [Use Eval Groups to Let Related Evals Share a Sandbox](https://niceeval.com/docs/tutorials/eval-groups.md): When several groups of evals each need a heavy toolchain, group them with defineEvalGroup: each group shares one Sandbox and runs in sequence, while different groups run in parallel as usual. - [Reuse Run Conditions with Plugins](https://niceeval.com/docs/tutorials/plugins.md): Bundle Agent extensions, Sandbox steps, and host lifecycle hooks into a reusable experiment condition. ### Infrastructure and CI - [Run Evals Automatically in CI to Block Regressions](https://niceeval.com/docs/tutorials/ci-integration.md): Run the same niceeval exp on every PR and turn CI red when evals fail. Keep this run's Run ID for later steps to read, and hand a JUnit file to your CI platform for display. - [Report Run Results to External Systems Such as Braintrust](https://niceeval.com/docs/tutorials/reporters.md): When your team already uses Braintrust or its own dashboard, use a Reporter to send results there during the run; use JUnit for CI platforms; and use the Run ID in the receipt for later scripts to read results. - [Cut Losses When Credentials Expire or a Service Goes Down](https://niceeval.com/docs/tutorials/handle-execution-failures.md): Learn which failures NiceEval retries automatically; when you find faults that are bound to fail, such as expired credentials or a disconnected shared service, stop the remaining Attempts and, after fixing them, rerun only what did not finish. - [Check How the Sandbox Will Be Prepared Before Running](https://niceeval.com/docs/tutorials/debug-lifecycle-plan.md): Use niceeval debug to list the order in which one eval is prepared, installed, executed, and cleaned up in the Sandbox, with each step's actual parameters, without creating any resources. - [Run Docker Inside a Sandbox](https://niceeval.com/docs/tutorials/docker-in-docker.md): Give each Attempt an Incus VM with a dedicated kernel, Docker daemon, and data disk, and check environment and cache capacity. - [Prepare an Incus Eval Environment on NixOS](https://niceeval.com/docs/tutorials/nixos-managed-dind.md): Understand the boundary between Incus host provisioning and NiceEval, and check reference or development environments with doctor. ### Troubleshooting - [Debug failed eval results](https://niceeval.com/docs/troubleshooting/debugging.md): When an eval fails or errors, start from a specific Run or Attempt and use show to drill into errors, conversations, timing, and changes, to find whether the problem is in the application, the Adapter, the environment, or the assertion. - [Keep Sandbox state to debug problems](https://niceeval.com/docs/troubleshooting/debug-sandbox.md): Use --keep-sandbox or --pause-on-failure to keep or pause a failed Attempt's Sandbox, investigate directly via native Docker or Instabox CLIs, and clean up with native provider commands. - [Recover after a forced kill](https://niceeval.com/docs/troubleshooting/recover-after-kill.md): After kill -9, a CI timeout, or a power loss interrupts the process, keep the results that already finished, end the Run that lost its process, and reclaim leftover Sandboxes. ## Explanation ### Core Model - [NiceEval's overall model: the parts of an evaluation](https://niceeval.com/docs/explanation/overview.md): What eval cases, Experiments, Adapters, Sandboxes, the Runner, and Record each handle, how they connect into one run, and why the same eval runs more than once. - [Eval cases in NiceEval: lifecycle, verdicts, and files](https://niceeval.com/docs/explanation/evals.md): An eval case says what to send and what counts as correct, without binding to a system under test. Learn how eval cases are discovered, scheduled, scored, and reported. - [Experiment: what to evaluate and how to run it](https://niceeval.com/docs/explanation/experiment.md): An Experiment is a check-in-able run configuration: which Agent and model the same set of eval cases runs against, which flags are on, and how many times to run. Eval cases and Experiments are kept separate so the same eval cases can run against different systems. - [Adapter: connect an Agent to NiceEval](https://niceeval.com/docs/explanation/adapter.md): An Adapter is the integration layer you write. This page explains what the send function receives and returns, how to pass the configuration of the system under test, the three integration tiers, and where the capabilities on t come from. - [Integration tiers: three levels of investment, three sets of capabilities](https://niceeval.com/docs/explanation/tier.md): Integration has three tiers based on where the Adapter connects and what additional observability data it receives: Tier 1 connects only send, Tier 2 adds OTel, and Tier 3 changes the application to expose Experiment flags. Learn what each tier provides and when to move up. - [NiceEval's post-run data flow](https://niceeval.com/docs/explanation/results-flow.md): Understand where run results are stored, and what each of the three read entry points, show and view, is responsible for. - [Record](https://niceeval.com/docs/explanation/record.md): Learn what the project results file .niceeval/record.sqlite stores, how to copy it for someone else, and what is validated when it is read. ### Interaction and Scoring - [How to drive an Agent in NiceEval: send, sessions, and HITL](https://niceeval.com/docs/explanation/drive.md): t.send() and the Turn it returns, t.sendFile(), multi-turn conversations, t.newSession(), and human-in-the-loop (HITL) handled through respond(). - [Human-in-the-loop (HITL): pause a turn, answer, and continue](https://niceeval.com/docs/explanation/hitl.md): What human-in-the-loop is, how it appears in Claude Code, Codex, and AI SDK applications, and how NiceEval turns it into an eval path: a waiting turn pauses, input.requested says what it needs, and t.respond answers for the human so the same turn can continue. - [NiceEval Assertions and Matches](https://niceeval.com/docs/explanation/assert.md): A Match describes how to compare a value. Calling a check method registers one Assertion immediately, and the methods chained after it only configure that one. - [Judge in NiceEval: assess open-ended output](https://niceeval.com/docs/explanation/judge.md): Use a rubric, anchors, and explicit material to assess open-ended output, adding images when needed. ### Execution Model - [How a run produces readable results](https://niceeval.com/docs/explanation/runner.md): Learn how the Runner creates a Run, publishes Attempts one by one, and why read commands only see published results. ## Reference ### API - [defineEval: Declare, Configure, and Run NiceEval Evals](https://niceeval.com/docs/reference/define-eval.md): defineEval reference: options, the test context t, the Turn return value, Sandbox helpers, and array and keyed-record test set exports. - [Define What You Evaluate: Adapter Reference](https://niceeval.com/docs/reference/define-agent.md): Context, assertion methods, and resource lifecycle for custom Adapters and Agent integrations. - [Standard Event Stream Reference](https://niceeval.com/docs/reference/events.md): The ten StreamEvent variants: each one's fields, when it is emitted, and which assertions read it. Producing this stream is an Adapter's main job. - [Capabilities and Evidence Coverage Reference](https://niceeval.com/docs/reference/capabilities.md): How an Agent's construction, runtime behavior, and required six-channel evidenceCoverage declaration together determine what NiceEval assertions can trust. - [defineExperiment: Experiment Configuration](https://niceeval.com/docs/reference/define-experiment.md): Reference for defineExperiment fields, types, and default behavior. - [defineConfig: Project Defaults](https://niceeval.com/docs/reference/define-config.md): defineConfig reference: Judge Runtime, Reporters, concurrency, timeout, and Sandbox defaults. - [niceeval/expect Match Reference](https://niceeval.com/docs/reference/expect.md): Pure Match factories from niceeval/expect: compose text, structure, and conditions, then register an Assertion with t.check. ### Adapters - [Built-in Agent Capabilities Reference](https://niceeval.com/docs/reference/builtin-agents.md): What NiceEval's built-in claude-code, codex, and bub Adapters can do, which assertions they support, and their known limits. - [Official Adapter Overview](https://niceeval.com/docs/reference/official-adapters.md): NiceEval's built-in Sandbox and non-Sandbox Adapters, their authentication, how to install MCP servers, Skills, and plugins for Sandbox Adapters, and how to use an Agent's official configuration file. ### CLI - [CLI](https://niceeval.com/docs/reference/cli.md): Read run records with show and view. ## Examples ### Choose an Example - [NiceEval Examples](https://niceeval.com/docs/examples/index.md): Pick a runnable NiceEval example by subject under test: Agent Framework integrations, full AI agent evals, coding-agent extension benchmarks, and real projects. ### Connect an Agent Framework - [Integrate AI SDK v7 with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/ai-sdk-v7.md): The complete before/after diff for connecting an AI SDK v7 chat app to NiceEval through its HTTP interface: zero app-side code changes. - [Integrate Claude Agent SDK with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/claude-sdk.md): The complete before/after diff for connecting a Claude Agent SDK assistant backend to NiceEval: zero app-side code changes; everything is added on the eval side. - [Integrate Codex SDK with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/codex-sdk.md): The complete before/after diff for connecting a Codex SDK coding-agent backend to NiceEval: zero app-side code changes. - [Integrate pi-agent-core with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/pi-sdk.md): The complete before/after diff for connecting a pi-agent-core (@earendil-works) assistant backend to NiceEval: the app side only adds one devDependency. - [Integrate LangGraph with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/langgraph.md): The complete before/after diff for connecting a pure Python LangGraph app with LangSmith OTel export to NiceEval. ### Complete Eval Projects - [Eval an AI Agent Application with a Hand-Written Adapter](https://niceeval.com/docs/examples/ai-agent-application.md): A runnable AI SDK v6 web agent eval project covering a hand-written adapter, tool calls, image understanding, multi-turn sessions, model comparison, and dual observability. - [Eval Coding Agent Extensions](https://niceeval.com/docs/examples/coding-agent-extensions.md): Evaluate whether Skills, prompts, and plugin benchmarks improve a coding agent's task results with a real workspace and baseline Experiment. > The links below point to documentation indexes. Follow each `/_llms/` index recursively until you reach documentation pages. ## Indexes - [Simplified Chinese (77 pages)](https://niceeval.com/docs/_llms/zh-hans.txt): Tutorials, Explanation, Reference, Examples. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.