# niceeval

## Docs

- [NiceEval: an eval tool for agents](https://niceeval.com/docs/introduction.md): NiceEval helps teams measure, evaluate, and improve AI in production: compare models, iterate on agents, catch regressions, and improve with real data.
- [Set up evals for your agent project](https://niceeval.com/docs/tutorials/quickstart.md): Install NiceEval, write three files, and run the first eval against your own application in about 10 minutes.
- [Put configuration and secrets where they belong](https://niceeval.com/docs/tutorials/configuration.md): Which layer of code holds attempts, timeout, concurrency, and judge — and which environment variables carry API keys and provider tokens.
- [Bring a Coding Agent into a Project](https://niceeval.com/docs/tutorials/agent-onboarding.md): A complete onboarding path for Coding Agents: explore the project, confirm the path with the user, configure a Judge, write an Adapter, Experiment, and evals, then run the first experiment.
- [Connect your Agent](https://niceeval.com/docs/tutorials/connect-your-agent.md): Write an Adapter, configure an Experiment, and run the first eval; then pass configuration and parameters from the Experiment to the Adapter and the app under test.
- [Write Send](https://niceeval.com/docs/tutorials/write-send.md): In seven steps, write the Adapter's send function: send a message, continue the session, record usage, map tool events, handle human-in-the-loop (HITL), hook up OTel traces, and pass through Experiment flags.
- [OTel Integration](https://niceeval.com/docs/tutorials/connect-otel.md): Send the OTel spans your app already emits to NiceEval, then view a per-turn call waterfall in niceeval view. Eval assertions still come from the events and usage send returns.
- [Sandbox Agents: Evaluate Claude Code, Codex, and bub](https://niceeval.com/docs/tutorials/sandbox-agent.md): Use NiceEval built-in Agents or a custom Adapter to run coding-agent CLIs in Docker or cloud Sandboxes.
- [Sandbox](https://niceeval.com/docs/tutorials/sandbox-providers.md): The official Docker, Vercel, and E2B Sandbox providers. Speed up evals with prebuilt snapshots.
- [Reuse Sandboxes: share setup across attempts](https://niceeval.com/docs/tutorials/sandbox-reuse.md): Use sandboxReuse to let attempts share a Sandbox. Reset returns only the workdir to its ledger point; persistent state outside it needs an explicit policy.
- [Authoring evals: single-turn, multi-turn, and dataset patterns](https://niceeval.com/docs/tutorials/authoring.md): Write evals with defineEval, covering single-turn conversations, multi-turn conversations, data-driven testing, Sandbox workspaces, and the eval lifecycle and fixtures.
- [Experiments and their lifecycle](https://niceeval.com/docs/tutorials/write-experiment.md): How to write a good experiment. Choose an agent, pass model and flags, configure attempts, budget, concurrency, and sandbox, and use setup / teardown to start and stop experiment-shared services.
- [Experiment matrix: compare agents and models with run matrices](https://niceeval.com/docs/tutorials/experiments.md): Use NiceEval experiments to run the same evals across multiple agents, models, and feature flags, then compare pass rate, cost, and latency.
- [Sandbox Fixture: evaluate coding agents with tasks](https://niceeval.com/docs/tutorials/fixtures.md): Use .eval.ts to give a coding agent an isolated workspace, send it a real task, and verify the result with Sandbox files, commands, diffs, and a Judge.
- [Data-driven testing (dataset fan-out): run one eval suite across many rows of data](https://niceeval.com/docs/tutorials/dataset-fanout.md): Export an array or a keyed record from a .eval.ts file to fan one eval out into many cases. Use loadYaml or loadJson to read external datasets, with stable IDs.
- [Evaluation kinds, assertions, and cost limits](https://niceeval.com/docs/tutorials/evaluation-kinds.md): Pass and points evals, value and scoped assertions, LLM-as-judge, and efficiency checks.
- [Rerun and Carry Results](https://niceeval.com/docs/tutorials/rerun-and-cache.md): See whether each slot executes, is adopted automatically, or is accepted by an operator, and how those relationships enter the Record.
- [Run NiceEval in CI](https://niceeval.com/docs/tutorials/ci-integration.md): Use the process exit status for a CI gate, retain the final receipt, and provide a JUnit file to the platform.
- [Connect the Current Run to External Systems](https://niceeval.com/docs/tutorials/reporters.md): Use a Reporter, JUnit, or process feedback to send information from one current Invocation to an external system.
- [View NiceEval Results](https://niceeval.com/docs/tutorials/viewing-results.md): Use show or view to select a Sample from published Records and inspect a fixed ReportExecution.
- [Publish a Static Report](https://niceeval.com/docs/tutorials/publish-report.md): Use view --out to export a self-contained report site that opens offline and is separate from its source Record.
- [Write a Custom Report](https://niceeval.com/docs/tutorials/custom-reports.md): Declare the facts each page needs, then execute once in memory to produce terminal, web, and static-export results.
- [Report Appearance and Safety Boundaries](https://niceeval.com/docs/tutorials/theming.md): Understand the current Report host's fixed appearance and why Reports cannot inject theme files, CSS, or browser code.
- [Let a Coding Agent Iterate from Results](https://niceeval.com/docs/tutorials/agent-feedback-loop.md): Let a Coding Agent read receipt run IDs, then use show or view to inspect results from a quiescent Record.
- [Debug Record and Report Problems](https://niceeval.com/docs/troubleshooting/debugging.md): Use visible errors, Sample states, and channel states to diagnose viewing, rerun, and export problems.
- [Keep a Sandbox alive to debug it](https://niceeval.com/docs/troubleshooting/debug-sandbox.md): Use --keep-sandbox to keep a failed attempt's Sandbox alive as a wakeable environment, use niceeval sandbox enter to go in and inspect it by hand, and use sandbox list / stop to see and clean up what's kept.
- [NiceEval's Evaluation and Reporting Path](https://niceeval.com/docs/explanation/overview.md): Learn how an Experiment, Record, Sample, and Report compose an eval that can be viewed and shared.
- [Evals in NiceEval: lifecycle, verdicts, and files](https://niceeval.com/docs/explanation/evals.md): An eval is a test case: a description and a test function, agent-neutral. Learn how evals are discovered, scheduled, scored, and reported.
- [Experiment: who to evaluate and how to run](https://niceeval.com/docs/explanation/experiment.md): An experiment is a commit-ready run configuration: which agent to evaluate, which model to use, which flags to enable, and how many times to run the same eval suite. Why evals and experiments are separate — late binding lets the same eval run against different subjects.
- [Adapter: Connect an Agent to NiceEval](https://niceeval.com/docs/explanation/adapter.md): The Adapter is the adapter you write. This page explains what the send function receives and returns, how system-under-test configuration is passed in, the three integration tiers, and where the capabilities on t come from.
- [Integration Tiers: Three Levels of Investment, Three Sets of Capabilities](https://niceeval.com/docs/explanation/tier.md): Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: Tier 1 is send only, Tier 2 adds OTel, Tier 3 modifies the application to expose experiment flags. What each tier buys and when to move up.
- [How NiceEval drives agents: send, sessions, and HITL](https://niceeval.com/docs/explanation/drive.md): t.send() and the Turn it returns, t.sendFile(), multi-turn conversations, t.newSession(), and HITL through t.respond() / t.respondAll().
- [Human-in-the-loop (HITL): pause, answer, continue](https://niceeval.com/docs/explanation/hitl.md): What human-in-the-loop is, what it looks like in Claude Code, Codex, and AI SDK apps, and how NiceEval turns it into an eval path: a waiting turn, an input.requested event describing what it's waiting for, and t.respond answering for the human so the same turn continues.
- [NiceEval assertions: values, scoped facts, tests, and cost](https://niceeval.com/docs/explanation/assert.md): NiceEval's assertion vocabulary — value assertions, scoped assertions, project-test assertions, and efficiency checks — plus gate vs soft severity and the rules that fold them into a verdict.
- [LLM-as-judge in NiceEval: assessing open-ended output](https://niceeval.com/docs/explanation/judge.md): How t.judge and turn.judge use a separate judge model to score factuality, closed-ended quality, and summarization faithfulness — plus model resolution and severity.
- [How the Runner Executes Evals](https://niceeval.com/docs/explanation/runner.md): Learn how the Runner creates Runs, chooses executed or adopted Attempts, and returns a receipt for later reading.
- [defineEval: declare, configure, and run NiceEval evals](https://niceeval.com/docs/reference/define-eval.md): Reference for defineEval options, the test context t, Turn return values, Sandbox helpers, and array and keyed-record dataset exports.
- [defineAgent and defineSandboxAgent: adapter reference](https://niceeval.com/docs/reference/define-agent.md): Reference for defineAgent and defineSandboxAgent: AgentContext, AgentSession, the Sandbox interface, and shared sandbox helpers.
- [Standard event stream reference](https://niceeval.com/docs/reference/events.md): The ten StreamEvent variants: what each one means, when adapters emit it, and which assertions consume it. Producing this stream is the adapter's core job.
- [Capabilities and evidence coverage](https://niceeval.com/docs/reference/capabilities.md): How construction, runtime behavior, and the required six-channel evidenceCoverage declaration determine which NiceEval assertions can be trusted.
- [defineConfig: project defaults](https://niceeval.com/docs/reference/define-config.md): defineConfig reference: judge, reporters, concurrency, timeout, and sandbox defaults.
- [niceeval/expect matchers and custom assertion reference](https://niceeval.com/docs/reference/expect.md): Reference for niceeval/expect: includes, equals, matches, similarity, satisfies, plus the shape assertions includesUrl and hasSections. Chain .gate() or .atLeast(0.7), or build custom matchers with makeAssertion.
- [Built-in agent capabilities reference](https://niceeval.com/docs/reference/builtin-agents.md): Which capabilities NiceEval's built-in claude-code, codex, and bub adapters implement, which assertions they map to, and known limits.
- [Official adapters overview](https://niceeval.com/docs/reference/official-adapters.md): What NiceEval's built-in Sandbox and non-Sandbox adapters are, how each authenticates, how to install MCP servers, Skills, and plugins on the Sandbox ones, and how to use an agent's official config file.
- [Read Result Data in a Report](https://niceeval.com/docs/reference/results-data.md): Declare result reads with AnalysisSample and NiceEval's official opaque projectors from niceeval/report.
- [Report API](https://niceeval.com/docs/reference/report-components.md): Public boundaries for ReportScope, FactRequirement, ReportPlan, ReportInput, single execution, and static export.
- [NiceEval CLI: Commands, Flags, and Exit Codes](https://niceeval.com/docs/reference/cli.md): Reference for NiceEval commands, flags, and exit codes, including current-project and historical Report selection.
- [NiceEval Examples](https://niceeval.com/docs/examples/index.md): Pick a runnable NiceEval example by subject under test: Agent Framework integrations, full AI agent evals, coding-agent extension benchmarks, and real projects.
- [Integrate AI SDK v7 with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/ai-sdk-v7.md): The complete before/after diff for connecting an AI SDK v7 chat app to NiceEval through its HTTP interface: zero app-side code changes.
- [Integrate Claude Agent SDK with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/claude-sdk.md): The complete before/after diff for connecting a Claude Agent SDK assistant backend to NiceEval: zero app-side code changes; everything is added on the eval side.
- [Integrate Codex SDK with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/codex-sdk.md): The complete before/after diff for connecting a Codex SDK coding-agent backend to NiceEval: zero app-side code changes.
- [Integrate pi-agent-core with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/pi-sdk.md): The complete before/after diff for connecting a pi-agent-core (@earendil-works) assistant backend to NiceEval: the app side only adds one devDependency.
- [Integrate LangGraph with NiceEval non-invasively](https://niceeval.com/docs/examples/integrations/langgraph.md): The complete before/after diff for connecting a pure Python LangGraph app with LangSmith OTel export to NiceEval.
- [Eval an AI Agent Application with a Hand-Written Adapter](https://niceeval.com/docs/examples/ai-agent-application.md): A runnable AI SDK v6 web agent eval project covering a hand-written adapter, tool calls, image understanding, multi-turn sessions, model comparison, and dual observability.
- [Eval Coding Agent Extensions](https://niceeval.com/docs/examples/coding-agent-extensions.md): Evaluate whether Skills, prompts, and plugin benchmarks improve a coding agent's task results, using a real workspace, a baseline experiment, and a custom report.