> ## Documentation Index
> Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring a Coding Agent into a Project

> A complete onboarding path for Coding Agents: explore the project, confirm the path with the user, configure a Judge, write an Adapter, Experiment, and evals, then run the first experiment.

This page is the execution path for a Coding Agent that brings NiceEval into a project. It assumes the project already has the `niceeval` dependency, `niceeval init` has run, and you reached this page from the packaged `INDEX.md`. Speak with the user in their language. Treat the packaged documentation as the authority for every API, field, and CLI behavior; do not invent details from training memory.

## Step 1: Explore the project, then confirm the path with the user

This step determines the rest of the workflow. **Inspect the code first, present the findings for the user to confirm, and ask only about facts you could not discover.** Do not open with a long questionnaire or make an unexamined assumption. Discover:

1. **What kind of Agent this is.** Read the README, `package.json` dependencies, routes, and Agent-loop code. Identify the stack—AI SDK, LangGraph, OpenAI Agents SDK, Claude Agent SDK, or a custom loop—and the core use case, such as support, SQL, or coding work.
2. **How the frontend and Agent communicate.** Is it HTTP, gRPC, or WebSocket? Is the protocol standard or custom: AI SDK UI Message Stream, OpenAI Responses or Chat Completions, an SDK event stream, or custom JSON/SSE frames? This decides whether a built-in Adapter needs no mapping or a custom `send` must map events.
3. **Whether the backend already has OTel.** Look for OTel SDK initialization, AI SDK telemetry, LangSmith, OpenLLMetry, OpenInference, or related instrumentation. An existing setup makes Tier 2 nearly free.
4. **Whether the user already has A/B tests or feature flags.** Existing variation switches are an immediate Tier 3 entry point because an Experiment can pass them through as `flags`.
5. **Which Judge to use.** Semantic scoring with `t.judge.autoevals.*` needs a Judge model separate from the Agent under test and an OpenAI-compatible `/chat/completions` endpoint. Ask which key and Judge model the user wants. There is no built-in default. A missing key does not silently pass: a Judge assertion becomes `unavailable`, and a non-optional assertion makes the Attempt `errored`. Use precise assertions only when the user deliberately omits a Judge or explicitly chains `.optional()`.
6. **Whether the Agent itself needs a Sandbox.** A coding-agent CLI, or a Skill, Plugin, Hook, or MCP server written for one, must modify files and run commands in an isolated workspace. It cannot use an HTTP-service-style `send` path.

   Recommend `dockerImageSandbox({ image: "node:24-slim" })` by default, but confirm whether the local machine or CI has Docker and whether the user needs Vercel Sandbox or another remote provider. Use Docker when the user has no objection. The provider is declared on the Eval or Experiment; there is no CLI flag, project-wide default, or auto-detection. See [Choose a Sandbox Provider](/docs/tutorials/sandbox-providers).

After exploring, introduce the [integration tiers](/docs/explanation/tier) and give a recommendation:

* **Tier 1 (send only):** No application change. The full assertion set—text, Judge, multi-turn, tool, and HITL—already works here.
* **Tier 2 (send + OTel):** The application sends spans to NiceEval as well, which enables the call waterfall in `niceeval view`. Existing instrumentation makes this nearly free.
* **Tier 3 (application changes + flags):** Expose internal variations as `flags` for feature A/B testing. Existing A/B switches are ready-made entry points.

**Recommend Tier 1 first, then Tier 2 by default.** If the exploration found existing OTel, say explicitly that Tier 2 only sends one more copy of the spans and costs almost nothing. Recommend Tier 3 only when the user wants a variation comparison.

Choose the right documentation for the discovered shape. Do not start writing an Adapter before reading it.

| System under test                                                      | Read                                                                                                                                                                    |
| ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| An app built with Vercel AI SDK and a `useChat` backend                | [Built-in Agent Reference](/docs/reference/builtin-agents): use `uiMessageStreamAgent` without writing event mapping.                                                        |
| A coding-agent CLI such as claude-code, codex, or bub that edits files | [Evaluate Agents in a Sandbox](/docs/tutorials/sandbox-agent): configure `sandbox`; recommend `dockerImageSandbox({ image: "node:24-slim" })` after confirming the provider. |
| A Skill, Plugin, Hook, or MCP server for Claude Code or Codex          | [Evaluate Coding Agent Extensions](/docs/examples/coding-agent-extensions): it also runs in a Sandbox and needs the same provider confirmation.                              |
| A custom Agent loop, LangGraph, OpenAI Agents SDK, or deployed Agent   | Start with [Connect Your Agent](/docs/tutorials/connect-your-agent), then read [Write send](/docs/tutorials/write-send) for the full custom `send` workflow.                      |
| A pure function with no independent service                            | First read [Connect Your Agent](/docs/tutorials/connect-your-agent) on why not to call it directly. Confirm this is genuinely the desired edge case before continuing.       |

## Step 2: Configure a Judge

Configure the Judge service identified in Step 1. Judges use an **OpenAI-compatible `/chat/completions`** endpoint in `niceeval.config.ts`:

```ts theme={null}
import { defineConfig } from "niceeval";

export default defineConfig({
  judge: {
    model: "gpt-5.4-mini",                // Required: there is no built-in default model
    // Add these only for a non-OpenAI compatible service, such as DeepSeek or a gateway:
    // baseUrl: "https://api.deepseek.com/v1",
    // apiKeyEnv: "DEEPSEEK_API_KEY",     // Reads the key from this variable; otherwise NICEEVAL_JUDGE_KEY
  },
});
```

**A compatible gateway needs an explicit `baseUrl`.** Supplying only a key sends the request to the official endpoint; the gateway credential then looks expired even though the endpoint was wrong.

Remember these points:

* **Misconfiguration cannot pretend to pass.** An unresolved model or key, rejected gateway authentication, or Judge timeout records the assertion as `unavailable` with a reason and makes the Attempt `errored`. A result that could not be judged is neither a pass nor the Agent's failure. Use `.optional()` only when one assertion may deliberately be absent, such as an experimental assertion on a development machine without a key.
* **Verify the setup once.** Run a lightweight eval against fixed text before calling the Agent: `t.judge.autoevals.closedQA("Does this text express success?", { on: "operation completed successfully" }).gate(0.8)`. Then use the receipt's Run ID with `niceeval show --run <runId> --page attempt-<attemptId>` and confirm that each assertion has a score.
* **Do not silently skip a Judge during autonomous onboarding.** First look for an available key, such as `NICEEVAL_JUDGE_KEY` or `DEEPSEEK_API_KEY`. If one exists, point `apiKeyEnv` at it and verify as above. Only when no key exists should the workflow use precise and shape assertions, and the final handoff must say that a Judge was not configured and why.
* The Judge model must be **separate from the Agent under test** so a model does not score itself. See [Judge](/docs/explanation/judge) for precedence and scoring shapes, and [defineConfig Reference](/docs/reference/define-config) for the complete `judge` fields.

## Step 3: Write the three pieces

After reading the documentation selected in Step 1, write these in order:

1. **Adapter** (`agents/*.ts` or the user's established directory). Use `defineAgent` to implement `send`; put static configuration in factory arguments rather than hard-coding it or reading `process.env`. See [Adapter](/docs/explanation/adapter), [defineAgent Reference](/docs/reference/define-agent), and [Event Reference](/docs/reference/events). Two common mistakes matter: connect the endpoint or mode that exercises the system's core capability, not merely the easiest endpoint; and declare `evidenceCoverage` truthfully. Mapping only final text does not justify `completeEvidenceCoverage`; a false claim is worse than a conservative one.
2. **Experiment** (`experiments/*.ts`). Reference the Adapter and declare `model`, `flags`, `attempts`, and related settings. Put model comparisons in separate experiment files, each with one pinned `model`. `evals: (eval) => boolean` decides which evals each Experiment runs. Paths provide IDs and batch execution; reports consume physical results and coverage facts from the current Sample.
3. **Eval** (`evals/*.eval.ts`). **Understand the application first, then write an eval around its real core use case.** Read its README, routes, tool definitions, or system prompt. For a support Agent, ask a real support question; for a SQL Agent, use a real query task. Avoid a generic “hello” and a meta-question such as “what can you do?” Both fail to exercise the system's work.

   Start with one input, `t.succeeded()`, and one assertion about the expected answer, but treat that shape as a connectivity scaffold rather than the delivery standard. Before you finish, meet both requirements:

   * **An assertion must fail when the system hallucinates.** Do not assert a word already present in the input. Asking “What is X?” then asserting that the answer contains “X” lets the Agent pass by repeating the prompt. Assert real expected facts, structure through `hasSections()`, a real URL through `includesUrl()`, or semantic correctness through `t.judge`.
   * **Add at least one negative case.** Give the system input it should not be able to answer, such as a nonexistent table or unavailable topic. Assert that it clearly says it cannot find or perform the task rather than inventing a plausible result. This is often the most valuable early failure shape for an Agent connected to real data or retrieval.

   See [Write Evals](/docs/tutorials/authoring) for authoring, [Evaluation Kinds](/docs/tutorials/evaluation-kinds) for assertions and question forms, and [defineEval Reference](/docs/reference/define-eval) for the signature.

Do not break these architectural rules while writing an Adapter:

* **Do not call the system directly in-process.** Even when the Agent runtime and eval share a codebase, the Adapter should use HTTP or the appropriate transport. Do not replace `fetch` with an import of the system under test. See [Connect Your Agent](/docs/tutorials/connect-your-agent) for the reason.
* **Do not manage the system-under-test process from the eval.** Do not spawn the application or open another port. The user starts it the normal way, such as `pnpm dev`. When the Adapter cannot connect, report a clear “start the application first” error instead of launching it.

## Step 4: Run and verify

```sh theme={null}
<package-manager> exec niceeval exp models
<package-manager> exec niceeval view --experiment models
```

Use [Coding Agent Feedback Loop](/docs/tutorials/agent-feedback-loop) to read the receipt and obtain Run IDs. Then inspect source, execution, timing, or diff pages with `niceeval show --run <runId> --page <planned-route>` as you observe, edit, and rerun. See [Viewing Results](/docs/tutorials/viewing-results) for the viewer workflow.

Diagnose an initial failure by its shape: a thrown `fetch` error means the application is not running or the URL is wrong. A failed `t.succeeded()` means the application returned a non-success status. If only a content assertion fails, the integration works; adjust the assertion or the application.

## Step 5: Finish by telling the user what changed

Before summarizing, run this finish check. **If any item fails, return to Step 3 and complete it; do not hide it in the final summary.**

* [ ] Eval inputs exercise the system's core use case, not a meta-question or placeholder greeting.
* [ ] Every content assertion fails when the system repeats the input or hallucinates; its asserted terms are not already in the input.
* [ ] At least one negative case makes an unsupported request and asserts an explicit inability to complete it.
* [ ] When a key is available, the Judge is configured and a Judge score has appeared in `niceeval view`. If no key exists, the summary explains that.
* [ ] The Experiment's declared `model` and `flags` are actually consumed by the Adapter. There is no dead configuration or invented model value.

After the first run works, summarize before proposing more work. State the connected system under test, the files created for the Adapter, Experiment, and evals, the `niceeval exp <experiment-path>` and explicitly selected `niceeval view` commands, and what the first run showed. Do not refactor existing user code or add unrelated abstractions unless asked.

## Step 6: Ask whether the user wants a deeper integration

After the summary, present optional next steps. For each one, explain what it enables, roughly how much code it changes, and what it buys. Let the user choose; do not continue on your own.

| Capability                                     | Change size                                                                                                                   | Benefit                                                                                                                         | Documentation                                                                          |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| Tool-call assertions such as `t.calledTool()`  | Adapter only: map application responses to the standard event stream, usually 10–30 lines                                     | Evals can verify that the Agent chose the right tool and arguments instead of only judging final text                           | [Write send](/docs/tutorials/write-send), [Event Reference](/docs/reference/events)              |
| Multi-turn conversations and session isolation | Adapter only: connect `ctx.session` with a typed slot or `id` plus `capture()`, from a few to tens of lines                   | Evals can cover multi-turn workflows and use `t.newSession()` to verify that sessions do not leak                               | [Drive a Conversation](/docs/explanation/drive), [Write send](/docs/tutorials/write-send)        |
| Human approval flows (HITL)                    | Adapter only: return `waiting` with `input.requested`, then continue after the answer, roughly 10–20 lines                    | Evals can cover approval and rejection behavior                                                                                 | [HITL](/docs/explanation/hitl)                                                              |
| Call waterfall (Tier 2)                        | With existing OTel, send one more copy of spans through a few configuration lines; otherwise add standard OTel initialization | `niceeval view` shows the timing and token timeline for each model and tool call in the application without changing assertions | [Configure OTel](/docs/tutorials/connect-otel)                                              |
| Feature A/B comparison (Tier 3)                | Change the application to expose variations as `flags`; the size depends on the app                                           | Compare prompt changes, tool sets, or feature switches from an Experiment                                                       | [Organize Experiments](/docs/tutorials/experiments), [Integration Tiers](/docs/explanation/tier) |

All of these are incremental Adapter or application changes. Existing evals do not need to change. See [Integration Tiers](/docs/explanation/tier) for what each investment buys and when it is worthwhile. If Step 1 found existing OTel, proactively recommend the waterfall option because it costs almost nothing.