Onboard a Project from Scratch with a Coding Agent

Step 1: Explore the project, then confirm with the user

1. What kind of Agent this is: read the README, package.json dependencies, routes, and agent loop code to determine what it is built with (AI SDK, LangGraph, OpenAI Agents SDK, Claude Agent SDK, a custom loop…) and what its core use case is (customer support? SQL? coding tasks?).
2. How the frontend talks to the Agent: HTTP, or gRPC / WebSocket? Is the protocol standard or custom: a standard protocol like AI SDK UI Message Stream or OpenAI Responses / Chat Completions, a passthrough of the SDK’s native event stream, or user-defined JSON/SSE frames? This directly decides whether the Adapter is built in (zero mapping) or a handwritten send (you write the event mapping).
3. How the real target runs: confirm the build or install command, start command, readiness check, port, dependent services, credential sources, and how to stop it. The Adapter only connects to the target. It does not start a substitute implementation inside send or the eval. Credentials go only to the run boundary that needs them, never into eval source, the workspace, or the run handoff.
4. Whether the backend already uses OTel: search for OTel SDK initialization, AI SDK telemetry, or instrumentation such as LangSmith / OpenLLMetry / OpenInference. If it exists, Tier 2 costs almost nothing.
5. Whether the user already has A/B tests or feature flags: if the application already has variant switches, the Experiment’s flags can pass straight through to them (a ready-made Tier 3 entry point).
6. What the Judge uses: semantic evaluation (t.judge(material, definition)) needs a Judge model separate from the Agent under test, over an OpenAI-compatible protocol. Ask which service the user has a key for and which Judge model they want. A missing key never passes silently: Judge assertions are recorded as unavailable. Without Judge configuration, write exact Matches first.
7. Whether the Agent itself must go into a Sandbox: if the system under test is a coding agent CLI, or a Skill/Plugin/Hook/MCP server written for a coding agent (which edits files and runs commands in an isolated workspace), it must use a Sandbox. It cannot be called with send directly like an HTTP service. The default suggestion is dockerSandbox({ source: { type: "image", image: "node:24-slim" } }). If the task or existing configuration already specifies a Provider, keep it. Only ask when the Provider, Docker availability, or remote execution needs would still change the plan. The Provider is declared on the eval or the Experiment; there is no CLI flag, project-level default, or auto-detection. See Choose a Sandbox Provider.

• Tier 1 (send only): no change to the application. The full assertion set (text, Judge, multi-turn, tools, HITL) is available at this tier.
• Tier 2 (send + OTel): the application also sends a copy of its OTel spans to NiceEval, and you view the call waterfall in niceeval view. With existing instrumentation (found in point 3), it takes zero changes.
• Tier 3 (invasive changes + flags): expose internal application variants as flags for feature A/B testing. Existing A/B switches (found in point 4) are a ready-made entry point.

Step 2: Configure the Judge

import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({
    model: "gpt-5.4-mini",                // Required: there is no built-in default model
    // When using an OpenAI-compatible gateway, add these two explicitly:
    // baseUrl: "https://api.deepseek.com/v1",
    // apiKeyEnv: "DEEPSEEK_API_KEY",
  }),
});

• A misconfiguration never pretends to pass. When the model or key cannot be read, or the request times out, the Judge assertion is recorded as unavailable with the reason. When the gateway rejects the request or returns the wrong shape, it is recorded as errored. Results never show a score that could not be obtained as a failure or a zero.
• Verify once after configuring. Declare a rubric and anchors with defineJudge, then call t.judge(material, definition).gate(0.8). After the run, check the score with pnpm exec niceeval show --run <run-id>. See Configure and Verify the Judge for the steps.
• Without a user present (autonomous onboarding), do not just skip the Judge. First check whether the environment already has a key for the chosen Provider. If it does, verify it. If not, write exact Matches first and state at the end that the Judge is not configured.
• The Judge model must be separate from the Agent under test, so a model does not grade its own output. Execution configuration is evaluated from the Experiment and the project config’s judgeProvider; for definitions and materials, see Judge.

Step 3: Write the Adapter, Experiment, and evals

1. Adapter (agents/*.ts or the directory the user’s project uses): implement send with defineAgent. Configuration goes through factory parameters; do not hard-code it or read process.env. The contract is in Adapter, API signatures in the Adapter reference, and event mapping in the events reference. Two common pitfalls: choose the endpoint or mode that enters the system’s core capability, not the one that is easiest to get running. For example, if the platform under test has both a “plain LLM chat” mode and a “connect to the database and execute” mode, connecting the former only evaluates an underlying model proxy, not the product itself. Declare evidenceCoverage honestly according to what you actually map. If you only map the final text, do not use completeEvidenceCoverage. The declaration affects assertion completeness, and over-claiming is worse than being conservative.
2. Experiment (experiments/*.ts): reference the Adapter above and declare model, flags, attempts, and so on. For a model comparison, write two experiment files, each with its own model. evals: (eval) => boolean decides which evals each Experiment runs. The file path decides the Experiment ID, and also which Experiments one command can run as a batch.
3. Evals (evals/*.eval.ts): first find out what this application does, then write an eval that follows its real function. Read its README, routes, tool definitions, or system prompt, find its core use case (ask a real support question of a support bot, give a SQL agent a real query task), and use that for the first eval’s input and assertions. Two kinds of input do not qualify: placeholder input unrelated to the application, such as “hello”, and meta questions to the system about itself, such as “what are you / what can you do”. Those are not tasks users give it. Still start from the smallest form: one input, t.succeeded() plus one content assertion on the expected answer. But the minimal form is only scaffolding to get things wired up, not the delivery standard. Before finishing, it must also meet two conditions:
   • Assertions must turn red when the system makes things up. Do not assert on words already in the input (ask “what is X” and then assert the answer contains “X”, and a system that repeats the question passes). Assert on substantive content unique to the expected answer: specific facts, structure (hasSections()), real links (includesUrl()), or a semantic verdict with t.judge(material, definition).
   • At least one negative case. Give the system an input it should not be able to answer (a nonexistent table, a topic retrieval cannot find), and assert that it clearly says it cannot find or do it, instead of inventing a plausible result. For an agent connected to real data or a retrieval source, this is the failure mode most worth testing first. See Write Evals for how to write them, Eval Kinds and Assertions for assertions and eval kinds, and the defineEval reference for signatures.

• Do not make an in-process direct call. Even if the agent runtime and the evals live in the same codebase, the Adapter must use HTTP (or the matching transport). Do not replace fetch with a direct import of the function under test. See “why not call it directly” in Connect Your Agent.
• The eval side does not manage the global process under test. Do not spawn the application or open ports inside individual eval tests or send(). When the target service is not yet running, start it following the target project’s own best practices (using its official CLI, documentation commands, containers, or inside Experiment before/after lifecycle hooks) and verify readiness. If startup fails, check the project’s output logs to resolve dependency or configuration issues, and retry until it is truly healthy, ensuring the evaluation connects and yields genuine scores.

Step 4: Run and verify

pnpm exec niceeval exp models --dry   # Confirm which evals each model configuration runs
pnpm exec niceeval exp models         # Run both model configurations by path
pnpm exec niceeval show               # View results in the terminal

• errored means the run itself did not complete and produced no result that can be scored. Check the target service, endpoint, credentials, dependencies, timeouts, and the Adapter’s error mapping, fix them, and run again.
• failed means the run completed but the business assertions were not met. First look at the assertions and evidence of the same Attempt. When the system under test really answered wrong, there is no need to turn it green.
• After a scored eval completes, still look at the score and each assertion. A completed or passed overview does not mean a full score.

pnpm exec niceeval show
pnpm exec niceeval show @<locator>
pnpm exec niceeval show @<locator> --execution
pnpm exec niceeval show @<locator> --source

Step 5: Wrap up and tell the user what you did

• The eval input is a core use case of the system under test, not a meta question about itself or a placeholder greeting
• Every content assertion turns red when the system repeats the question or makes things up (the asserted words are not already in the input)
• There is at least one negative case (an input it should not be able to answer, asserting that it clearly says it cannot)
• With a key, the Judge is configured and a Judge score was seen in niceeval show or niceeval view. Without a key, the summary says so
• The model / flags declared in the Experiment are actually consumed by the Adapter (no dead configuration nobody reads, and no invented model values the system does not have)
• The real target was started the project’s own way and confirmed ready; credentials went only to the run boundary that needs them and were not written into eval source, the workspace, or the handoff
• At least one Attempt completed without being errored; business failures are kept honestly, and the same locator was used to check that scores, assertions, and evidence are complete

Step 6: Ask the user whether to go deeper