Bring a Coding Agent into a Project

​

Step 1: Explore the project, then confirm the path with the user

1. What kind of Agent this is. Read the README, package.json dependencies, routes, and Agent-loop code. Identify the stack—AI SDK, LangGraph, OpenAI Agents SDK, Claude Agent SDK, or a custom loop—and the core use case, such as support, SQL, or coding work.
2. How the frontend and Agent communicate. Is it HTTP, gRPC, or WebSocket? Is the protocol standard or custom: AI SDK UI Message Stream, OpenAI Responses or Chat Completions, an SDK event stream, or custom JSON/SSE frames? This decides whether a built-in Adapter needs no mapping or a custom send must map events.
3. Whether the backend already has OTel. Look for OTel SDK initialization, AI SDK telemetry, LangSmith, OpenLLMetry, OpenInference, or related instrumentation. An existing setup makes Tier 2 nearly free.
4. Whether the user already has A/B tests or feature flags. Existing variation switches are an immediate Tier 3 entry point because an Experiment can pass them through as flags.
5. Which Judge to use. Semantic scoring with t.judge.autoevals.* needs a Judge model separate from the Agent under test and an OpenAI-compatible /chat/completions endpoint. Ask which key and Judge model the user wants. There is no built-in default. A missing key does not silently pass: a Judge assertion becomes unavailable, and a non-optional assertion makes the Attempt errored. Use precise assertions only when the user deliberately omits a Judge or explicitly chains .optional().
6. Whether the Agent itself needs a Sandbox. A coding-agent CLI, or a Skill, Plugin, Hook, or MCP server written for one, must modify files and run commands in an isolated workspace. It cannot use an HTTP-service-style send path. Recommend dockerImageSandbox({ image: "node:24-slim" }) by default, but confirm whether the local machine or CI has Docker and whether the user needs Vercel Sandbox or another remote provider. Use Docker when the user has no objection. The provider is declared on the Eval or Experiment; there is no CLI flag, project-wide default, or auto-detection. See Choose a Sandbox Provider.

• Tier 1 (send only): No application change. The full assertion set—text, Judge, multi-turn, tool, and HITL—already works here.
• Tier 2 (send + OTel): The application sends spans to NiceEval as well, which enables the call waterfall in niceeval view. Existing instrumentation makes this nearly free.
• Tier 3 (application changes + flags): Expose internal variations as flags for feature A/B testing. Existing A/B switches are ready-made entry points.

​

Step 2: Configure a Judge

import { defineConfig } from "niceeval";

export default defineConfig({
  judge: {
    model: "gpt-5.4-mini",                // Required: there is no built-in default model
    // Add these only for a non-OpenAI compatible service, such as DeepSeek or a gateway:
    // baseUrl: "https://api.deepseek.com/v1",
    // apiKeyEnv: "DEEPSEEK_API_KEY",     // Reads the key from this variable; otherwise NICEEVAL_JUDGE_KEY
  },
});

• Misconfiguration cannot pretend to pass. An unresolved model or key, rejected gateway authentication, or Judge timeout records the assertion as unavailable with a reason and makes the Attempt errored. A result that could not be judged is neither a pass nor the Agent’s failure. Use .optional() only when one assertion may deliberately be absent, such as an experimental assertion on a development machine without a key.
• Verify the setup once. Run a lightweight eval against fixed text before calling the Agent: t.judge.autoevals.closedQA("Does this text express success?", { on: "operation completed successfully" }).gate(0.8). Then use the receipt’s Run ID with niceeval show --run <runId> --page attempt-<attemptId> and confirm that each assertion has a score.
• Do not silently skip a Judge during autonomous onboarding. First look for an available key, such as NICEEVAL_JUDGE_KEY or DEEPSEEK_API_KEY. If one exists, point apiKeyEnv at it and verify as above. Only when no key exists should the workflow use precise and shape assertions, and the final handoff must say that a Judge was not configured and why.
• The Judge model must be separate from the Agent under test so a model does not score itself. See Judge for precedence and scoring shapes, and defineConfig Reference for the complete judge fields.

​

Step 3: Write the three pieces

1. Adapter (agents/*.ts or the user’s established directory). Use defineAgent to implement send; put static configuration in factory arguments rather than hard-coding it or reading process.env. See Adapter, defineAgent Reference, and Event Reference. Two common mistakes matter: connect the endpoint or mode that exercises the system’s core capability, not merely the easiest endpoint; and declare evidenceCoverage truthfully. Mapping only final text does not justify completeEvidenceCoverage; a false claim is worse than a conservative one.
2. Experiment (experiments/*.ts). Reference the Adapter and declare model, flags, attempts, and related settings. Put model comparisons in separate experiment files, each with one pinned model. evals: (eval) => boolean decides which evals each Experiment runs. Paths provide IDs and batch execution; reports consume physical results and coverage facts from the current Sample.
3. Eval (evals/*.eval.ts). Understand the application first, then write an eval around its real core use case. Read its README, routes, tool definitions, or system prompt. For a support Agent, ask a real support question; for a SQL Agent, use a real query task. Avoid a generic “hello” and a meta-question such as “what can you do?” Both fail to exercise the system’s work. Start with one input, t.succeeded(), and one assertion about the expected answer, but treat that shape as a connectivity scaffold rather than the delivery standard. Before you finish, meet both requirements:
   • An assertion must fail when the system hallucinates. Do not assert a word already present in the input. Asking “What is X?” then asserting that the answer contains “X” lets the Agent pass by repeating the prompt. Assert real expected facts, structure through hasSections(), a real URL through includesUrl(), or semantic correctness through t.judge.
   • Add at least one negative case. Give the system input it should not be able to answer, such as a nonexistent table or unavailable topic. Assert that it clearly says it cannot find or perform the task rather than inventing a plausible result. This is often the most valuable early failure shape for an Agent connected to real data or retrieval.
   See Write Evals for authoring, Evaluation Kinds for assertions and question forms, and defineEval Reference for the signature.

• Do not call the system directly in-process. Even when the Agent runtime and eval share a codebase, the Adapter should use HTTP or the appropriate transport. Do not replace fetch with an import of the system under test. See Connect Your Agent for the reason.
• Do not manage the system-under-test process from the eval. Do not spawn the application or open another port. The user starts it the normal way, such as pnpm dev. When the Adapter cannot connect, report a clear “start the application first” error instead of launching it.

​

Step 4: Run and verify

<package-manager> exec niceeval exp models
<package-manager> exec niceeval view --experiment models

​

Step 5: Finish by telling the user what changed

• Eval inputs exercise the system’s core use case, not a meta-question or placeholder greeting.
• Every content assertion fails when the system repeats the input or hallucinates; its asserted terms are not already in the input.
• At least one negative case makes an unsupported request and asserts an explicit inability to complete it.
• When a key is available, the Judge is configured and a Judge score has appeared in niceeval view. If no key exists, the summary explains that.
• The Experiment’s declared model and flags are actually consumed by the Adapter. There is no dead configuration or invented model value.

​

Step 6: Ask whether the user wants a deeper integration