Bring a Coding Agent into a Project

​

Step 1: Explore the project, then confirm the path with the user

1. What kind of Agent this is. Read the README, package.json dependencies, routes, and Agent-loop code. Identify its stack—AI SDK, LangGraph, OpenAI Agents SDK, Claude Agent SDK, or a custom loop—and its core use case, such as support, SQL, or coding work.
2. How the frontend and Agent communicate. Is it HTTP, gRPC, or WebSocket? Is the protocol standard or custom: AI SDK UI Message Stream, OpenAI Responses or Chat Completions, an SDK-native event stream passthrough, or custom JSON/SSE frames? This directly determines whether to use a built-in Adapter with zero mapping or hand-write send with event mapping.
3. Whether the backend already has OTel. Look for OTel SDK initialization, AI SDK telemetry, LangSmith, OpenLLMetry, OpenInference, or related instrumentation. An existing setup makes Tier 2 nearly free.
4. Whether the user already has A/B tests or feature flags. Existing variation switches are an immediate Tier 3 entry point because an Experiment can pass them through as flags.
5. Which Judge to use. Semantic evaluation with Judge Matches from niceeval/expect needs a Judge model separate from the Agent under test and an OpenAI-compatible /chat/completions endpoint. Ask which service key the user has and which Judge model they want. A missing key does not silently pass: a Judge Assertion becomes unavailable; when it participates in pass grading, or has .score() / .orStop() in a scored eval, the Attempt’s grading is unavailable. Without a Judge configuration, start with precise Matches only.
6. Whether the Agent itself needs a Sandbox. A coding-agent CLI, or a Skill, Plugin, Hook, or MCP server written for a coding agent, must edit files and run commands in an isolated Workspace. It must use a Sandbox rather than an HTTP-service-style send path. Recommend dockerSandbox({ source: { type: "image", image: "node:24-slim" } }) by default. Follow a Provider already specified in the task or configuration; ask only when the Provider, Docker availability, or remote-execution need would change the plan. The Provider is declared on the eval or Experiment; there is no CLI flag, project-wide default, or auto-detection. See Choose a Sandbox Provider.

• Tier 1 (send only): No application change. The full assertion set—text, Judge, multi-turn, tools, and HITL—already works here.
• Tier 2 (send + OTel): The application also sends OTel spans to NiceEval, where you can review the call waterfall in niceeval view. Existing instrumentation makes this nearly free.
• Tier 3 (invasive changes + flags): Expose internal variations as flags for feature A/B testing. Existing A/B switches are ready-made entry points.

​

Step 2: Configure a Judge

import { defineConfig } from "niceeval";

export default defineConfig({
  judge: {
    model: "gpt-5.4-mini",                // Required: there is no built-in default model
    // Add these only for a non-OpenAI-compatible service (DeepSeek, a gateway, and so on):
    // baseUrl: "https://api.deepseek.com/v1",
    // apiKeyEnv: "DEEPSEEK_API_KEY",     // Read the key from this variable. Without it, use OPENAI_API_KEY
  },
});

• A misconfiguration does not pretend to pass. If the model or key cannot be resolved, the gateway rejects authentication, or an evaluation request times out, the Judge Assertion is unavailable with a reason. The reading surface does not disguise a missing measurement as a mismatch or zero.
• Verify it once after configuration. Use defineJudge to declare a rubric and anchors, then call t.judge(material, definition).gate(0.8). After it runs, use niceeval view --run <run-id> to inspect the result.
• Do not skip a Judge directly when no user is present during autonomous onboarding. First check whether the environment already has a usable key, such as OPENAI_API_KEY or DEEPSEEK_API_KEY. If it does, point apiKeyEnv to it and verify. If not, write precise Matches first and state at handoff that no Judge is configured.
• Keep the Judge model separate from the Agent under test so a model does not score its own output. Configuration resolves field by field across the eval, Experiment, and project configuration. See Judge for the three factories.

​

Step 3: Write the three pieces

1. Adapter (agents/*.ts, or the directory established by the user’s project). Use defineAgent to implement send. Put configuration in factory parameters; do not hard-code it or read process.env. See the Adapter contract, Adapter Reference signature, and Event Reference for event mapping. Two easy mistakes matter: choose an endpoint or mode that exercises the system under test’s core capability, not merely the easiest path to make work—if a platform has both plain LLM chat and a database-connected execution mode, connecting the former evaluates a low-level model proxy rather than the product. Also, declare evidenceCoverage truthfully based on the actual mapping. If you map only final text, do not use completeEvidenceCoverage; the declaration changes assertion completeness, and overstating it is worse than being conservative.
2. Experiment (experiments/*.ts). Reference the Adapter above and declare model, flags, attempts, and so on. Model comparison uses two Experiment files, each pinned to one model. evals: (eval) => boolean decides which evals each one runs. Paths only provide IDs and batch execution; fixed Inspection operations consume the current published physical results and coverage facts.
3. Eval (evals/*.eval.ts). First discover what the application does, then write one eval aligned with a real capability. Read its README, routes, tool definitions, or system prompt to find the core use case. A support bot should receive a real support question; a SQL Agent should receive a real query task. Use that as the input and Assertions for the first eval. Two inputs are unacceptable: a placeholder unrelated to the application such as “Hello,” and a meta-question asking the system under test what it is or can do. Those are not tasks users ask it to perform. Start with the smallest shape—one input, t.succeeded(), and one content Assertion about the expected reply—but that shape is only scaffolding. Before handoff, it must also meet both conditions:
   • A content Assertion turns red when the system under test hallucinates. Do not assert a word already in the input—asking “What is X?” and then asserting that the reply contains “X” passes when the system simply repeats the prompt. Assert substantive content unique to the expected answer: concrete facts, structure with hasSections(), a real link with includesUrl(), or a Judge Match registered through check.
   • Include at least one negative case. Give input the system should not be able to answer, such as a nonexistent table or unavailable retrieval topic. Assert that it clearly says it cannot find or do it instead of fabricating a plausible-looking result. For Agents connected to real data or retrieval sources, this is the failure shape most worth testing first. See Authoring Evals for the form, Evaluation Kinds and Assertions for Assertions and kinds, and defineEval Reference for the signature.

• Do not make an in-process direct call. Even if the Agent runtime and eval live in the same repository, the Adapter must use HTTP or the appropriate transport layer. Do not replace fetch with a direct import of the function under test. See “Why not call it directly?” in Connect Your Agent.
• Do not manage the system-under-test process from the eval side. Do not spawn the application or open another port. The user starts it normally, such as with pnpm dev. If the Adapter cannot connect, report a direct error such as “start the application first”; do not start a service itself.

​

Step 4: Run and verify

<package-manager> exec niceeval exp models   # Run both model configurations by path
<package-manager> exec niceeval view          # Inspect comparison results in the viewer

​

Step 5: Finish by telling the user what changed

• The eval input is a core use case of the system under test, not a meta-question about itself or a placeholder greeting.
• Every content Assertion turns red when the system repeats the prompt or hallucinates; asserted words do not already occur in the input.
• At least one negative case provides input that should not be answerable and asserts an explicit inability to do it.
• When a key is available, the Judge is configured and a Judge score has appeared in niceeval view. When no key is available, the summary says so.
• The Experiment’s declared model / flags are actually consumed by the Adapter. There is no dead configuration and no invented model value the system under test does not support.

​

Step 6: Ask whether the user wants a deeper integration