Eval an AI Agent Application with a Hand-Written Adapter - niceeval

This project shows how to use defineAgent to connect an existing HTTP agent to NiceEval. It’s a good fit for applications that need control over their own protocol and event mapping.

View the full source
Local directory: examples/zh/ai-sdk/

If your application uses AI SDK v7’s UI Message Stream, prefer the built-in uiMessageStreamAgent instead. See the corresponding non-intrusive example: AI SDK v7.

What this project demonstrates

Part	Implementation
System under test	AI SDK v6 web agent, HTTP `POST /api/turn`
Adapter	`defineAgent` + custom `AgentEvent` → `StreamEvent` mapping
Evals	Weather tool, image understanding, multi-turn text, multi-turn image context
Experiment	Each file pins one model; the group compares DeepSeek against GPT
Observability	The app keeps reporting to Langfuse while also sending this turn’s OTLP data to NiceEval

The adapter is only responsible for connecting and translating. The URL and model belong to the experiment; evals only describe the interaction and what a good result looks like. This separation lets the same evals be reused across different models and deployment instances.

Directory structure

examples/zh/ai-sdk/
├── src/                     # web agent under test
├── adapter/adapter.ts       # hand-written NiceEval adapter
├── evals/                   # tool, image, and multi-turn evals
├── experiments/             # model comparison
└── niceeval.config.ts

Run it

This example calls real models — there’s no mock mode. Copy the environment variables first and fill in credentials for an OpenAI-compatible service:

cd examples/zh/ai-sdk
pnpm install
cp .env.example .env
pnpm dev

In a second terminal, run:

cd examples/zh/ai-sdk
pnpm exec niceeval list
pnpm exec niceeval exp compare-models weather-tool
pnpm exec niceeval view

Next steps

Connect Your Agent: Start from a minimal adapter and connect your own protocol.
Write Send: Gradually fill in sessions, tools, HITL, and OTel.
Experiment Matrix: Organize model and configuration comparisons.