Skip to main content
defineConfig is the default export of niceeval.config.ts at the project root, and only holds project-wide defaults. Agent, model, flags, attempts, and per-experiment budget belong in defineExperiment files under experiments/.
report takes the defineReport product itself (import your own report file), not a path string. niceeval show and niceeval view load it when --report is absent, and fall back to the built-in default report when it is not set; --report overrides it per run, and --report standard goes back to the built-in one. Writing reports is covered in Write a custom report.

​
Config fields

​
name

Project name, shown in the hero (<h1>) at the top of niceeval view; falls back to a generic title when omitted. Can be a plain string, or per-locale text (e.g. { en: "...", "zh-CN": "..." }) that switches with the view’s language.

​
workspace

Root of the workspace directory uploaded into the Sandbox; falls back to the project root when omitted. This is where an eval’s Sandbox view starts from.

​
judge

Project-level default judge configuration (model / baseUrl / apiKeyEnv); EvalDef.judge can override it per eval.

​
reporters

Project-level default reporter list (e.g. writing results to disk / uploading them); EvalDef.reporters is merged with it.

​
maxConcurrency

Project-level default concurrency ceiling; the CLI flag / experiment setting of the same name takes priority (there is no environment variable layer).

​
timeoutMs

Project-level default timeout for a single attempt (milliseconds); the CLI flag / experiment / EvalDef setting of the same name takes priority.

​
telemetry

OTLP receiving configuration, the single entry point within a niceeval project (does not read NICEEVAL_OTLP_* env vars). port pins the receiving port (fixed-port mode: a long-running service points OTEL_EXPORTER_OTLP_ENDPOINT once at http://localhost:<port>/v1/traces, and it never needs to change no matter how many eval runs happen). Omit it and each run dynamically allocates a temporary port (handed to the adapter via ctx.telemetry). Trade-off: with a fixed port, only one niceeval process can run on the same machine at a time, and it errors if that port is already taken by another process — pick a free port and write it back here. host is the receiving-end hostname reported to the adapter (not the listen address, which is always 0.0.0.0); defaults to “127.0.0.1”. Override it here only when you have already provided a controlled tunnel or a reachable route; Docker Sandboxes place the receiver inside the Sandbox by default and do not rely on an implicit host gateway.

​
pricing

User overrides / additions on top of the built-in price table (o11y/prices.json), looked up by model (see Observability · usage and cost). Keys support either an exact model name or a provider/* wildcard (for bulk overrides on self-hosted/gateway discounts by provider); exact keys take priority over wildcards. Only used when there is no gateway-measured actual cost (usage.costUSD) — measured cost always takes priority over an estimate. NiceEval keeps environment preparation in ordinary code: files an eval itself needs are written in test(t), and an agent’s own preparation is written in the adapter’s setup. Config deliberately has no sandbox field and no provider auto-detection. Declare a template-bearing SandboxLayer on the Eval or Experiment so link planning can identify its owner and pair it before any resource is created.