defineConfig: Project Defaults

defineConfig is the default export from niceeval.config.ts at the project root and holds only project-level defaults. Agent, model, Flags, Attempts, and Experiment budgets belong in defineExperiment files under experiments/.

import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({ model: "gpt-5.4-mini", maxOutputTokens: 512 }),
  maxConcurrency: 4,
  timeoutMs: 300_000,
});

Config fields

`name`

name?: LocalizedText;

The project name, shown in the hero (<h1>) at the top of niceeval view; it falls back to a generic title when omitted. It can be a string, or text in multiple locales (such as { en: "...", "zh-CN": "..." }) that switches with the view language.

`judgeProvider`

judgeProvider?: JudgeProvider;

The project-level default Judge Provider; an Eval or Experiment can replace the model or the whole Provider.

`reporters`

reporters?: Reporter[];

The project-level Reporter list, for example for persisting or uploading results.

`maxConcurrency`

maxConcurrency?: number;

The project-level default concurrency limit; the CLI flag or Experiment setting with the same name has higher priority (there is no environment-variable layer).

`maxBuildConcurrency`

maxBuildConcurrency?: number;

Run-level concurrency for preparing Sandbox images; independent of Attempt concurrency, with a safe default of 2 when omitted.

`maxBeforeCacheConcurrency`

maxBeforeCacheConcurrency?: number;

Run-level concurrency for preparing the PreparedArtifact prefix DAG; independent of build and Attempt concurrency.

`sandboxCache`

sandboxCache?: SandboxCacheConfig;

The project-level before cache default; an Experiment or an explicit CLI flag can override it.

`timeoutMs`

timeoutMs?: number;

The project-level default timeout for one Attempt (milliseconds); the CLI flag, Experiment, or EvalDef setting with the same name has higher priority.

`pauseOnFailure`

pauseOnFailure?: boolean;

Suspend the Attempt and retain the Sandbox when an error or failure occurs, displaying native direct connect commands for debugging before cleanup.

`telemetry`

telemetry?: { host?: string; port?: number };

OTLP receiver configuration, the only entry point inside a niceeval project (it does not read NICEEVAL_OTLP_* environment variables). port pins the receiving port (fixed-port mode: a long-running service points OTEL_EXPORTER_OTLP_ENDPOINT once at http://localhost:<port>/v1/traces, and never changes it no matter how many Eval runs follow). When omitted, each run dynamically assigns a temporary port (handed to the Adapter through ctx.telemetry). The tradeoff: with a fixed port, only one niceeval process can run on the machine at a time, and an error occurs if another process holds that port; pick a free port and write it back here. host is the receiver hostname reported to the Adapter (not the listening address, which is always 0.0.0.0); it defaults to "127.0.0.1". Override it here only when you already provide a controlled tunnel or reachable route; a Docker Sandbox puts the receiver inside the Sandbox by default and does not rely on an implicit host gateway.

`pricing`

pricing?: globalThis.Record<string, PriceOverride>;

User overrides or additions on top of the built-in price table (o11y/prices.json), looked up by model (see Observability · Usage and Cost). A key can be an exact model name or a provider/* wildcard (to override self-hosted or gateway discounts in bulk per Provider); exact keys take priority over wildcards. The Runner uses it to compute the budget estimate for maxCost; the Adapter physical-call collector seals a matching explicit fixed profile as a call-bound estimate proof only when reported cost is absent. Inspection reads only sealed proofs; it does not take mutable config as an operation input, and does not backfill historical calls by endpoint.

Cost boundary: this pricing only provides the Runner’s Config/runtime price-table input. Whether or not Usage.costUSD already exists, the Runner independently produces estimatedCostUSD from the model, the reported token usage, and this table. The Experiment budget and t.maxCost() use that estimate; Usage.costUSD does not drive it. Reading results uses fixed Inspection operations, which do not turn runtime price configuration into another input.

NiceEval keeps environment preparation in ordinary code: files an Eval needs go in test(t), and an Agent’s own preparation goes in the Adapter’s setup. Config deliberately has no sandbox field and does not auto-detect a Provider. Declare a template-bearing SandboxLayer on the Eval or Experiment so link planning settles its owner and pairing result before creating any resources.

API

Adapters

CLI