Put configuration and secrets where they belong

Values in NiceEval have exactly two homes:

Configuration lives in code — CLI flags, experiment files under experiments/, and niceeval.config.ts at the root. How many attempts, how long before timeout, how much concurrency, which judge model, which report to show by default: all of it lives here, with no environment variable counterpart.
Secrets live in environment variables — API keys and provider tokens, plus facts like NO_COLOR that describe which terminal you are printing to.

So every value has exactly one source. What niceeval exp --dry prints is what actually takes effect; no environment variable is quietly changing it behind your back. CLI and runtime copy is English; the browser view has its own English and Chinese switch.

Configuration: pick the layer by how long the value should last

When the same value appears in more than one layer, resolution is CLI flag → experiment → config → built-in default, stopping at the first one present. Just this once — put it on the command:

npx niceeval exp ci --attempts 5 --timeout 600000

Always, for this experiment — put it in the experiment file:

// experiments/ci.ts
import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/agents";

export default defineExperiment({
  agent: codexAgent(),
  model: "gpt-5.4",
  attempts: 3,
  timeoutMs: 600_000,
  budget: 5,
});

agent, model, and flags can only be written here — there is no flag for them. Switching agent or model means copying an experiment file, which is what keeps “what did this run go against” recorded in the snapshot and reproducible afterwards. Shared by the whole project — put it in niceeval.config.ts:

import { defineConfig } from "niceeval";

export default defineConfig({
  judge: { model: "gpt-5.4-mini" },
  maxConcurrency: 4,
  timeoutMs: 300_000,
});

What `niceeval.config.ts` holds

Field	What it does
`name`	Project name shown at the top of `niceeval view`; can be given per language
`report`	Default report definition, loaded by a bare `show` / `view` (see custom reports)
`judge`	Default judge configuration: `model`, `baseUrl`, `apiKeyEnv`
`workspace`	Workspace root uploaded into the sandbox
`reporters`	Default reporter list (write to disk, push to an experiment platform)
`maxConcurrency`	Default concurrency ceiling
`timeoutMs`	Default per-attempt timeout
`telemetry`	OTLP receiver configuration (`host` / `port`); OTel intake is configured only here
`pricing`	Price table overrides, keyed by model name or a `provider/*` wildcard

Types and full descriptions are in the defineConfig reference; the full flag table is in the CLI reference.

Environment variables: secrets and terminal facts only

These are all the environment variables NiceEval reads. Each agent, sandbox, and the judge recognizes exactly one name and never goes hunting for another key in the environment.

Variable	Used by	Notes
`ANTHROPIC_API_KEY`	`claudeCodeAgent()`, `openClawAgent()`	Overridable with the factory option `apiKey`
`ANTHROPIC_BASE_URL`	`claudeCodeAgent()`	Gateway address, overridable with `baseUrl`
`CODEX_API_KEY`	`codexAgent()`	Not `OPENAI_API_KEY`; overridable with `apiKey`
`CODEX_BASE_URL`	`codexAgent()`	OpenAI-compatible proxy address, overridable with `baseUrl`
`BUB_API_KEY`	`bubAgent()`	Overridable with `apiKey`
`BUB_API_BASE`	`bubAgent()`	Overridable with `apiBase`
`NICEEVAL_JUDGE_KEY`	judge	The judge’s default key variable
`E2B_API_KEY`	`e2bSandbox()`
`VERCEL_API_TOKEN`	`vercelSandbox()`
`VERCEL_TEAM_ID`	`vercelSandbox()`
`VERCEL_PROJECT_ID`	`vercelSandbox()`
`NO_COLOR`	CLI output	When set, no colors and no box drawing

To keep the judge’s key under a different variable name, point at it from the config:

export default defineConfig({
  judge: {
    model: "gpt-5.4-mini",
    baseUrl: "https://gateway.example.com/v1",   // the gateway address is configuration
    apiKeyEnv: "MY_GATEWAY_KEY",                  // the key is a secret, read from this variable
  },
});

When the gateway address itself is not something you want committed, NiceEval doesn’t need to offer a variable for it — configuration is code, so read it yourself (.env is already loaded by then):

export default defineConfig({
  judge: {
    model: "gpt-5.4-mini",
    baseUrl: process.env.MY_GATEWAY_URL,   // your variable name, read by you
    apiKeyEnv: "MY_GATEWAY_KEY",
  },
});

The difference is only this: the variable name belongs to your project, instead of NiceEval baking in a name and going looking for it in the environment. Locally, put secrets in .env in the current directory — the CLI loads it at startup (without overwriting variables that already exist), so no export every time:

# .env
ANTHROPIC_API_KEY=sk-ant-...
NICEEVAL_JUDGE_KEY=sk-...

.env is how secrets get delivered, not a second config file — putting something like NICEEVAL_TIMEOUT in it has no effect. In CI, pass only secrets the same way:

- run: npx niceeval exp ci --strict --junit ./artifacts/niceeval-junit.xml
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Seen these in an old script? Move them into configuration

Old form	Write it as
`NICEEVAL_RUNS`	`--attempts`, or `attempts` on an experiment
`NICEEVAL_TIMEOUT`	`--timeout`, or `timeoutMs` on an experiment / config
`NICEEVAL_BUDGET`	`--budget`, or `budget` on an experiment
`NICEEVAL_MAX_CONCURRENCY`	`--max-concurrency`, or `maxConcurrency` in config
`NICEEVAL_JUDGE_MODEL`	`judge.model` in config or on an eval
`NICEEVAL_JUDGE_BASE`	`judge.baseUrl` in config or on an eval
`NICEEVAL_OTLP_HOST` / `NICEEVAL_OTLP_PORT`	`telemetry: { host, port }` in config

The judge also no longer borrows a key from CODEX_API_KEY / OPENAI_API_KEY or guesses its endpoint from OPENAI_BASE_URL. When the app under test repurposes the standard OPENAI_* names for something else, the judge no longer gets dragged along.

defineConfig reference

Type and full description of every config field.

CLI reference

Commands, the full flag table, and exit codes.

Write an experiment

Which values belong to one concrete run.

CI integration

Passing secrets in CI.

Get Started

Connect Your Agent

Write Evals

Run, Observe & Publish

Troubleshooting

Put configuration and secrets where they belong

Configuration: pick the layer by how long the value should last

What `niceeval.config.ts` holds

Environment variables: secrets and terminal facts only

Seen these in an old script? Move them into configuration

Next

defineConfig reference

CLI reference

Write an experiment

CI integration

​Configuration: pick the layer by how long the value should last

​What niceeval.config.ts holds

​Environment variables: secrets and terminal facts only

​Seen these in an old script? Move them into configuration

​Next

defineConfig reference

CLI reference

Write an experiment

CI integration

Configuration: pick the layer by how long the value should last

What `niceeval.config.ts` holds

Environment variables: secrets and terminal facts only

Seen these in an old script? Move them into configuration

Next