Put Configuration and Secrets Where They Belong

You want CI to run each eval 5 times and local runs only once, without committing an API key to the repository. NiceEval splits values into two kinds, each in its own place:

Configuration goes in code: CLI flags, Experiment files under experiments/, and niceeval.config.ts at the root. Attempts, timeouts, concurrency, and which model the Judge uses all live here, and none of them have a matching environment variable.
Secrets go in environment variables: API keys and Provider tokens, plus settings that describe the current terminal, such as NO_COLOR.

So each value has exactly one source. What niceeval exp --dry prints is the value that really takes effect; no environment variable changes it behind your back. CLI and runtime text is in English; niceeval view in the browser can switch between Chinese and English.

Configuration: choose the layer by how long the value should live

First decide how widely the value should apply, then decide which layer to write it in. When the same value appears in several layers, it is taken in the order CLI flag → Experiment → eval → config → built-in default; once an earlier layer has a value, later ones are not consulted. The config only supplies defaults and never overrides other layers: a timeoutMs declared by an eval itself is not replaced by the project default. Different only this once: write it on the command.

pnpm exec niceeval exp ci --attempts 5 --timeout 600000

Always this way for this experiment: write it in the Experiment file.

// experiments/ci.ts
import { defineExperiment } from "niceeval";
import { codexAgent } from "niceeval/agents";

export default defineExperiment({
  agent: codexAgent(),
  model: "gpt-5.4",
  attempts: 3,
  timeoutMs: 600_000,
  budget: 5,
});

agent, model, and flags can only be written here; there is no matching CLI flag. To switch Agent or model, copy an Experiment file. That way, which Agent and model each run used is recorded in the results and can be reproduced later. Shared across the whole project: write it in niceeval.config.ts.

import { defineConfig } from "niceeval";
import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({ model: "gpt-5.4-mini" }),
  maxConcurrency: 4,
  timeoutMs: 300_000,
});

What `niceeval.config.ts` can hold

Field	What goes in it
`name`	Project name, shown at the top of `niceeval view`. Can have one per language
`judgeProvider`	Default Judge Provider, built with a named factory from `niceeval/judge`
`workspace`	Root of the workspace uploaded into the Sandbox
`reporters`	Default reporter list (write to disk, report to an experiment platform)
`maxConcurrency`	Default concurrency cap
`timeoutMs`	Default timeout per attempt
`telemetry`	OTLP receiver configuration (`host` / `port`); OTel integration is configured only here
`pricing`	Price table overrides, by model name or `provider/*` wildcard

Each field’s type and full description are in the defineConfig reference. The full flag table is in the CLI reference.

Environment variables: only secrets and the terminal

Put secrets in the environment variables below. These are all the environment variables NiceEval reads; each Agent, Sandbox, and Judge only recognizes its own few names and never reads other keys.

Variable	Used by	Notes
`ANTHROPIC_API_KEY`	`claudeCodeAgent()`	Can be overridden by the factory parameter `apiKey`
`ANTHROPIC_BASE_URL`	`claudeCodeAgent()`	Gateway address; can be overridden by the factory parameter `baseUrl`
`CODEX_API_KEY` / `CODEX_BASE_URL`	`codexAgent()`	Key required, proxy address optional; can be overridden by the factory parameters `apiKey` / `baseUrl`
`BUB_API_KEY` / `BUB_API_BASE`	`bubAgent()`	Both required; can be overridden by the factory parameters `apiKey` / `baseUrl`
`OPENCLAW_API_KEY` / `OPENCLAW_BASE_URL`	`openClawAgent()`	When the key is omitted, falls back to `ANTHROPIC_API_KEY`
`OPENCODE_API_KEY` / `OPENCODE_BASE_URL`	`openCodeAgent()`	When the key is omitted, falls back to `ANTHROPIC_API_KEY`
`HERMES_API_KEY` / `HERMES_API_BASE`	`hermesAgent()`	When the key is omitted, falls back to `OPENROUTER_API_KEY`, then `ANTHROPIC_API_KEY`
`OPENAI_API_KEY`	`OpenAIProvider`	Default key for OpenAI or an explicit compatible gateway
`AI_GATEWAY_API_KEY`	`VercelProvider`	Vercel AI Gateway key
`OPENROUTER_API_KEY`	`OpenRouterProvider`	OpenRouter key
`TYPESAFE_API_KEY`	`TypesafeProvider`	TypeSafe System One key
`E2B_API_KEY`	`e2bSandbox()`
`VERCEL_API_TOKEN`	`vercelSandbox()`
`VERCEL_TEAM_ID`	`vercelSandbox()`
`VERCEL_PROJECT_ID`	`vercelSandbox()`
`NO_COLOR`	CLI output	When set, no colors or box lines are drawn

To have the Judge read its key from a different variable name, spell out the variable name in the config:

import { OpenAIProvider } from "niceeval/judge";

export default defineConfig({
  judgeProvider: OpenAIProvider({
    model: "gpt-5.4-mini",
    baseUrl: "https://gateway.example.com/v1",   // The gateway address is configuration, so it goes here
    apiKeyEnv: "MY_GATEWAY_KEY",                  // The key is a secret, so read it from this environment variable
  }),
});

When the gateway address should not be committed to the repository either, read an environment variable yourself in the config. The config is code, and .env has already loaded by the time it is read:

export default defineConfig({
  judgeProvider: OpenAIProvider({
    model: "gpt-5.4-mini",
    baseUrl: process.env.MY_GATEWAY_URL,   // You name the variable and read it yourself
    apiKeyEnv: "MY_GATEWAY_KEY",
  }),
});

The only difference: the variable name is decided by your project, not a name built into NiceEval that it guesses from the environment. For local development, put secrets in a .env file in the current directory. The CLI loads it automatically at startup without overriding environment variables that already exist, so you do not need to export every time:

# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

.env is only for secrets; it is not a second configuration file. Writing variables like NICEEVAL_TIMEOUT into it has no effect. CI likewise passes only secrets:

- run: pnpm exec niceeval exp ci --junit .niceeval/junit.xml
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Which layer to write in when tuning these values

Configuration has no environment variable layer, so each item below can only be written in these places:

What to tune	Where to write it
How many times each eval runs	`--attempts`, or the experiment’s `attempts`
Timeout per Attempt	`--timeout`, or the experiment / config `timeoutMs`
Spending cap	`--budget`, or the experiment’s `budget`
Concurrency cap	`--max-concurrency`, or the config’s `maxConcurrency`
Judge model and endpoint	A named Provider in the config or Experiment; an eval can switch only the model with a `judge` string
OTel receiver address	The config’s `telemetry: { host, port }`

Each Judge Provider reads only its own default key variable, or the variable named explicitly by the factory’s apiKeyEnv. It never borrows keys from other Providers or from the application under test.

defineConfig reference

Type and full description of each configuration field.

CLI reference

Commands, full flag table, and exit codes.

Write experiments

Which values belong to one specific run.

CI integration

Pass secrets in CI.

Get Started

Connect Your Application

Write Evals

Evaluate Coding Agents

Experiments and Comparisons

Migration

Review Results and Iterate

Concurrency and Reuse

Infrastructure and CI

Troubleshooting

Put Configuration and Secrets Where They Belong

Configuration: choose the layer by how long the value should live

What `niceeval.config.ts` can hold

Environment variables: only secrets and the terminal

Which layer to write in when tuning these values

Read next

defineConfig reference

CLI reference

Write experiments

CI integration

Configuration: choose the layer by how long the value should live

What niceeval.config.ts can hold

Environment variables: only secrets and the terminal

Which layer to write in when tuning these values

Read next

defineConfig reference

CLI reference

Write experiments

CI integration

What `niceeval.config.ts` can hold