> ## Documentation Index > Fetch the complete documentation index at: https://niceeval.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Put configuration and secrets where they belong > Which layer of code holds attempts, timeout, concurrency, and judge — and which environment variables carry API keys and provider tokens. Values in [NiceEval](https://niceeval.com/) have exactly two homes: * **Configuration lives in code** — CLI flags, experiment files under `experiments/`, and `niceeval.config.ts` at the root. How many attempts, how long before timeout, how much concurrency, which judge model, which report to show by default: all of it lives here, with no environment variable counterpart. * **Secrets live in environment variables** — API keys and provider tokens, plus facts like `NO_COLOR` that describe *which terminal you are printing to*. So every value has exactly one source. What `niceeval exp --dry` prints is what actually takes effect; no environment variable is quietly changing it behind your back. CLI and runtime copy is English; the browser view has its own English and Chinese switch. ## Configuration: pick the layer by how long the value should last When the same value appears in more than one layer, resolution is **CLI flag → experiment → config → built-in default**, stopping at the first one present. **Just this once** — put it on the command: ```bash theme={null} npx niceeval exp ci --attempts 5 --timeout 600000 ``` **Always, for this experiment** — put it in the experiment file: ```ts theme={null} // experiments/ci.ts import { defineExperiment } from "niceeval"; import { codexAgent } from "niceeval/agents"; export default defineExperiment({ agent: codexAgent(), model: "gpt-5.4", attempts: 3, timeoutMs: 600_000, budget: 5, }); ``` `agent`, `model`, and `flags` can only be written here — there is no flag for them. Switching agent or model means copying an experiment file, which is what keeps "what did this run go against" recorded in the snapshot and reproducible afterwards. **Shared by the whole project** — put it in `niceeval.config.ts`: ```ts theme={null} import { defineConfig } from "niceeval"; export default defineConfig({ judge: { model: "gpt-5.4-mini" }, maxConcurrency: 4, timeoutMs: 300_000, }); ``` ### What `niceeval.config.ts` holds | Field | What it does | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Project name shown at the top of `niceeval view`; can be given per language | | `report` | Default report definition, loaded by a bare `show` / `view` (see [custom reports](/docs/tutorials/custom-reports#set-it-as-the-project-default)) | | `judge` | Default judge configuration: `model`, `baseUrl`, `apiKeyEnv` | | `workspace` | Workspace root uploaded into the sandbox | | `reporters` | Default reporter list (write to disk, push to an experiment platform) | | `maxConcurrency` | Default concurrency ceiling | | `timeoutMs` | Default per-attempt timeout | | `telemetry` | OTLP receiver configuration (`host` / `port`); OTel intake is configured only here | | `pricing` | Price table overrides, keyed by model name or a `provider/*` wildcard | Types and full descriptions are in the [defineConfig reference](/docs/reference/define-config); the full flag table is in the [CLI reference](/docs/reference/cli). ## Environment variables: secrets and terminal facts only These are all the environment variables NiceEval reads. Each agent, sandbox, and the judge recognizes exactly one name and never goes hunting for another key in the environment. | Variable | Used by | Notes | | -------------------- | -------------------------------------- | ----------------------------------------------------------- | | `ANTHROPIC_API_KEY` | `claudeCodeAgent()`, `openClawAgent()` | Overridable with the factory option `apiKey` | | `ANTHROPIC_BASE_URL` | `claudeCodeAgent()` | Gateway address, overridable with `baseUrl` | | `CODEX_API_KEY` | `codexAgent()` | Not `OPENAI_API_KEY`; overridable with `apiKey` | | `CODEX_BASE_URL` | `codexAgent()` | OpenAI-compatible proxy address, overridable with `baseUrl` | | `BUB_API_KEY` | `bubAgent()` | Overridable with `apiKey` | | `BUB_API_BASE` | `bubAgent()` | Overridable with `apiBase` | | `NICEEVAL_JUDGE_KEY` | judge | The judge's default key variable | | `E2B_API_KEY` | `e2bSandbox()` | | | `VERCEL_API_TOKEN` | `vercelSandbox()` | | | `VERCEL_TEAM_ID` | `vercelSandbox()` | | | `VERCEL_PROJECT_ID` | `vercelSandbox()` | | | `NO_COLOR` | CLI output | When set, no colors and no box drawing | To keep the judge's key under a different variable name, point at it from the config: ```ts theme={null} export default defineConfig({ judge: { model: "gpt-5.4-mini", baseUrl: "https://gateway.example.com/v1", // the gateway address is configuration apiKeyEnv: "MY_GATEWAY_KEY", // the key is a secret, read from this variable }, }); ``` When the gateway address itself is not something you want committed, NiceEval doesn't need to offer a variable for it — configuration is code, so read it yourself (`.env` is already loaded by then): ```ts theme={null} export default defineConfig({ judge: { model: "gpt-5.4-mini", baseUrl: process.env.MY_GATEWAY_URL, // your variable name, read by you apiKeyEnv: "MY_GATEWAY_KEY", }, }); ``` The difference is only this: the variable name belongs to your project, instead of NiceEval baking in a name and going looking for it in the environment. Locally, put secrets in `.env` in the current directory — the CLI loads it at startup (without overwriting variables that already exist), so no `export` every time: ```bash theme={null} # .env ANTHROPIC_API_KEY=sk-ant-... NICEEVAL_JUDGE_KEY=sk-... ``` `.env` is how secrets get delivered, not a second config file — putting something like `NICEEVAL_TIMEOUT` in it has no effect. In CI, pass only secrets the same way: ```yaml theme={null} - run: npx niceeval exp ci --strict --junit ./artifacts/niceeval-junit.xml env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} ``` ## Seen these in an old script? Move them into configuration | Old form | Write it as | | ------------------------------------------- | ----------------------------------------------------- | | `NICEEVAL_RUNS` | `--attempts`, or `attempts` on an experiment | | `NICEEVAL_TIMEOUT` | `--timeout`, or `timeoutMs` on an experiment / config | | `NICEEVAL_BUDGET` | `--budget`, or `budget` on an experiment | | `NICEEVAL_MAX_CONCURRENCY` | `--max-concurrency`, or `maxConcurrency` in config | | `NICEEVAL_JUDGE_MODEL` | `judge.model` in config or on an eval | | `NICEEVAL_JUDGE_BASE` | `judge.baseUrl` in config or on an eval | | `NICEEVAL_OTLP_HOST` / `NICEEVAL_OTLP_PORT` | `telemetry: { host, port }` in config | The judge also no longer borrows a key from `CODEX_API_KEY` / `OPENAI_API_KEY` or guesses its endpoint from `OPENAI_BASE_URL`. When the app under test repurposes the standard `OPENAI_*` names for something else, the judge no longer gets dragged along. ## Next Type and full description of every config field. Commands, the full flag table, and exit codes. Which values belong to one concrete run. Passing secrets in CI.