Skip to main content
Coding tasks often use a hidden test to score. Only after the Agent finishes changing the code does the eval write the test file into the Sandbox and run it; if the test passes, the eval passes. The Agent cannot see this test while it works, so it cannot game it. This test file is the scoring standard. Changing it amounts to swapping the task, and old results should no longer count. Read it with loadText, and NiceEval records the file content and counts it as an input of this eval. Change the file by one byte, and this eval reruns automatically on the next run while other evals keep reusing their old results. If you read it yourself with fs.readFile, you get the same content, but NiceEval does not know about the read. When you change the test file, it still reuses old results, and you see old conclusions judged by the old test.

Read scoring files with loadText

  1. Put scoring files in the repository next to the evals, not in the starting project given to the Agent:
  2. Read them at the module top level of the .eval.ts. NiceEval needs to know the scoring file content before the run to decide whether this eval must rerun:
    A path string relative to the project root also works: loadText("evals/fixtures/react-datepicker/pr-6058/tests/run-tests.sh"). loadText accepts a URL object directly; you do not need to import node:url.
  3. In test(t), wait until the Agent is done, then write the tests into the Sandbox and run them:
    Repository tasks such as Terminal-Bench work the same way. When preparing the Agent’s starting environment, include only the public Fixture; do not put hidden tests or test scripts into the image, working directory, or prompt. After t.send(...) returns, upload the original files read by loadText at the module top level into the Sandbox, run the original test script, and check both the expected results and the anti-leak check script. That way, changing a hidden file triggers a rerun, and the Agent cannot see it while working.

Confirm that changing a test triggers a rerun

Change datepicker_test.test.tsx a little, then preview the run plan:
This eval should show execute, and other unaffected ones should show reuse. After confirming, drop --dry to run for real, then view the results in the terminal with pnpm exec niceeval show, or in the browser with pnpm exec niceeval view. For how to read the plan, see Rerun Only the Evals Affected by a Change. When you changed a test but it shows reuse, first check whether the file was read with fs.readFile.

Common mistakes

  • Read at module top level during NiceEval discovery. loadText, loadJson, loadYaml, and loadCriteria can live in Eval files or statically imported shared modules; inputs are registered for every reachable Eval. In a process that has never entered discovery, they only read and return, without registration, so ordinary unit tests can import shared modules directly. Later discovery in that process evaluates and registers the shared inputs again. Once the process has entered discovery, calls after discovery (including test(t) and lifecycle hooks) throw LoaderOutsideDiscoveryError. Its message includes the path and advice to move the read into a module-level constant.
  • Read files the Agent changed with t.sandbox, not loadText. Files in the Sandbox are this run’s results, different each time, and should not decide whether to rerun. Read them with await t.sandbox.readText(path), then pass the string to t.check.
  • When the scoring standard is structured data (such as a case lookup table), use loadYaml / loadJson; see Data-Driven Tests.