Grade Hidden Tests: Include Criteria Files in the Cache Fingerprint

​

Steps

1. Keep the criteria files in the repository with the eval:
   evals/
     react-datepicker/
       pr-6058.eval.ts
     fixtures/react-datepicker/pr-6058/
       tests/datepicker_test.test.tsx   # hidden test
       tests/run-tests.sh               # test script
   
2. Read them at the module top level of the .eval.ts file:
   import { defineEval } from "niceeval";
   import { loadText } from "niceeval/loaders";
   import { commandSucceeded } from "niceeval/expect";
   
   const fixture = (p: string) =>
     new URL(`../fixtures/react-datepicker/pr-6058/${p}`, import.meta.url);
   
   const hiddenTest = await loadText(fixture("tests/datepicker_test.test.tsx"));
   const runTests = await loadText(fixture("tests/run-tests.sh"));
   
   A project-root-relative string works too: loadText("evals/fixtures/react-datepicker/pr-6058/tests/run-tests.sh"). loadText accepts a URL directly; you do not need to import node:url.
3. Write the files into the Sandbox and execute them in test(t):
   export default defineEval({
     description: "react-datepicker pr-6058",
     async test(t) {
       await t.send("Fix the misaligned changeMonth panel. Do not change the test files.");
       await t.sandbox.writeText("src/test/datepicker_test.test.tsx", hiddenTest);
       await t.sandbox.writeText("tests/run-tests.sh", runTests);
       t.check(await t.sandbox.runCommand("bash", ["tests/run-tests.sh"]), commandSucceeded());
     },
   });
   

​

Notes

• loadText must be at module top level. Cache reuse is decided before execution. Reading it inside test(t) happens too late, so NiceEval reports an error telling you to move it to the top level.
• Read Agent output with t.sandbox, not loadText. Files created by the Agent inside the Sandbox are evidence for this run and vary each time; they do not belong in the fingerprint. Read them with await t.sandbox.readText(path), then pass the returned string to t.check.
• Use loadYaml / loadJson when criteria are structured data such as a case table. See Data-driven Testing.