How an eval runs
An Eval defines a task, assertions, and needed Fixtures. An Experiment selects Evals and declares the Agent, model, concurrency, budget, and other run conditions. An Adapter connects the system under test; a Sandbox provides an isolated place to execute work. The Runner creates one Run for every selected Experiment. A Run lists its expected slots, which define the denominator for that run. Each slot has at most one Member. A Member identifies the Attempt occupying its slot. An Attempt isorigin when it belongs to its current slot; it is reference when a Member adopts an Attempt from another Run. An Attempt is stored once in its origin Run, and a Member that refers to it does not copy business facts.
Record stores published facts
A Record lives in.niceeval/record/. It is an opaque, portable fact dataset that can be copied as a whole or put in Git. Application code does not read its internal structure or write its files.
Its Core stores Runs, Members, Attempts, and their exact relationships. Business facts such as Verdicts, Usage, assertions, and diagnostics live in owner-local Attachments.
After a writer finishes a Run’s Core and Attachments, it creates complete. A directory without that marker is not a published fact; readers ignore it and report an incomplete-run warning. Run another Invocation to produce a different result.
Sample determines comparison scope
Without a locator or--run, show and view select every result whose identity still matches the current project. A locator or --run selects historical facts. The internal host forms a pure AnalysisSample from the frozen Record; a reader or live handle is not public API.
The project-current selection scans every published Run. Using Evaluation and Eligibility identity facts, it keeps every slot whose Experiment, Eval, attempt ordinal, evaluation kind, input identity, and config identity still match the current project. It does not select one Run by time.
An explicit --run selection preserves each named Run’s complete expected-slot denominator; project-current establishes its current denominator from matching slots. Each selected slot is included, not-recorded, core-invalid, or excluded. After selection establishes that framework, a problem in another business Attachment cannot silently shrink it.
Projections and Reports organize selected facts
The official opaque projector turns internal facts into typed views. A Projection aligns those values with the Sample to form aProjectedSample. Calculations use those values for pass rates, costs, or diagnostic distributions; Pages and PageFamilies organize them into report pages.
One Report produces one immutable ReportExecution. It retains no reader, path, or deferred I/O. show, view, and static export consume that execution only.
When view detects a change, it builds the next execution. Only a complete rebuild replaces the result on screen; a failed rebuild keeps the last-good execution and shows the problem.
Deliver a Report
niceeval show presents a Report in the terminal. niceeval view presents it locally in a browser. niceeval view --out <directory> produces a self-contained static report site. Pass one or more explicit --run values when you need a fixed historical scope.
A static report site opens offline. Its browser reads only the export directory, never the network or source Record. See Publish a Static Report for publishing steps.
Continue reading
- Runner: Understand Invocations, Runs, and current-process feedback.
- Viewing Results: Select a Sample and inspect a Report.
- Rerun and Carry Results: Understand carrying and accepting Attempts.