EvalSeal
Git-style receipts for LLM evals — repeated runs, flip rates, evaluator fingerprints, and verifiable CI gates
EvalSeal treats a single eval score as what it is: one sample of a random process. It runs each eval N times, seals exactly what graded it, and refuses to compare two runs that were graded differently. Published on PyPI as evalseal.
Why One Score Is Not Enough
Two models on 150 GSM8K problems, five runs each, same grading:
| gemini-2.5-flash | gemma-4-26b-a4b-it | |
|---|---|---|
| Accuracy | 0.976 | 0.961 |
| Items that flipped between runs | 1 of 150 | 9 of 150 |
A single run of each looks like a close race. The paired test on the 1.5-point gap is inconclusive, and EvalSeal says so rather than calling it a win. What the receipts do show is that one model answers the same way when asked again and the other often does not.
Key Features
- Repeated runs — A flip rate and Wilson interval per case, not one aggregate number.
- Evaluator fingerprint — The rubric, prompt template, judge model, sampling parameters, scorer, dataset, and case set are hashed together. If any of them change,
diffreportsnon_comparableand names the fields that moved. - Drift vs. variance —
evalseal driftseparates verdicts that moved because the grader changed, because the judge is a sampler, or because the model did. - Checkable receipts — A hash-linked ledger, optional Ed25519 signatures, artifact digests including the response cassette, and portable anchor files with RFC 3161 timestamp support.
- Pre-registration — Fix the suite, dataset, case ids, and repeat count before any number exists;
gate --preregthen reports each clause by name. - CI gating on comparability — A pull-request workflow replays a committed cassette with no API key, verifies the ledger, gates against a policy, and explains why the check passed or failed.
Tech Stack
Python · Typer · Pydantic · httpx · Ed25519 (cryptography) · RFC 3161 · GitHub Actions · PyPI