EvalSeal

Git-style receipts for LLM evals — repeated runs, flip rates, evaluator fingerprints, and verifiable CI gates

EvalSeal treats a single eval score as what it is: one sample of a random process. It runs each eval N times, seals exactly what graded it, and refuses to compare two runs that were graded differently. Published on PyPI as evalseal.

Why One Score Is Not Enough

Two models on 150 GSM8K problems, five runs each, same grading:

  gemini-2.5-flash gemma-4-26b-a4b-it
Accuracy 0.976 0.961
Items that flipped between runs 1 of 150 9 of 150

A single run of each looks like a close race. The paired test on the 1.5-point gap is inconclusive, and EvalSeal says so rather than calling it a win. What the receipts do show is that one model answers the same way when asked again and the other often does not.

Key Features

  • Repeated runs — A flip rate and Wilson interval per case, not one aggregate number.
  • Evaluator fingerprint — The rubric, prompt template, judge model, sampling parameters, scorer, dataset, and case set are hashed together. If any of them change, diff reports non_comparable and names the fields that moved.
  • Drift vs. variance — evalseal drift separates verdicts that moved because the grader changed, because the judge is a sampler, or because the model did.
  • Checkable receipts — A hash-linked ledger, optional Ed25519 signatures, artifact digests including the response cassette, and portable anchor files with RFC 3161 timestamp support.
  • Pre-registration — Fix the suite, dataset, case ids, and repeat count before any number exists; gate --prereg then reports each clause by name.
  • CI gating on comparability — A pull-request workflow replays a committed cassette with no API key, verifies the ledger, gates against a policy, and explains why the check passed or failed.

Tech Stack

Python · Typer · Pydantic · httpx · Ed25519 (cryptography) · RFC 3161 · GitHub Actions · PyPI

GitHub PyPI