groundcheck — Evaluation Harness for RAG Systems

groundcheck — Evaluation Harness for RAG Systems An architecture diagram generated by Archify. Golden Set v3 · 39 questions · sha256 · Architecture component Golden Set v3 39 questions · sha256 Composition Check · negative-control ratio · Architecture component Composition Check negative-control ratio groundcheck CLI · one dependency · Architecture component groundcheck CLI one dependency Run Orchestrator · observes, never scores · Architecture component Run Orchestrator observes, never scores Adapter · one contract, any RAG · Architecture component Adapter one contract, any RAG System Under Test · shopfloor, or scripted · Architecture component System Under Test shopfloor, or scripted Run Record · what was observed · Architecture component Run Record what was observed Deterministic Checks · true / false / unknown · no model participates — set operations and normalised comparison Deterministic Checks true / false / unknown Retrieval Metrics · set operations · no model participates — set operations and normalised comparison Retrieval Metrics set operations CI Gate · fails the build · no model participates — set operations and normalised comparison CI Gate fails the build Per-Category Report · no single global number · no model participates — set operations and normalised comparison Per-Category Report no single global number eval load + verify sha256 counts the mix each question question answer + citations re-evaluated later retrieved sets per category recall@k · MRR · P@k exit code no model participates — set operations and normalised comparison Legend Frontend Backend Database Security External

A typo must not pass

  • • An unknown key in the golden set is a hard error, never a warning
  • • Write 'forbiden_numbers' and the check that catches the split-table bug just stops running
  • • The report stays green and the published metric quietly becomes a lie

A digit glued to a letter is not a number

  • • E-114 does not contribute 114, nor M24 the 24
  • • Extract them and the correct answer 'alarm E-114 indicates overtemperature' scores ungrounded
  • • These rules came from labelling the golden set and watching it fail, not from imagining cases

What it refuses to do

  • • A run stores observations only — a filler zero in an eval JSON gets copied into a README
  • • The report reloads the suite and refuses to report if its sha256 moved
  • • 23% of the golden set is negative controls: a system that always answers would score perfectly