Evals / 04
Regression gates
An eval that runs and an eval that can stop a merge are different tools. The gate is the moment your evals stop being a dashboard and start being a decision.
What it is
Wiring the golden set into CI so that any change to a prompt, a retrieval pipeline, or a model runs the evals, and the result decides what happens next: merge, warn, or block. A working gate has two tiers. Must-pass cases are contracts: safety, money movement, policy, anything where one failure is one too many, and any regression blocks. Score cases measure quality in aggregate, and a drop beyond a set threshold warns or blocks.
Why it exists now
Prompts change weekly. Models get swapped underneath you. Three engineers touch the same system prompt in one sprint. Without a gate, evals are theater: everyone nods at the dashboard and ships anyway, and the question "did we get worse?" is answered by customers. With a gate, it is answered by the pipeline, before merge.
The gate is also what makes a model upgrade boring, in the good way. Swap the model, run the suite, read the diff. That is the whole migration story, and it only exists if the gate does.
The 20-minute kata
- Take your golden set and split it into two tiers: must-pass (the contract cases) and score (the quality cases). Be stingy with must-pass; every case in it can block a ship.
- Write a script that runs both tiers against the current prompt and exits nonzero if any must-pass case fails, printing the failing case, not just a number.
- Make it print the score diff against the last saved baseline.
- Wire it as a CI step or a pre-commit hook on the prompt file.
- Ship one real change through it.
The first time it blocks something real, the team stops debating whether evals are worth it.
What good looks like
- Two tiers, contract and quality, not one blended number that hides a refund failure inside a good average.
- A must-pass failure prints the case that failed. Engineers debug cases, not scores.
- Runs are cheap and fast enough to run on every pull request, with sampling or caching where needed.
- The baseline is pinned per branch, so diffs compare like with like.
- A broken gate fails loud. A gate that silently skips is worse than no gate, because everyone believes it ran.
- An override path exists for emergencies, and using it leaves a paper trail.
How it's tested
"What happens, mechanically, if someone merges a prompt change today that breaks refund handling? Walk me through it step by step."
The red flag: "we'd catch it in staging," which means a human might notice. Second red flag: evals run nightly and nobody is on the hook to read them.