Evals / 06
Eval-driven iteration
The working loop of LLM engineering: baseline, one deliberate change, re-run, read the diff, keep or revert. The unit of progress is an explained diff.
What it is
A discipline for changing LLM systems on purpose. Before any change, you know the baseline score and its noise. Then you change exactly one thing: the prompt, or the retrieval, or the model, never all three. You re-run the suite and read the per-case flips, not just the aggregate. Then you decide, in writing: keep, with the cases it fixed, or revert, with what you learned.
Why it exists now
LLM systems invite fiddling. Tweak five things, eyeball two outputs, feel good, ship. Stochasticity makes it worse: run the same suite twice and the score moves on its own, so an unmeasured "improvement" is often noise wearing a costume. Eval-driven iteration is the difference between engineering and folklore, and it is becoming how teams review prompt changes: the pull request shows the eval diff or it does not merge.
The 20-minute kata
- Re-run your suite twice with no changes. The gap between the two scores is your noise floor. Write it down.
- Make the one change you have been meaning to make. One.
- Re-run. Write three lines: cases fixed, by name. Cases broken, by name. Aggregate move, compared against the noise floor.
- Decide: keep or revert, in one sentence that a teammate could audit later.
If the aggregate moved less than the noise floor, you learned that too: the change did nothing, and now you know instead of believe.
What good looks like
- The noise floor is known before anyone takes credit for an improvement.
- Case-level flips get read. A +3 aggregate hiding two broken must-pass cases is a regression, not a win.
- One variable per iteration. When the model and the prompt change together, nobody knows which one did it.
- Reverts are cheap and unremarkable, because prompts are versioned next to their evals.
- An iteration log exists, so the team compounds learning instead of re-discovering the same dead ends.
- Wins on the golden set get spot-checked against fresh cases, because a set you optimize against is a set you can overfit.
How it's tested
"Tell me about the last prompt change you kept and the last one you reverted. What did the diff show in each case?"
The red flag: no reverted change comes to mind. That means either nothing is measured, or everything ships.