Evals / 08
When metrics lie
When a measure becomes a target, it stops being a good measure. Everything earlier in this branch gives your metrics power. This node is about auditing them before they abuse it.
What it is
Goodhart's law, applied to LLM evals. The traps have specific shapes here: prompts overfit to the golden set, so the score climbs while the product stands still. Judges get gamed by the optimization loop itself, because most judges reward length, confidence, and answers placed first. Metrics get substituted, so optimizing thumbs-up quietly optimizes agreeableness. And aggregates mask slices, so the average rises while a rare, critical case class dies unnoticed.
Why it exists now
The moment evals enter CI and dashboards, they acquire power over what ships and who looks good. Anything with power gets gamed, sometimes by people, more often by the optimization loop nobody watched. The discipline that made your system measurable now needs an immune system. This is it.
The 20-minute kata
Two audits, ten minutes each:
- Hold-out check. Write five brand-new cases nobody has optimized against. Run them. If they score meaningfully worse than your golden set, the suite has been overfit and its score is inflated. The gap is the size of the lie.
- Length test. Take five judged outputs. For each, make a wrong-but-verbose variant and a terse-but-correct variant. Ask the judge to pick. Every verbose-wrong preference you find is judge bias currently steering your system.
Write both findings down where the next person changing the metric will trip over them.
What good looks like
- A hold-out slice exists, and it rotates into the golden set so no case stays optimizable forever.
- The judge is audited on a cadence for length, position, and self-preference bias, not just calibrated once and trusted forever.
- Slices have their own minimums. An aggregate is never allowed to speak for a must-pass segment.
- Metric definitions are reviewed like schema changes, because changing what you count changes what people build.
- Eval review has a standing question: "what would game this?" asked while the metric is designed, not after it was.
- When a metric is caught gamed, the fix is the metric, not a lecture to the people who optimized it. Incentives did what incentives do.
How it's tested
"Your eval score went from 78 to 91 in a month. What would convince you the product actually got better?"
The red flag: satisfaction with the number itself. No hold-out story, no judge audit, no fresh-case check. A score with no skeptic attached is marketing.