Evals / 07
Cost-aware evals
An eval suite that costs too much to run stops being run. Evals only govern a system while they are cheap enough to run always, so the eval system has unit economics of its own to engineer.
What it is
Treating the eval pipeline as a system with a budget: knowing the cost of one full run, tiering the judging (a small calibrated model for pass and fail, a large model only for taxonomy tags and disagreements), caching results for cases whose inputs and prompt have not changed, and splitting cadence so pull requests run a cheap slice while the full suite runs nightly.
Why it exists now
The failure mode is predictable and everywhere: a team builds a thorough suite, a full run costs real money and twenty minutes, engineers start skipping it, and six weeks later the gate is decoration. The opposite failure is quieter: a five-case suite that is cheap, always green, and catches nothing. Between them is an engineering problem, and it has the same shape as any unit-economics problem: measure the cost, then attack the biggest line item without losing coverage.
The 20-minute kata
- Price one full run of your suite: cases, times calls per case, times average tokens, times model rate. Write the number down.
- Multiply by your team's monthly merge rate. That is the real bill for per-PR gating.
- Now cut it five to ten times without losing coverage: hash each case's inputs plus prompt and skip unchanged ones; route pass-fail judging to a small model you have calibrated; keep the large model only for cases where the small judge disagrees with itself or a tag is needed.
- Re-price. Both numbers go in the repo, next to the suite.
The goal is a per-PR cost so boring that skipping the gate never comes up.
What good looks like
- The cost of a full run is a number people know, not a shrug.
- Judging is tiered: cheap judge first, escalation on disagreement, expensive judge only where it earns its rate.
- Unchanged cases are free, because results are cached against a content hash.
- Pull requests run the must-pass tier and the affected slice; the full suite runs on a schedule.
- Eval spend is a tracked line item, like CI minutes, with a budget someone owns.
- Nobody has skipped the gate for cost reasons in living memory. That is the metric this microskill optimizes.
How it's tested
"What does one full run of your eval suite cost, and how often does it actually run?"
The red flag: not knowing. The second red flag: knowing, and the number being high enough that everyone quietly avoids running it.