Evals / 05
Offline vs online
Offline evals tell you whether a change is safe to ship. Online signals tell you what production actually thinks of it. Each one lies when asked to do the other's job.
What it is
Offline evaluation is everything you run before ship: the golden set, curated inputs, checkable outcomes, a controlled environment. Online evaluation is everything you watch after: resolution and escalation rates, thumbs, drop-offs, sampled judge audits of live traffic, A/B tests. The microskill is knowing what each can and cannot tell you, and building the pipe between them.
Why it exists now
A golden set is small and curated. Production invents inputs you never imagined, in volumes that surface the one-in-a-thousand failure weekly. Passing offline does not mean succeeding online. The reverse trap is worse: teams that watch only online metrics learn about regressions from their users, days late, through numbers that are noisy and confounded.
The two are a flywheel, not a choice. Offline gates the change. Online discovers the failures you did not know to test. Every online surprise becomes a golden-set case, and the offline suite grows where production actually hurt.
The 20-minute kata
- Pick one shipped LLM feature. Draw a two-column table: "checked before ship" and "watched after ship."
- Fill the left column with your actual offline evals, and the right with the online signals that actually exist today, including where each one lives.
- Mark the holes: an offline check with no online counterpart, and an online signal with no offline case behind it.
- Close one hole in each direction: define one online metric you can start collecting this week, and write one golden-set case born from a production surprise you remember.
Most teams discover one column is nearly empty. That column is the assignment.
What good looks like
- Every launch has both columns filled in before it ships, and someone named on the right-hand column.
- Online failures have a paved road back into the golden set, and it gets used the same week they appear.
- Online metrics are treated as proxies. A thumbs-up measures mood, not correctness, so it is paired with sampled judge or human audits of real conversations.
- Behavior changes ride an A/B or a staged rollout with guardrail metrics, not just the target metric.
- There is a written rollback criterion before launch, not a debate after.
How it's tested
"Your offline evals pass at 95 percent. What do you watch in the first week after launch, and what specifically would make you roll back?"
The red flag: no rollback criterion, or "our users will tell us" offered as the monitoring plan.