Evals / 02
LLM-as-judge
Using a model to grade another model's outputs is the only way evals scale. An uncalibrated judge is a random number generator with confidence. Calibration is the skill.
What it is
An LLM-as-judge is a second model call that scores an output against a rubric: did the response answer the question, cite the right source, stay in policy. The judge lets you run a 50-case golden set in minutes instead of an afternoon. Calibration is the step almost everyone skips: proving, with numbers, that the judge's grades agree with a human's before you let it grade anything unsupervised.
Why it exists now
Human review is still the primary evaluation method at most companies using LLMs, and it does not scale past a handful of cases. The teams that get past it do so with judges. But a judge is itself an LLM system, which means it fails the same ways your product does: rubric ambiguity, position bias, grade inflation on long confident answers.
This is now interview material. Teams that run evals ask one question that instantly sorts candidates: "how do you know your judge is right?" There is a real answer, and it has a number in it.
The 20-minute kata
- Take 20 real outputs from one task. Grade them yourself first, pass or fail, before any judge exists. Write your labels down. This is your answer key, and you cannot change it later.
- Write a judge prompt with a rubric of at most 3 criteria and 2 worked examples: one clear pass with the reason, one clear fail with the reason.
- Force the judge to output a verdict and a one-line reason. Binary or 3-point scale. Never 1-to-10; models cannot hold ten anchors and neither can you.
- Run the judge on the same 20 outputs. Compute simple agreement: matches out of 20.
- Read every disagreement. Most are rubric ambiguity, not model failure. Fix the rubric, not the judge model. Re-run.
Stop when agreement reaches roughly 85 percent or you can explain every remaining disagreement. Now, and only now, the judge grades new cases.
What good looks like
- The judge was graded against blind human labels before it was trusted. The agreement number is written down where the team can see it.
- The rubric has anchored examples in the prompt, not just criteria names.
- Verdicts are binary or 3-point, with a required reason string, so disagreements can be audited.
- A slice of judge verdicts gets human-audited on a schedule, because judges drift when models, prompts, or users do.
- The judge model is not blindly the same model that produced the output, and where it is, the team knows the self-preference risk and has checked for it.
How it's tested
"You said you use LLM-as-judge. How do you know the judge is right? What agreement rate did you measure, and against whose labels?"
The red flag they are screening for: a judge prompt written in an afternoon, grading production traffic since, agreement never measured. Confidence without calibration.