Judge Rubrics and LLM-as-Judge

The eval suite from the previous lesson has a hole in the middle of it. For anything an LLM produces that is not a classification or a lookup, there is no straightforward way to say whether an answer is right. A customer-support reply either is or is not the kind of reply you wanted. A summary either is or is not grounded in the document. A code change either is or is not equivalent to what was asked for.

The way you score these outputs decides whether your eval suite is honest. This lesson is about doing it so that a second person — or a second model — would reach the same verdict.

What makes a rubric

Start with the hard test and work backwards.

That sounds obvious and it is enormously hard. Most rubrics in use are instructions that ask the rater to decide whether the answer is "good", "complete" or "accurate." Every one of those words hides a judgement, and different raters will make the judgement differently without noticing. The scores look rigorous and are not.

The way out is to decompose. Instead of "is this a good support reply," the rubric asks a list of yes/no questions a rater cannot disagree on:

{
  "criteria": [
    {
      "id": "addresses_question",
      "question": "Does the reply address the specific question the customer asked, as opposed to a related one?",
      "score": "0 or 1"
    },
    {
      "id": "grounded",
      "question": "Is every factual claim in the reply supported by one of the retrieved passages?",
      "score": "0 or 1"
    },
    {
      "id": "no_hallucinated_policy",
      "question": "Does the reply avoid asserting a policy that is not present in any retrieved passage?",
      "score": "0 or 1"
    },
    {
      "id": "tone",
      "question": "Is the tone consistent with the brand voice examples in the appendix?",
      "score": "0 or 1"
    }
  ],
  "pass_threshold": 4
}

Four binary criteria is more work to write than one five-point scale, and it is the whole trick. Every criterion is a question the rubric author has answered for the rater in advance. What is left to disagree about is small, and often a disagreement is itself a signal the rubric needs another criterion.

The weighted-sum versions people reach for — "clarity 1-5, correctness 1-5, helpfulness 1-5" — look more sophisticated and encode fewer decisions. A "3 for clarity" is a disagreement waiting to happen. A "yes" to "does the reply quote the warranty duration present in document D14" is not.

LLM-as-judge

Scoring four binary criteria across a dataset of a thousand examples with humans is possible, and occasionally necessary, and almost never what you want to pay for. LLM-as-judge — a model applying the rubric for you — is the practical answer when the scaffolding is right.

Three pieces make the scaffolding.

Structured output. The judge returns a JSON object with one field per criterion and an overall verdict. No free text, no "my reasoning is…" It is easier to see when the judge is wrong when the shape of the answer is fixed, and a structured answer is one you can parse into a number without a regex.

Few-shot calibration. Before the actual inputs, the prompt includes a handful of examples — three or four — showing what a "1" looks like on each criterion and what a "0" looks like. The examples do not have to be from the same dataset. They do have to agree on what the rubric means. Candidates often skip this step and end up with a judge whose idea of "grounded" is subtly different from the author's, which only surfaces after the scores are treated as truth.

A stronger model than the one being judged. The judge is being asked to notice when the output is wrong. A judge the same size as the subject can be flattered by a confident wrong answer, because the two share the same failure modes. A judge noticeably stronger has the capacity to be sceptical and is cheap enough compared to the cost of a bad release that the rule is almost always right.

The bias you will hit

Pairwise comparison — ask the judge which of two answers is better — is a useful shape and has one reliable failure mode. The judge prefers whichever option you showed it first. This has been measured enough times to treat as a certainty rather than a hypothesis.

The fix is boring and works. Randomise the order of A and B across the dataset, so position effects average out of the aggregate. If the sample is small, also offer a tie option so the judge has somewhere to put the pairs that are genuinely indistinguishable instead of defaulting to the first slot. Both together are better than either alone.

Scoring one answer in isolation — "rate this on the rubric" — does not have position bias in the same way, but it has a sibling: length bias. Longer answers get higher scores on average, on rubrics that do not specifically penalise padding. Adding an explicit "concise" criterion is not a cure, but it narrows the gap.

Calibration is not optional

The part people skip because it is annoying.

An LLM-as-judge setup is not validated by being internally consistent. The judge can be perfectly consistent and completely wrong — the two properties are independent. The only honest validation is to take a sample of outputs, have humans score them on the same rubric the judge uses, and compare.

Agreement is reported as a percentage on each criterion separately, because the aggregate hides where the judge is drifting. A judge with 95% agreement on "addresses_question" and 62% on "grounded" is a judge you can use for the first criterion and not for the second, and the only way to find that out is to measure them separately.

Every time the judge changes — a new model, a rubric edit, a calibration example added — the human agreement check runs again. This is why the calibration sample is kept small: it is run often, and a sample of fifty you actually get through weekly is more useful than a sample of five hundred you run once and forget to repeat.

What to take away

A rubric is a rubric when two reasonable people would reach the same verdict, which is not the state most rubrics are in — the shortest path there is binary criteria that remove the rater's judgement from the loop. LLM-as-judge is a reliable practice when the scaffolding is right: a stronger model than the subject, structured output, few-shot calibration ahead of the inputs, and randomised order on pairwise. Validation means calibration against a human-scored sample on every criterion separately — without it the judge is consistent rather than correct, and the eval suite built on top of it is measuring something nobody has checked.

Next: where the examples themselves come from, and how to wire the result into a gate that actually blocks a bad change.