Offline vs Online Evals

Everything so far has been about evals that run on a fixed set of inputs you chose. They are cheap, they are deterministic (modulo sampling), and they catch almost every regression a change you can see will cause.

Almost every one is not every one. This lesson is about the other kind, and the incident category that is the only reason you need it.

The two questions

An eval answers a question. Which question depends on what you set it up to measure, and the two broad categories are different enough that teams who confuse them tend to build the wrong one.

Offline evals answer: did my change make the system better or worse on inputs I already have? The inputs are fixed. The model is fixed. The dependencies are mocked or pinned. Any change in the score traces to something you did. This is the eval that belongs in CI before a pull request merges, because it is the one that can block a bad commit.

Online evals answer: how is the system behaving on real traffic right now? The inputs are whatever shows up. The model is whatever is deployed. The dependencies are whatever they are, including the ones you did not know you had. A change in the score traces to anything in the world.

The two are complements, not alternatives. An FDE who ships with one and not the other is shipping with a known gap.

The incident only online catches

Here is the category that forces the issue.

Your system calls a vendor model behind an opaque endpoint. The vendor rotates the model to a newer version with a different name, or adjusts how it handles system prompts, or changes some undocumented default in the sampler. Your offline evals run green because the offline evals run against a fixed version of your pipeline and do not know the world moved. Your production answers quietly get worse — maybe a point on citation quality, maybe half a point on tone — and nobody notices until a customer does.

Variants of this happen all the time, not only with vendor model rotations. The retrieval index was re-sharded by the team that owns the storage layer. A content source rotated its format and the chunker now splits in a worse place. A dependency that normalises inputs was upgraded and strips a character class it used to keep. The system is unchanged in your repo and its behaviour has drifted.

The honest fact is that an offline eval cannot catch this. It is not a limit of the eval design — it is the definition. An eval that runs against a fixed pipeline cannot measure what the moving pipeline is doing. The online eval exists to close this loop, and nothing else does.

What an online eval looks like

Less romantic than the name suggests.

An online eval, in its simplest honest form, is: sample a fraction of production responses, run your rubric against them (with the LLM-as-judge setup from the last lesson), and track the score. Compare against a rolling baseline. Alert when it drops by more than the delta you set, the same way you would on a CPU metric.

That is the floor. Above it, the useful shapes are:

A shadow deployment runs the new version in parallel with the current one on real traffic, scores both, and compares. The new version is not served to the user; its outputs go into the eval pipeline. This is what lets you measure a change against actual distribution before you ship it.

A canary serves the new version to a small percentage of traffic — a single-digit percentage is normal — and scores the sampled outputs against the control. A failing canary rolls back automatically.

An A/B test is the formal version: two groups, a chosen metric, a sample-size calculation done ahead of time, and a stop rule that is not "when it looks like a winner."

1. Step 1

   Sample traffic

   Take a fraction of real responses (and in shadow/canary, pair them with the new version's response on the same input).

2. Step 2

   Score with the rubric

   LLM-as-judge from the previous lesson, with the same rubric the offline suite uses so the two are comparable.

3. Step 3

   Track and alert

   Rolling baseline, alert on delta drops. Per-slice breakdowns for tenants and categories that matter.

4. Step 4

   Close the loop

   A failing online check triggers a rollback if it is a canary, or feeds an incident if it is a steady state.

The A/B test, honestly

A/B tests are easy to run and hard to run well. Three failure modes are specific enough to name.

Peeking. You run the test and check the dashboard every day. On day two B is ahead by 1.4%, you ship it, and in week three the picture has reversed because the sample was never large enough to distinguish 1.4% from noise. The fix is a stop rule set before the test begins — a sample size or a time window — and no decision made before it is met. Sequential tests exist and have their own caveats, but the honest default is: set the stop rule, do not look.

Aggregate wins that hide slice losses. B wins overall and loses on three tenants. Aggregated, B ships. In production, three customers cancel. The fix is the same slice breakdown the offline gate uses, carried through to the online test. If the three tenants collectively represent ten percent of revenue, an aggregate win of one percent that costs you half of that revenue is not a win.

Metric mismatch. The customer cares about whether the right answer was given. The A/B test measures click-through on the first suggested result. The two are related and not the same, and what gets optimised is what gets measured. The A/B metric has to be close enough to the customer outcome that a lift on the former means a lift on the latter — and the way to check that is to score a sample with the rubric and confirm the two move together.

A worked example

A customer-support assistant has a well-tuned offline regression gate: 86% on a 500-item set, delta gate at minus two, slice gates on the top three tenants. Everything is green for weeks.

On a Tuesday, customer complaints rise. Support tickets with phrases like "it gave me the wrong number" tick up — not enough to page, enough to notice. Offline evals keep passing. A diff against main shows no code change in the agent, no prompt change, no index change.

The online eval — a nightly rubric-scored sample of production responses — is the one that caught it on Monday night. The "grounded" criterion dropped from 0.92 to 0.87 against the rolling baseline, the alert fired, and by Tuesday afternoon an investigation found the vendor had rolled out a new model snapshot that handled long system prompts differently. The offline suite, pinned to the old snapshot in its request, never saw it.

The incident makes two things obvious in retrospect that were obvious in advance to a team that had both. The offline eval was doing its job. The online eval was the only thing that could have caught this. And the next lesson's contract — explicitly pin vendor model versions where possible, and run continuous online checks where it is not — writes itself.

What to take away

Offline evals answer "did my change make the system better" — fixed dataset, fixed pipeline, deterministic, cheap, runs on every pull request. Online evals answer "how is the system behaving right now" — real traffic, real deployment, catches everything the fixed pipeline cannot see, including the vendor rotation and the quietly re-sharded index that offline evals are blind to by definition. A/B tests are the formal version and fail predictably on peeking, on aggregate wins that hide slice losses, and on metric mismatch; a stop rule set ahead of time and the same slice breakdown the offline gate uses are the defences. Online does not replace offline — the two answer different questions and the FDE ships both.

Next: deployment. Auth, data residency, audit — the part of the stack the customer security team spends the longest on.