Dataset Curation and Regression Gates

Judge rubrics tell you how to score an output. This lesson is about what the outputs are of — the inputs you score the system on, where they come from honestly, and how the score they produce is wired into something that actually stops a bad change from shipping.

Where the examples come from

Four sources, each with a specific job, and only together an honest set.

Production samples. The best raw material you have — real inputs the system saw in the real environment, with the problems the real environment contains. They cannot be sampled through the system you are about to evaluate, which is the single most common mistake. If the retriever picks what gets logged, your "random" sample is a sample of what the retriever can already handle, and the new retriever will look worse on it than it actually is. Sample at the point of input, before any component decides anything.

Customer-provided ground truth. When the customer has a prior system or human-written answers, those are gold. They anchor the eval to what the customer already considers correct, which is the only standard that survives the handoff from POC to production. Ask for them in the first discovery call, not the week before launch.

Hand-curated edge cases. The cases you have seen the system get wrong once and never want it to get wrong again. Each one is a line in the eval set. New ones get added as fast as they turn up. A set of two hundred of these is one of the most valuable artefacts an FDE builds on a customer site, because the set itself encodes what the team has learned the system can fail on.

Adversarial samples. Inputs authored to stress a specific behaviour — prompt injection attempts, ambiguous grounding, out-of-scope requests. These exist to prevent regressions in categories the real traffic may be thin on today but which the next integration will light up.

Leakage, and how to find it

The failure that quietly inflates every eval score is also the easiest one to overlook, because it hides in the part of the pipeline nobody wants to re-check once it works.

Leakage is when an example in the eval set is one the model has already seen — because it was in the fine-tuning corpus, because it was in the few-shot examples, because it is paraphrased from a document that was in the retrieval index during training. Every form of leakage makes the score look better than the system is, and every form of leakage makes the score less useful the moment the model meets a real input.

Detecting it is tedious and tractable. For each example in the eval set, check that no near-duplicate appears in any training or fine-tuning dataset. A near-duplicate means embedding distance under a threshold, not just a string match — a paraphrase leaks as effectively as a verbatim copy. The check is cheap to run on every new version of the dataset, and it is the one check that pays for itself the first time it finds something.

Where it finds leakage the response is binary. The example comes out of the eval set. If the leakage was systematic — someone fine-tuned on a corpus that overlapped your eval source — the scores from that run get retracted, not adjusted, because an adjusted number is a number nobody knows how to interpret and will be cited next week as if it were clean.

Synthetic data with asterisks

LLMs are tempting as eval-set augmenters. The temptation produces honest value and a specific trap.

Synthetic examples are useful for expanding coverage of failure classes that are rare in your real data. You know your agent has a bad time with multi-turn clarifications, you generate thirty examples of multi-turn clarifications, you add them to the adversarial slice. Good.

Synthetic examples are not useful for establishing floor performance. Their distribution is the distribution of the model that generated them, which is the thing you were trying to measure the system against, which is the loop you did not want to close. A score of 94% on a synthetic set tells you how the system does on examples that look like the ones a model generated. It does not tell you how it will do on Monday.

The rule that holds up: synthetic data augments, never replaces. If you cannot ethically collect real inputs because of privacy, that is a problem to solve with data-handling contracts, not with a model prompt.

The regression gate

The score matters because something is wired to it. Without that, see the previous lesson.

A regression gate lives in CI, runs on every pull request, and has three levers it can pull.

A pass threshold. The absolute minimum the aggregate score has to clear to merge. This is the cheapest gate and the least useful one once the system is working — an improving system blows past the threshold and the gate stops noticing anything.

A delta gate. A ceiling on how much the score is allowed to drop from the current main-branch baseline. Typically the absolute drop is set to zero on the aggregate and a small negative (like minus two points) on individual criteria, to allow for noise without allowing for silent regression. This is the gate that catches "the prompt change that helped most answers but broke citations."

Slice gates. Thresholds on breakdowns of the score — per tenant, per language, per category. A slice gate is what catches "we went up overall but dropped ten points on the healthcare customer," which the aggregate would have merged. For any customer whose contract depends on a specific behaviour, their slice is a separate gate.

1. Step 1

   PR opened

   CI runs the system against the regression set, applies the rubric, reports a per-criterion and per-slice breakdown.

2. Step 2

   Gate evaluates

   Pass threshold, delta against main, and slice gates each run. A failure on any one blocks the merge.

3. Step 3

   Merged change

   The score for the new main is written to the baseline. The next PR's delta compares against this value, not the one before.

4. Step 4

   Nightly online check

   The next lesson's topic — the gate is not the whole picture, only the part that fits before the release.

The override you want

The gate has to be overridable or it will be removed the first time it blocks a release the team believes in. The question is what the override costs.

The override that works: a merge-despite-gate requires a comment on the pull request naming which criterion was dropped and why, and the override is logged. A weekly review reads the overrides. If the same criterion is being overridden every week, that is signal — either the rubric is wrong, or the system has regressed in a way the team has accepted, and the baseline should be lowered deliberately rather than continuously defeated.

An override that just takes one click, with no explanation and no log, is not an override. It is a gate that stops nothing.

What to take away

An eval set is honest when it draws from production samples taken at the point of entry, customer-provided ground truth, hand-curated edge cases and authored adversarial examples — four sources, four jobs, and only together a set that tells you what you need to know. Leakage is the failure that inflates every score and hides in the part of the pipeline you trust; cheap embedding-distance checks catch it before the score is treated as truth. Synthetic examples augment coverage and never replace real inputs for floor performance. The gate itself is a pass threshold, a delta gate against main, and slice gates per tenant and per category; the override is logged and reviewed, which is what makes the gate survive the first release it blocks.

Next: the one incident category this entire chapter cannot catch — the silent degradation that only shows up in real traffic.