Why Evals Are the Spine

Every POC an FDE joins already has prompts, retrieval, some tool calls and a demo that sort of works. What separates the ones that ship from the ones that stall is not more of any of those things.

It is an eval suite.

The mechanism, not the model

The honest definition of an eval suite is small and awkward: it is the mechanism by which you can be wrong on purpose. You write down what the system should do on a set of inputs you chose, you run the system against them, and a number comes out. Change anything — the prompt, the retriever, the model, the chunking — and the number changes. The direction of the change is what lets you do the next thing.

Without it, improvement is a negotiation between whoever has the most recent anecdote. With it, improvement is a sequence of commits each of which either moved the number or did not.

The round candidates lose

If you read enough write-ups of the OpenAI FDE loop you notice a pattern. The candidates who get far almost all stumble in the same place — and it is not the applied-AI case study, and it is not the coding bar. It is the moment the interviewer says, "okay, how would you know this is working?"

The failure is one of sequencing. The candidate can describe an eval suite when the question is open. They can even sketch a rubric. What they do not do is build it first. Their worked example starts with a prompt, moves to a retriever, bolts on an agent, and then — in the final two minutes — allows that of course you would write some evals.

An FDE who has done this on a customer site a few times does the opposite. They build the eval harness on day one with ten hand-written cases and a rubric that fits on a napkin. Everything after that is measured against it. When the retriever changes, the number moves or it doesn't. When the model changes, the number moves or it doesn't. The conversation changes shape: you stop arguing from screenshots and start arguing from deltas.

The loop screens for this because the role demands it. An FDE lands in a room where nobody has agreed on what good looks like, and the first week's job is building the thing that lets them agree.

What it buys you

Two things that are hard to get any other way.

The first is the right to say "this change is better." Not to believe it — anyone can believe it. To say it to a customer with evidence that holds up when their technical lead asks how you measured. "The regression suite gained four points on retrieval-grounded answers and did not lose anything on the twelve adversarial cases" is a sentence the room moves on from. "It feels better" is a sentence the room sticks on.

The second is the right to deploy on Friday. This sounds like a joke and is not. A team without a regression gate deploys on Monday because they want five working days to respond to whatever breaks. A team with one deploys on Friday because the gate is what noticed the thing that would have broken, and the release is a routine action. Every change the gate blocked is one that did not happen in front of a customer. Every change it let through is one you can leave alone for the weekend.

The stage you need to get to is the last one, and the shortest path is to skip the middle one. A dashboard is a comforting intermediate that usually turns out to be the final state, because wiring the gate is harder than wiring the dashboard and the pressure to do so is weaker. Resist it.

The anti-pattern to name out loud

The failure mode has a specific shape: evals as a demo dashboard rather than a gate in CI. It looks like success. There are charts. There are weekly numbers. There is a channel where someone posts the score. The one thing there is not is anything that stops a bad change from merging.

The test is honest and small. Open the last five pull requests. Was any of them blocked by the eval? If the answer is no, and none of them deserved to be, fine. If the answer is no, and at least one of them did, you have a dashboard, not a suite. Spend the two days to wire the gate before you spend the two weeks tuning the next thing.

What this course does next

The next three lessons are the mechanics. Judge rubrics and LLM-as-judge is how you score an output with a judgement that is reproducible. Dataset curation and regression gates is where the eval set actually comes from, and how you wire it so that a bad change cannot land. Offline versus online is the two different questions evals can answer, and the incidents that only one of them catches.

Treat this part of the course the way the loops do: as the thing that decides whether you are an FDE.

What to take away

An eval suite is the single investment that separates a POC that ships from one that stalls, because it is the mechanism by which every later change can be shown to be an improvement. Candidates lose the OpenAI loop not because they do not understand evals but because they build them last — an FDE builds them first. A dashboard is not a suite: the test is whether a bad change is ever blocked from merging. The point of the next three lessons is the gate, not the chart.

Next: judge rubrics and LLM-as-judge — how you produce a score for an output that two people would agree on.