What Evaluation Even Means

The word "evaluation" has carried a lot of baggage into applied AI, and most of it is wrong. It is not benchmarking. It is not grading. It is not a compliance report. For an FDE, evaluation is a specific and modest thing: it is the measurement instrument that lets you make a decision. This lesson sketches what that means, so that when Part 6 of the course arrives and spends four lessons on it, you already have the shape right.

Evaluation is measurement

A thermometer is an eval. It does not grade the room, and it does not win a competition against other thermometers. It tells you what the temperature is, so you can decide whether to turn on the heating.

That framing rules out a lot of what people call evals. A one-off benchmark run is a measurement, but it does not decide anything: nobody is comparing every shipped change against it. A customer satisfaction report is a measurement, and it is too slow and too biased to decide anything on the timescale of a release. A live user-feedback dashboard is a measurement, and its signal is sparse enough that regressions hide in it for weeks.

The evals that pay for themselves are the ones that run on every pull request, on every model swap, on every prompt edit, and tell you whether the change made things better, worse or neither.

The three questions an eval answers

At the risk of oversimplifying something that gets four lessons later, almost every useful eval at an FDE engagement is answering one of three questions.

1. Step 1

   Did this change regress anything?

   Run the fixed suite before and after. If the pass rate drops on any slice, surface it in CI and block the merge. This is the gate every production system needs.

2. Step 2

   Does the new version beat the old on this slice?

   Run both against the same inputs, score the outputs with a judge rubric or ground truth, and report the win rate. This is what lets you compare models, prompts or retrieval strategies honestly.

3. Step 3

   How often does the model fail on this known-hard class?

   A small, deliberately hard eval set — the twenty examples where your system struggles. Running it tracks whether the hard cases are getting better, worse, or stuck.

The first eval gates releases. The second eval decides roadmap. The third eval tells you when a capability improvement has arrived at the hard slice, instead of only moving the aggregate.

Why cheap evals beat clever evals

Teams new to this build their first eval to be the most rigorous one. It takes an hour to run. They write it, use it twice, and never again — because it is too slow to fit into the loop of making changes.

The eval that earns its place is the one you run on every change. That usually means:

• A small labelled set, in the dozens to low thousands of examples.
• Automated scoring — a judge rubric, a schema check, a string match, a retrieval overlap metric — with no human in the loop for routine runs.
• A pass or fail that you can read in under a minute.
• Fast enough to run inside a CI job without doubling the pipeline time.

A deeper eval has a place: pre-release sweeps, quarterly capability reports, customer-visible numbers. But if the first eval you build cannot run in CI, you have built the second eval first, and the discipline will not take hold.

What the customer is going to ask

On a first customer call, evaluation almost never comes up by name. What comes up is a form of the same question: how will we know this is working? The FDE's answer is that the eval is the thing that answers that question.

discovery

1. >You typed: What tells you today whether your current support workflow is working?

3. WhyBoth are lagging signals. Neither of them would catch an agent getting silently worse at refund policies over a month. Name that.

4. >You typed: Both of those are valuable, and both are weeks behind what's happening. What I want to build in parallel is a fast check that runs on every change and tells us within minutes whether a change made the agent better or worse on the tickets you've already resolved.

6. >You typed: Give me a few hundred of the tickets your team has already closed in a specific category. We'll measure how often the agent arrives at the same resolution, and we'll watch that number move as we change the prompt, the retrieval, or the model.

7. WhyThat is the whole eval pitch. It is modest, it is cheap, and it is the single thing that keeps the engagement from running on vibes for six months.

The spine of the course

Part 6 of this course spends four lessons on evals — judge rubrics, dataset curation, regression gates, offline versus online. Everything between here and there assumes evals exist, because the moves that an FDE makes in prompting, retrieval, agents and deployment are all moves whose rightness or wrongness is only readable through an eval.

If you remember one thing from the foundations part, remember this: the eval is the measurement instrument that turns the rest of your work from opinion into evidence. The FDE who ships the measurement first ships the system faster.

What to take away

Evaluation is not grading — it is measurement, for the purpose of making a decision. The three questions evals answer in practice are whether a change regressed anything, whether a new version beats an old one on a slice, and how often the model fails on a known-hard class. Cheap evals that run in CI beat clever evals that run occasionally, and the customer-facing version of this is answering "how will we know this is working" with an eval, not a dashboard.

Next: prompt engineering and structured output, starting with the part most people skip — scoping the prompt before writing it.