A Worked Case Study

This lesson walks one FDE engagement from end to end. The customer is fictional and the specifics are made up, but the arc and the trade-offs are drawn from how real engagements go wrong and right. The point is to see the whole shape at once — discovery, scoping, build, evals, deployment and the incident that always arrives on day 14.

The customer and the ask

Northside Business Trust, a mid-market finserv with 2,300 employees and a compliance team of nine. Quarterly expense-report review had crept to six weeks per cycle, blowing through the regulator-mandated 30-day window. The VP of Compliance opened the first call with a sentence we will quote because it is the usual one: "We want an AI that flags expense violations."

Discovery

Four questions reshaped the ask in the first forty-five minutes.

discovery

1. >You typed: Walk me through one report, from submission to sign-off, as it is today.

3. >You typed: What do you find in that sample that actually causes action?

5. >You typed: So the violations are mostly about who signed off, not what was spent?

7. WhyThe named solution — flag violations by line item — was the wrong shape. The real workflow is auditing approval chains, which is a different build.

8. >You typed: Who on your team decides whether a review is good enough to close the quarter?

The exit-criteria conversation named the number: get quarterly review from six weeks to three, and reduce the false-negative rate on approval-chain violations below Priya's manual baseline. The named production owner was Priya. The written path to production was a short memo stating that if the POC hit those numbers by week four, compliance would move all quarterly review to the system by Q4.

The POC — four weeks

1. Step 1

   Week 1 — install measurement

   Priya labelled 80 historical reports with the correct compliance outcome. Baseline numbers for a fresh model run against those 80 were recorded before any prompt engineering. A thin end-to-end path — pull a report, ask the model for violations, log the result — was running by Friday.

2. Step 2

   Week 2 — reach signal

   Prompt and retrieval iteration against the 80-example eval, with Priya spot-checking misses live. Approval-chain violations climbed from 61 percent recall at baseline to 88 percent by the end of the week.

3. Step 3

   Week 3 — make it real

   Integrate against actual Concur data, not a copy. Expose two MCP tools: fetch_report and fetch_approval_chain. Hand the result to a human for the first week, not for approval but for shadowing — the agent flagged, Priya decided.

4. Step 4

   Week 4 — decide

   Final eval, memo, decision meeting with the compliance VP and the CFO in the room. Numbers hit, go decision, production path signed.

The MCP server, under review

The POC exposed a narrow MCP surface rather than a full workflow API. The reasoning was a question the FDE could answer out loud during the eventual security review: what can an attacker who compromises the agent do with this server?

security-review

2. >You typed: Two. fetch_report by ID returns the report and line items. fetch_approval_chain by report ID returns the chain of approvers with timestamps. Both read-only, both scoped to the compliance tenant.

4. >You typed: No. Writes would require a new review and a different trust boundary. The agent flags; a human writes the resolution.

6. >You typed: Only reports already in the quarterly audit sample, which is selected by Priya's team and injected as a scope. The tool rejects IDs outside that scope with an explicit error.

7. WhyThat last answer is the one the security review was looking for — a boundary enforced on the server, not trusted from the model.

Deployment shape

The customer's data could not leave their VPC. Hosted was off the table. On-prem would have taken six months of procurement for the compliance department. VPC deployment was the right shape and the one the FDE recommended — the model running against the customer's own cloud, the MCP server deployed inside the same VPC, logs and traces flowing to the customer's existing observability stack.

That decision was named on the path-to-production memo in week one, so the cost of VPC deployment was in the scope conversation before the POC started. No one was surprised in week four.

Day 14 — the incident

Fourteen days after go-live, Priya's team flagged that several of the agent's compliance explanations referenced "the 2026 Travel Policy Revision" — a document nobody had told anyone had been uploaded three days earlier by a well-meaning admin. The policy's wording was slightly different from the previous version, and the agent was now citing clauses that no longer existed in the way it was quoting them.

The agent kept working in the technical sense. No timeout, no error, no drop in latency. The outputs were plausible and wrong, which is the exact failure mode an eval harness catches that monitoring does not.

The response:

1. Step 1

   Contain

   Pin the retrieval corpus to the pre-upload snapshot while the incident was investigated. The agent kept running, now against the known-good corpus.

2. Step 2

   Diagnose

   Re-run the eval against the new policy document. Recall held, but citation accuracy dropped to 71 percent — the model was citing clauses by their old numbers in a document that had renumbered them.

3. Step 3

   Fix

   Add a citation-grounding check to the agent's output path: every cited clause had to appear verbatim in the retrieved chunks or the answer was returned without the citation.

4. Step 4

   Prevent

   Add a regression test to the eval harness: the test fails if the retrieval corpus changes without a new eval run. A new upload is no longer a silent event.

Iteration and what the memo looked like at the end of the quarter

Quarterly review dropped from six weeks to eighteen days. The false-negative rate on approval-chain violations fell below Priya's manual baseline by about a third. The one-page memo at quarter-end quoted those numbers, noted the day-14 incident with the fix, and recommended expanding the system to cover T&E policy violations in the next quarter — a scope decision the customer, not the FDE, drove.

What to take away

A real FDE engagement is a sequence of the moves this course has taught, one after another, each with the discipline that keeps the next one honest. Discovery reshapes the ask — the named solution was flagging line items, and the real workflow was auditing approval chains. Scoping sets the numbers and names the owner. The POC installs measurement first, reaches signal second, integrates third, decides fourth. The MCP surface is kept narrow enough that a security reviewer can draw the boundary on a whiteboard. Deployment shape is named in week one, not negotiated in week four. The day-14 incident will happen — plan for the version where the agent is technically fine and citing a policy that moved under it — and every incident ends with a test that would have caught it.

Next: the capstone exercise — the thing you build to show you can do all of that yourself.