Observability and Incident Response

A deployment that nobody can see inside is not deployed, it is hidden. The moment the system handles a real user, you need to answer three questions at any moment: what is it doing right now, is it doing it well, and if something breaks tomorrow will the person responding to the page be able to tell what happened.

Those three questions are what observability is for. The fourth question — what do we do when the answer to one of them is "no" — is what incident response is for. This lesson is about both.

What a useful trace looks like

An agent request is a tree, not a line. The user sends a message; the agent reaches for a tool; the tool returns something; the agent decides to call another tool or go back to the model; eventually the agent produces an answer. A trace that collapses that tree into a single span throws away the information that matters.

The shape that works:

1. Step 1

   user_turn

   The outer span. Carries the user id, the session id, the model tier requested, and the total latency the user actually saw.

2. Step 2

   model_call

   One span per model invocation. Carries the model name, the specific version, the token counts in and out, the cost, and the latency to first token and to completion.

3. Step 3

   tool_call

   One span per tool. Carries the tool name, the arguments (redacted as needed), the status, the latency, and the first-level error message if it failed.

4. Step 4

   retrieval

   One span per retrieval, with the query, the top k, the top scores, and which documents were actually passed into the next model call.

With this shape you can answer "why was this turn slow" by walking the tree and spotting the span that took most of the time. You can answer "what did the agent see from retrieval" by reading the retrieval span's output. You can answer "did the agent call the write tool" by looking for a span with the right name. Without it, you are reading logs by hand.

Two attributes earn their place on every span. The model version, because a regression often correlates with a provider point release and without the version you cannot see the correlation. The session id, because the question "show me every turn this session had" is one you will ask the day a customer files a complaint referencing a specific chat.

The four numbers on the wall

A good production dashboard has many graphs, but the on-call engineer looks at four:

1. p95 latency per user turn. Not the mean. The mean is dominated by the fast turns; the p95 is where the users whose patience ran out sat. A p95 that drifts up by 20% in a day is the earliest signal of a problem that is not yet an outage.
2. Tool-call failure rate, per tool. Not an aggregate. A 40%-failing tool can be invisible in a 2% aggregate failure rate, and the aggregate is the number almost every team shows. Break it out.
3. Rolling eval score on a held-out set. Not just the CI eval. Replay a small basket of real-traffic-shaped prompts every N minutes against production and chart the score. This is the single metric most likely to catch a regression before a user complains, and it is the metric most teams do not run.
4. Cost per user turn. The dollar number per interaction, over time. Cost drifts up when prompts quietly grow, when retrieval starts returning more chunks than it needs, or when the model falls back to a more expensive tier. Reviewing it once a day is cheaper than finding out from finance.

Put these four on one page and pin it somewhere the whole team sees it. Add a second row for the long tail — error rates by type, token usage by tenant, model-version mix, any customer-specific SLO you have signed — and let the first row stay small.

What an incident actually looks like

An incident is in progress. A customer messages their account manager: "the assistant is giving wildly wrong answers in our billing workflow since this morning." The on-call engineer picks it up. What happens next, in order:

incident

1. >You typed: Pulling up the customer's traces from the last hour. Looking for the first turn that produced a wrong answer.

2. Claude ran: traces: filter session_id=cust-8821-billing-* since=1h

3. WhyFirst step: find the earliest bad turn. The exact minute it started is almost always the signal that tells you what changed.

4. Output: earliest failure: 09:42 UTC
   model_version at 09:42: claude-sonnet-5-5-20251107
   previous version in trace history: claude-sonnet-5-5-20251014

5. WhyCorrelates with the point release that landed at 09:30. First suspicion established, not confirmed.

6. >You typed: Roll this customer back to the pinned previous version. Then confirm the symptom goes away before investigating further.

7. Claude ran: rollout: pin model_version=claude-sonnet-5-5-20251014 for tenant=8821

8. Claude ran: replay: billing_eval_set_v3 against production (tenant=8821)

9. Output: eval score: 0.93 (baseline 0.94). Symptom resolved.

10. WhyMitigation first, investigation second. Now the customer is unblocked, you can diagnose at normal pace rather than under fire.

11. >You typed: Open a diff against the eval set between the two model versions. See which prompts flipped.

12. Claude ran: eval: diff baseline=20251014 candidate=20251107 set=billing_eval_set_v3

Three habits worth naming from that trace. Mitigate before you understand. The first move is to restore the previous known-good state. Confirm the mitigation. Measure it with the same eval set the production dashboard measures against, because "the user stopped complaining" is not the same thing as "the problem is gone." Investigate at normal pace. Once the customer is unblocked, the pressure is off and the diagnosis improves.

The rollback strategies that make this possible

Those habits only exist when the system is designed for them. Three strategies that cost nothing in normal times and save hours in incidents:

• Pinned model versions. Never let the application code just say "claude sonnet 5-5." Always pin a specific point version per tenant, and change it through a deliberate deploy. The default of "whatever the provider is pointing the alias at" is the version of this that bites.
• Blue/green prompts. A prompt change is a release. Ship a new prompt behind a flag, route a percentage of traffic to it, measure it on the same eval set the old one was measured on. The all-tenants-at-once prompt change is the thing you want to make impossible.
• Feature-flagged tools. Each tool the agent can call lives behind a flag you can turn off per tenant. A tool that starts failing becomes a safe degradation (the agent works without it) rather than an outage. Many incidents are usefully mitigated by turning off the one tool that just broke.

Whose runbook

The runbook is the thing the customer's on-call team reads at 2am, and that means the customer's on-call team owns it. Your job is to write it with them, to keep it correct as the deployment changes, and to run a tabletop exercise at least once a quarter so it is not the first time anyone reads it in anger.

A runbook that works has three parts: who gets paged for which symptom, what the first three steps are for each symptom, and when to escalate — including the direct line to your on-call. It does not have a hundred steps. If it does, you have written documentation, not a runbook.

What to take away

Observability for an agent is a tree of spans, not a line of logs: a span per user turn, per model call, per tool call and per retrieval, with the model version and session id on each. Four numbers belong on the wall — p95 latency per turn, tool-call failure rate per tool, a rolling eval score against a realistic basket of prompts, and cost per turn — and the eval score is the one most teams miss. An incident is handled by mitigating first, confirming the mitigation with the same eval set the dashboard uses, and then investigating at normal pace. Three cheap design decisions make that possible: pinned model versions, blue/green prompts, feature-flagged tools. And the runbook belongs to the customer's on-call team, written with them, exercised quarterly.

Next part: the customer-facing craft that wraps all of this — discovery, scoping a POC that ships, the interview loops at the top labs, a worked case study, and the capstone.