Structured Output and Tool Schemas

The model produces text. You want a record. In between sits the schema, and the quality of the schema is the quality of the contract between the agent and the code that consumes it.

This lesson is about that contract, and the surprise that tool use is exactly the same mechanism viewed from the host side.

What binding to a schema actually does

Most LLM APIs let you attach a JSON schema to a request and promise, in exchange, that the response will conform to it. The guarantee is stronger than "the model will try" — the provider constrains decoding so that only tokens compatible with the schema are allowed at each position.

That is the practical win. Code downstream of a schema-bound response does not need try/except around a JSON parse, does not need a regex to pull a number out of prose, does not need to be ready for the response to be prose at all. The branches that used to defend against shape disappear, and what is left is code that handles meaning.

const schema = {
  type: "object",
  required: ["intent", "entities"],
  properties: {
    intent: { enum: ["refund", "cancel", "escalate", "other"] },
    entities: {
      type: "object",
      required: ["order_id"],
      properties: {
        order_id: { type: "string", pattern: "^ORD-[0-9]{6}$" },
        reason: { type: "string" },
      },
    },
  },
};
 
// The response is guaranteed to parse, be an object, have `intent` in the
// enum, and have `entities.order_id` matching the pattern. The consumer
// branches on intent and nothing else.

The schema is where you write what you insist on. Required fields. Enums. Patterns. Minimums. Every piece of discipline you push into the schema is a branch the consuming code never has to grow.

A tool call is a structured output

The two look like separate features and they are the same feature. The schema of a tool call has a function name and a typed argument object; the model emits one; the host interprets it as a call rather than a data record. Every provider models it this way, and understanding the symmetry is what makes the failure modes feel familiar.

const tools = [
  {
    name: "search_orders",
    description: "Find orders by customer id. Returns up to 20.",
    input_schema: {
      type: "object",
      required: ["customer_id"],
      properties: {
        customer_id: { type: "string", pattern: "^cust_[0-9a-f]{12}$" },
        limit: { type: "integer", minimum: 1, maximum: 20 },
      },
    },
  },
];

A tool definition is a schema with a verb attached. Everything a tight output schema does for structured output, a tight input schema does for a tool. A loose limit is a loose limit in both cases, and a model that fills it with "twenty" instead of 20 breaks the consumer either way.

What a demo hides

The demo runs on inputs the FDE already handled. The customer's data has apostrophes, umlauts, legacy records with missing fields, and the 2019 import job that left country as an empty string instead of absent. Every one of those is a case your schema can either anticipate or be surprised by.

The specific discipline is to collect twenty real examples from the customer during discovery and run the schema against them before anything else. If one of the twenty does not validate, you have found a real constraint of the customer's data, and the schema is the right place to decide whether to accept it or reject it.

Retries make the median look fine

The usual reflex when a response violates a schema is to retry it with a reminder prompt. "Your last response did not match the schema. Please produce valid JSON matching this schema." It works more often than it should, and hides the thing you need to see.

A response that violates the schema once will often violate it differently on the retry. You pay for a second generation, a third if you are unlucky, and the tail of your latency and cost distributions sprouts a lump you did not budget for. The success rate graph is reassuring. The bill is not.

The durable fix is a tighter schema with the specific field that failed made required or patterned, and a sentence in the system prompt that teaches the model what the field is for. "Produce a confidence between 0 and 1 scaled by how much of the question your sources answered" is better guidance than "include a confidence field."

What to take away

Binding output to a JSON schema moves the obligation of shape from the consumer to the producer, and every piece of discipline you push into the schema — required fields, enums, patterns, minimums — deletes a branch of defensive parsing downstream. A tool call is the same mechanism with a side effect: a function name and typed arguments, validated the same way. The failure modes a one-session demo hides are unicode, optional fields, partial violations and silent retries — all of which are findable by running the schema against twenty real examples from the customer before you ship. And when a response violates the schema, tighten the schema and teach the model with a sentence in the prompt, instead of papering the tail with retries.

Next: prompt caching, the cheapest cost lever an FDE has in a conversation with a customer's finance team.