Tokens, Context and Cost

An FDE spends a large share of the first customer call talking about three numbers: what a token is, what the context window holds, and what the invoice is going to look like on day thirty. This lesson is about getting the mental model right, because every later chapter — retrieval, agents, evals — re-enters this same conversation from a different direction.

What a token actually is

A token is not a word. It is a subword fragment produced by the model's tokenizer, which is a lookup table trained on text. Short common words map to one token: "cat", "and", "the". Longer or less common words are split: "tokenization" is usually "token" + "ization", "deployed" is often "deploy" + "ed". Code, URLs, non-English text and whitespace all tokenize differently than English prose, and the ratio drifts.

Why that matters on a call: a customer who reads "128k context window" as "128,000 words" has absorbed a number that is off by a factor of about 1.3, and in Chinese, Japanese or Korean the gap can be far worse. The habit to build early is counting in tokens, not words, and quoting both when you describe a document.

What the context window holds

A context window is the total number of tokens the model can read and write in one request. It covers both ends: the input you send and the output it generates. A 200k-token window with a prompt of 190k tokens leaves the model room for only 10k tokens of reply, and a prompt of 199.5k tokens may refuse to respond at all.

The split between the two is not symmetrical in cost either. Providers typically price input tokens and output tokens separately, with output costing three to five times more per token, and the reason is hardware.

Carrying this with you means two things on a design call. First, a long prompt with a short answer is a prefill-dominated bill, and prompt caching (the next chapter) is the lever that moves it. Second, a short prompt with a long answer — a draft, a summary, an agent plan — is a decode-dominated bill, and the lever there is choosing a smaller model or capping the length.

From a price list to a unit economic

Vendor pricing pages read as abstract until you convert them into per-customer numbers. The conversion is arithmetic, and the FDE should be the person in the room doing it on a whiteboard.

1. Step 1

   Count the tokens of one request

   Measure the real prompt, including any retrieval, templates and tool definitions. Count the output tokens on a representative response. Do not estimate.

2. Step 2

   Multiply by price per million, for each side

   Input tokens times the input price. Output tokens times the output price. Add them. That is the cost of a single request.

3. Step 3

   Multiply by expected traffic

   Requests per active user, times active users, times billing period. Now you have a monthly number for the customer to react to.

4. Step 4

   Compare against revenue, not against the price list

   A request that costs twelve cents is cheap in one business and lethal in another. The honest comparison is against the per-user revenue, not against the vendor's cheapest tier.

The failure mode to watch for is quoting the vendor's cheapest tier as the price. A demo that runs the small model for a few hundred requests is not the same system as a production deployment that hits the frontier model for a few hundred thousand, and the invoice will read like a surprise if the day-one conversation hand-waved it.

What the customer actually wants to hear

Customers do not want a benchmark walkthrough. They want three numbers: what each request will cost, what the monthly bill will be at their traffic, and where the lever is if that number is wrong. An FDE who can produce all three in a notebook on a shared screen has done something a slide deck cannot do.

The lever almost always lives in one of four places. The model (smaller is cheaper, and the eval tells you whether it is still good enough). The prompt (shorter is cheaper, and retrieval often replaces it). The cache (caching the stable prefix is the single largest latency and cost win for most agent workloads). The output cap (capping tokens with a stop condition is a line of code and sometimes shaves a quarter off the bill).

What to take away

A token is a subword fragment, and the four-characters-per-token rule only holds for English prose. The context window counts input and output together, and the two have different cost curves because one is parallel prefill and the other is sequential decode. Pricing on a vendor page is not the number the customer cares about — the number they care about is the monthly bill at their own traffic, and the FDE's job is to build that figure in front of them and name the levers that move it.

Next: the model families, and the discipline of choosing the cheapest model that still passes your eval.