Sampling, Temperature and the Honest Limits of Determinism

The sampling knobs on a language model are seductive because they feel like precision controls. They are not. They are the thin layer between the model's probability distribution and the single token it emits, and understanding what they actually do — and what they cannot do, no matter how hard you set them — is how an FDE avoids promising a customer a kind of determinism that is not for sale.

What temperature actually is

Every time a language model picks the next token, it is choosing from a probability distribution over its entire vocabulary. The model computes a score — a logit — for every possible token, and softmax converts those scores into a probability.

Temperature divides the logits before softmax. A temperature of 1.0 does nothing: the distribution is used as the model produced it. A temperature below 1 sharpens the distribution — the highest-probability token gets a larger share, the rest shrink. A temperature above 1 flattens it — unlikely tokens become more likely.

Top-p is the other common knob. It does not change the distribution shape; it clips the tail. top_p: 0.9 means "only consider the smallest set of tokens whose probabilities add up to 0.9, and sample from them." It is a safety rail — the model cannot pick a token that was 0.1% likely — and it tends to pair well with a slightly non-zero temperature.

The two knobs interact in ways that are hard to reason about, and the honest advice is to pick one default and move it deliberately. For most FDE work, that is temperature 0 for anything the eval scores against a ground truth, and temperature somewhere between 0.3 and 0.7 for anything the user reads.

What a seed does

Some providers expose a seed parameter. In principle it controls the random number generator the sampler uses, so the same prompt with the same seed should produce the same output. In practice, it buys you best-effort reproducibility within one provider, on one model version, over short windows.

Why determinism is not for sale

Here is the part that catches an FDE off guard on their first production incident. Two identical requests, same model, same prompt, temperature 0, same seed, run minutes apart — and the outputs differ by a sentence.

The reason is not an API bug. The reason is that the GPU kernels that compute the logits batch incoming requests together, and floating-point addition on a GPU is not associative — the exact sum depends on the order the hardware accumulates in, which depends on what else was batched that millisecond. Logits that differ by 1e-7 can tip a near-tie between two tokens, and from there the sequences diverge.

Providers can and sometimes do offer stronger reproducibility guarantees at the cost of throughput. The default is not that.

1. Step 1

   Prompt hits the provider

   Batched with whatever other traffic arrived at the same moment. The batch composition is not under your control.

2. Step 2

   Logits computed

   Floating-point order of operations varies with batch shape. Probabilities move by tiny amounts.

3. Step 3

   Sampler picks a token

   At temperature 0 it picks argmax. When two tokens were near-tied, the tie-break can flip.

4. Step 4

   One flipped token diverges the sequence

   The next-token distribution now conditions on a different history. By the end of a paragraph, the two responses can be noticeably different.

What to promise a customer instead

A customer on a compliance call will ask the determinism question in some form: "will the same input always produce the same output?" The wrong answer is "yes, if you set temperature and seed." The right answer is honest: no, and here is what we built instead so that your answers are reliable.

What the FDE brings to the conversation is an eval suite. Instead of promising that one input produces one output, you are promising that the distribution of outputs stays within measurable quality bounds. The eval runs on every deploy, flags regressions, and gives you a number the customer's risk team can read.

What "it worked yesterday" means

The practical consequence of all this is one sentence worth posting to every team channel: "it worked yesterday" is never evidence. Not of the model being correct, not of the prompt being right, not of the regression being fixed. A single good run is one draw from a distribution you have not characterised.

The move that replaces it is running the same thing fifty times and reporting the pass rate. That is cheap. The habit of doing it before you claim a fix is what separates an FDE conversation from a vibes conversation, and it is the muscle Part 6 of this course will train in earnest.

What to take away

Temperature controls how sharply the sampler favours the most-likely token, top-p clips the tail of the distribution, and seed offers best-effort reproducibility within one provider. No combination of the three gives you bit-exact determinism, because the GPU stack that computes the probabilities is not reproducible across batches. The FDE's answer to a customer asking about reliability is an eval suite, not a determinism promise — and the first discipline to internalise is that a single good run is not evidence of anything.

Next: what evaluation actually means as a measurement problem, and why the word means something narrower than most teams use it for.