Field guide · temperature

How it picks the next word.

A model doesn't know what comes next. It has odds for every possibility, and one dial decides how boldly it bets. We opened up GPT-2 and turned the dial ourselves.

Scroll to follow the river ↓

The sentenceGPT-2 small · 124M parameters

What comes next?

Step 1 · raw scores (logits)top 12

Every piece gets a score.

The dialtemperature

1.0

chance ∝ e^{score ÷ T}

Hinton, Vinyals and Dean: Neural networks typically produce class probabilities by using a softmax output layer that converts the logit, z_i, computed for each class into a probability, q_i, by comparing z_i with the other logits: q_i = exp(z_i/T) / sum over j of exp(z_j/T).

Hinton, Vinyals and Dean: where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.

Holtzman et al. Figure 4: token probabilities for the phrase I don't know. repeated 200 times; the probability rises with each repetition.

Holtzman et al.: The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.

Holtzman et al.: To address this we propose Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution.

Fig. 04 · five settings, 30 promptsGPT-2 small

Cold loops. Hot rambles.

OpenAI API reference: What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both.

Anthropic API reference for temperature, marked deprecated: models released after Claude Opus 4.6 accept only temperature 1.0.

Take this with you

The model gives odds. The dial decides how boldly it bets.

Turn it down and it plays safe, sometimes so safe it repeats itself. Turn it up and it takes long shots, sometimes into nonsense. Nothing about what the model knows changes. Only how it chooses.