If using ChatGPT or an API in 2023, you might have followed the advice “turn the temperature down” when you wanted a more serious answer. That is going out of date. The following is a collaboration between me and Grok to lay this out simply:
What temperature was
Imagine the model is always choosing the next word from a long list of options, ranked from “very likely” to “weird but possible.” Temperature changes how adventurous that choice is.
- Low temperature: almost always pick the obvious next word. Answers feel steady, repetitive, “corporate.” Good when there is basically one right output, like extracting a date from a contract.
- High temperature: more willing to pick a less obvious word. Answers feel livelier but sometimes incorrect.
On older products you could often set this yourself.
What thinking level is
Newer models (Gemini 3, OpenAI’s reasoning models) do extra work before they talk to you. They draft a private scratchpad: break the problem into steps, check themselves, then write the answer you see. You don’t see the scratchpad even though you pay for it. That hidden draft is the “thinking.”
Thinking level (sometimes called reasoning effort) is not “how random should the next word be?” It is “how long is the model allowed to work on that draft?”
- Low / minimal: answer quickly and cheaply. Fine for “what’s the status of ticket 1842?”
- High: spend more time (and more tokens) on hard things — a gnarly spreadsheet, a multi-step plan, a tricky piece of code.
Vendors are pushing you off temperature
Google now tells Gemini 3 customers: leave temperature at the default (1.0). Don’t turn it down “to be safe.” On these models, turning it down can make the hidden draft get stuck — looping, worse math, weirder reasoning. The company would rather you pick a thinking level and write clearer instructions. This reminds me of how automobiles used to be simple enough for regular users to understand and customize the pieces. Now cars are full of computers.
Commercial labs do not publish every detail on
- Why 1.0 for general enterprise APIs
- How defaults were chosen from evals, safety, or customer mix
Some math for those interested:
Temperature is so often 1 because that is the “do nothing” setting, not because someone measured that 1.0 is the best personality for business chat. Zero is actually an intervention.
After the model scores every possible next word (those raw scores are logits), it turns the scores into probabilities with a softmax.
Temperature (T) sits in that step:
The probabilities are exactly what the model learned. You are sampling from its native distribution. So 1 is the unique value that means “don’t reshape the distribution.”
I had conversations with my co-author Josh Foster along the lines of, “why can’t we just set temperature lower to get better answers?” Here is Grok with the explanation:
Because “always pick the most likely next word” is not the same thing as “get the right answer.”
Temperature 0 (or as close as the API allows) is greedy decoding: at every step, take the single highest-probability token. That maximizes local confidence, not task correctness.
The next word is not the solution
The model is not scoring “this whole answer is true.” It is scoring “this token is a plausible continuation of the text so far.” Those come apart all the time:
- After “The capital of Australia is,” the top token might be “Sydney” because that collocation is common, not because the fact is right.
- In a long proof, the safe-looking next sentence can lock you onto a dead end.
Greedy traces get stuck
Once the model has written a slightly wrong step, every following “most likely” token is conditioned on that mistake. Temperature 0 has no way to try the second-best branch. You get:
- repetition (“as mentioned above… as mentioned above…”)
- on thinking models, the loops Google warns about: the scratchpad keeps re-checking the same step because that is now the highest-probability continuation
A little randomness is how the sampler leaves a bad local path.
Economist Writing Every Day 