The Softmax Diaries

Rin

Apr 18, 2026

One function, infinite drama

Softmax turns logits into a probability distribution:

$\mathrm{softmax}(z)_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

The temperature $T$ sets the mood: low $T$ makes the model confident and spiky, high $T$ makes it hesitant and flat.

lightbulbTipexpand_more

Sampling with $T \approx 0.7$ is a good default when you want variety without chaos.

It’s differentiable, it normalises, and it plays nicely with cross-entropy. Of course it’s everywhere.