Cross-entropy loss and perplexity explained

Cross-entropy loss measures how much probability a model assigned to the correct answer. It rewards confident correct predictions, penalizes confident mistakes sharply, and becomes perplexity when exponentiated for language-model evaluation.

What is cross-entropy loss?

For a single labeled example, cross-entropy loss is the negative natural logarithm of the probability assigned to the true class. If the model assigns probability 0.9 to the correct answer, the loss is −ln(0.9) ≈ 0.105. If it assigns 0.1, the loss is −ln(0.1) ≈ 2.303. Confident mistakes cost much more than uncertain mistakes.

For a dataset, average the individual losses. Lower is better when models are evaluated on the same examples, labels, and loss definition. Unlike accuracy, cross-entropy uses the full probability distribution, so it can distinguish a cautious correct prediction from a confident correct prediction.

What is the cross-entropy formula?

Here, y is the true binary label, p is the predicted probability of the positive class, and yₖ is 1 for the true class and 0 for the others. In the multiclass form, only the probability assigned to the true class contributes when labels are one-hot.

The natural logarithm produces loss in nats. If you divide by ln(2), you get bits. The base changes the unit and scaling, not the ranking of models evaluated on the same data.

Worked example: four correct-class probabilities

Suppose four examples have correct-class probabilities 0.8, 0.6, 0.9, and 0.7. We calculate one loss for each example, add them, and divide by four.

The mean cross-entropy is approximately 0.2990 nats. In bits, 0.2990 ÷ ln(2) = 0.4314 bits. The cross-entropy calculator recomputes the values from a probability list.

Notice which example contributes most: the 0.6 probability produces a loss of about 0.5108, more than the 0.9 example’s 0.1054. If one example received a probability close to zero, its loss would dominate the average. That sensitivity is useful when calibration matters, but it also makes data errors and mislabeled examples influential.

How is perplexity related to cross-entropy?

Perplexity is the exponential of mean cross-entropy when the loss uses natural logarithms:

For the example above, perplexity = e^0.2990 ≈ 1.3485. In a language model, perplexity is often described as the model’s average uncertainty or the effective number of equally likely choices, but that interpretation is an intuition rather than a literal count of words the model considered.

If loss is measured in bits, use 2 to the power of the mean loss instead: perplexity = 2^(loss in bits). The result is the same because 2^(loss/ln2) = e^loss. Always check the logarithm base before converting.

Perplexity is most useful for comparing the same tokenizer, dataset, language, and evaluation protocol. A lower perplexity on one corpus does not prove a model is better for your application. Different tokenizers change the number and identity of prediction events, so token-level perplexity is not automatically comparable across tokenizers.

How do probabilities and logits fit together?

Models usually produce logits, unnormalized real-valued scores, rather than probabilities. A softmax converts a vector of logits into probabilities that sum to 1. Cross-entropy is normally implemented from logits with a numerically stable log-softmax operation rather than by calculating probabilities and then taking a logarithm manually.

For binary classification, a sigmoid converts one logit into a probability. A binary cross-entropy implementation that accepts logits can avoid numerical problems when the logit is very large or very negative. In multiclass classification, use the loss function expected by the framework: passing probabilities to a function that expects logits can produce incorrect results or unstable gradients.

Label smoothing changes the target distribution from a one-hot vector to a softer distribution. That changes the loss definition and can improve training behavior, but it means the simple “only the correct class matters” explanation is no longer the full arithmetic. Compare models with the same smoothing and reduction settings.

What does lower cross-entropy tell you?

Cross-entropy is a proper scoring rule: in expectation, a model improves its score by reporting probabilities that match the data-generating uncertainty. That makes it useful for systems that must rank or triage cases, not only choose one label. A model that says “0.55” when an event happens about 55% of the time is better calibrated than one that says “0.99” for the same mixed group, even if both make the same hard decisions.

Loss and accuracy can disagree. Imagine two classifiers that are correct on 90 of 100 examples. One is moderately confident and the other is extremely confident; if the second model is very wrong on the ten failures, its cross-entropy can be much worse even though accuracy is identical. Use accuracy for the discrete decision you ship, and use loss or calibration diagnostics when the probability itself drives ranking, escalation, pricing, or risk.

Class imbalance and weighting need explicit attention. A weighted cross-entropy gives some examples more influence, so its average is not directly comparable with an unweighted loss unless the weighting scheme is the same. If the business cost of a false negative differs from a false positive, report the task metric alongside loss and choose the threshold separately. A lower unweighted loss does not automatically mean lower operational cost.

How does cross-entropy evaluation go wrong?

The first mistake is reporting accuracy as if it were loss. Two models can have the same accuracy while one assigns much higher probability to the correct answers and the other is overconfident on its mistakes. Cross-entropy exposes that difference; accuracy does not.

The second mistake is averaging the wrong quantity. Summed loss grows with dataset size, while mean loss is comparable only when the examples and weighting are comparable. Padding tokens in language-model batches must be masked consistently, or the loss can be dominated by tokens that should not count.

A third mistake is allowing zero probabilities into a manual calculation. −ln(0) is infinite. A stable implementation works from logits, and a hand-built calculator should reject zero rather than silently replace it with an arbitrary constant. If you use smoothing or clipping for analysis, document it.

Finally, do not compare perplexity across different tokenizers as if the values were on one common scale. Use the same evaluation text, tokenizer, context policy, masking, and loss reduction. For product decisions, supplement loss with task metrics such as accuracy, calibration, retrieval quality, or human acceptance.

Next steps: calculate and compare

Use the cross-entropy calculator with 0.8, 0.6, 0.9, and 0.7, then compare a second model on the same examples. For classification decisions, read the precision, recall, and F1 guide; for language-model input budgeting, read how many tokens are in 1,000 words?.

FAQ