Probabilities and purpose

A loss can reward better probability assignments without matching a thresholded score—or proving generalization.

A different kind of prediction

Leave the line-fitting example. In binary classification, the target $y$ is either 0 or 1. Suppose a model predicts a probability $p$ for class 1. This is not an unrestricted real-valued prediction or a raw logit: $p$ lies between 0 and 1.

Binary cross-entropy, also called binary log loss, evaluates the probability assigned to the observed label. With natural logarithm $\ln$ , for $0\lt p\lt1$ :

\begin{aligned}\ell(p,y)&=-y\ln p\\&\quad -(1-y)\ln(1-p)\end{aligned}

If $y=1$ , this becomes $-\ln p$ . If $y=0$ , it becomes $-\ln(1-p)$ . In either case, assigning more probability to the observed label lowers the penalty. The binary formula and natural-log convention make the meaning precise.

FIG-005A confident correct prediction pays less log loss.

Analytic −ln(p), with true label y=1. The plotted curve starts at p=0.01; it continues upward without bound as p approaches 0. Natural logs, no probability clipping; not a replica of a library's numerical safeguards.

For a positive target, probabilities 0.1, 0.5, and 0.9 produce losses about 2.303, 0.693, and 0.105. The theoretical loss approaches infinity as $p$ approaches zero, and is zero at $p=1$ . The finite plot begins at 0.01; it does not turn that infinite boundary into a finite penalty.

Mathematical endpoints versus library safeguards

The companion uses the theoretical limits: a perfectly correct endpoint has loss zero; a confidently wrong endpoint has infinite loss. Actual APIs may protect numerical calculations. PyTorch 2.8 BCELoss clamps logarithm outputs, while scikit-learn's log_loss clips probabilities. Our helper does not reproduce those endpoint safeguards.

Same loss, different decisions

Try a new, invented case. The true labels are $[1,0]$ . Model A predicts positive-class probabilities $[0.9,0.6]$ ; model B predicts $[0.6,0.4]$ . Predict class 1 whenever $p\geq0.5$ , otherwise class 0.

Which model has lower mean binary cross-entropy? Which has higher accuracy—the fraction of labels predicted correctly? Work out both before revealing the answer.

Compare the two answers

A assigns probabilities 0.9 and 0.4 to the correct labels. B assigns 0.6 and 0.6. Both products are 0.36:

\begin{aligned}L_A&=-\tfrac12\ln(0.9\times0.4)\\L_B&=-\tfrac12\ln(0.6\times0.6)\\L_A&=L_B=-\ln(0.6)\approx0.511\end{aligned}

But A's predicted labels are $[1,1]$ , so it gets one of two right. B predicts $[1,0]$ , getting both right. Equal mean loss does not imply equal accuracy. Accuracy counts decisions, while log loss also responds to the probabilities behind them. The fraction-correct definition is documented in accuracy_score.

The objective is not the whole task

A low training loss describes fit to training examples; it does not by itself establish performance on unseen data. Validation data can inform model choices, while a separate test set supports evaluation without tuning those choices to it. The cross-validation guide explains why reusing training or test data for the wrong role can mislead.

Our line achieved zero loss on three invented points. We supplied no held-out examples, so that result says nothing about generalization. Even a real holdout gives evidence under its sampling conditions, not a guarantee for every deployment setting.

Keep three questions separate: what does each error cost, how are those costs combined, and what evidence shows the resulting model serves the task? A loss function answers the first two. Good evaluation is still needed for the third.