III-15 · Precision, Recall, F1 and ROC

Name the denominator before trusting a metric

Compute precision, recall and F1 from the same confusion rows, expose undefined denominators, and replay an exact tied-score ROC sweep.

Probability and threshold laboratory

Count denominators → metrics → exact ROC sweep

Hand-authored labeled score fixture, shared by the threshold and metric chapters; no seed. These are illustrative scores, not a calibration claim.

s1≈ 0.9 → 1
s2≈ 0.8 → 1
s3≈ 0.8 → 1
s4≈ 0.4 → 0
s5≈ 0.1 → 0

Positive if score ≥ 0.5. Equality joins the positive class.

TP · Actual 1 / predicted 12

FP · Actual 0 / predicted 11

FN · Actual 1 / predicted 01

TN · Actual 0 / predicted 01

precision: ≈ 0.6667
recall: ≈ 0.6667
f1: ≈ 0.6667
auc: ≈ 0.5833

Change the active experimentDecision threshold0 to 1 · Enter to applyThreshold sliderSelected exampleExample score0 to 1 · Enter to applyActual binary label

Scores and labelsThe same score array supplies every count, metric and ROC point.: 5 × 1
Decision: p ≥ threshold
Score provenanceScores are fixed unless explicitly edited; moving a threshold never refits a model.: Hand-authored

Live state

\hat y_i=\mathbf1[p_i\ge t]

\operatorname{precision}=\frac{TP}{TP+FP},\quad\operatorname{recall}=\frac{TP}{TP+FN}

F_1=\frac{2TP}{2TP+FP+FN},\quad FPR=\frac{FP}{FP+TN}

One authoritative score per example; exact counts follow score ≥ threshold
ID	Score	Actual	Predicted	Cell
s1	0.9	1	1	TP
s2	0.8	0	1	FP
s3	0.8	1	1	TP
s4	0.4	0	0	TN
s5	0.1	1	0	FN

Each distinct score enters once; tied examples enter together. The first threshold is above all scores.
Threshold	TP	FP	TN	FN	TPR	FPR
+∞ (above all)	0	0	2	3	≈ 0	≈ 0
0.9	1	0	2	2	≈ 0.3333	≈ 0
0.8	2	1	1	1	≈ 0.6667	≈ 0.5
0.4	2	2	0	1	≈ 0.6667	≈ 1
0.1	3	2	0	0	≈ 1	≈ 1

Change the controls first, then inspect the active equation, numeric rows and shapes. Exact inputs and approximate computed values are distinguished; Reset restores the hand-authored experiment.

Compare one policy with the whole ranking

At the opening threshold $0.5$ , inspect the counts and metrics. Precision, recall and F1 are all $2/3$ on this specific fixture, but their denominators answer different questions. Press Predict no positives: precision becomes explicitly undefined, recall and F1 become zero because actual positives still exist.

Press No actual positives to expose the missing recall denominator and undefined ROC/AUC. No chart fabricates a rate for a missing actual class. Reset, replay the threshold sweep and watch the active ROC point move over exactly the score transitions, including tied examples.

Derive the three operating-point metrics

For one threshold, define

\operatorname{precision}=\frac{TP}{TP+FP},\qquad \operatorname{recall}=\frac{TP}{TP+FN},\qquad F_1=\frac{2TP}{2TP+FP+FN}.

Precision conditions on positive predictions: how many are actually positive? Recall conditions on actual positives: how many were found? F1 combines their balance, but the count expression is the primary definition here. It avoids propagating an undefined precision through a harmonic-mean expression when the count denominator remains meaningful.

This implementation uses undefined with a reason for every zero denominator. If there are actual positives but no predicted positives, precision is undefined, recall is zero and F1 is zero. If there are neither actual nor predicted positives, all three are undefined. If actual positives are absent but positive predictions exist, recall is undefined while precision and F1 are zero. Undefined is JSON null plus a reason; it is never silently replaced by a favorable score.

Work the canonical operating point

The same saved score column as III-14 is $[0.9,0.8,0.8,0.4,0.1]^\mathsf T$ , with labels $[1,0,1,0,1]^\mathsf T$ . These are hand-authored toy values without a seed or calibration claim. At $t=0.5$ , counts are $TP=2,FP=1,TN=1,FN=1$ , so precision $=2/3$ , recall $=2/3$ , and $F_1=4/6=2/3$ .

At $t=0.81$ , counts become $(1,0,2,2)$ . Precision rises to one while recall falls to $1/3$ and F1 to $2/(2+0+2)=1/2$ . “One metric improved” does not mean the decision is preferable. It depends on the required error tradeoff.

Accuracy is $(TP+TN)/n$ . With 99 actual negatives and one actual positive, always predicting negative gives 99% accuracy and zero recall. Precision is undefined. The sheet’s One actual positive preset gives a smaller inspectable imbalance example; it does not pretend five rows estimate a real population.

Derive the exact ROC sweep

ROC uses

TPR=\frac{TP}{TP+FN},\qquad FPR=\frac{FP}{FP+TN}.

These denominators are the fixed counts of actual positives and actual negatives. Start above every score at $(FPR,TPR)=(0,0)$ , then lower the threshold to each distinct stored score in descending order. With the $\ge$ rule, every tied score enters together. A uniform grid of thresholds can miss transitions and produce an inaccurate curve.

Threshold	TP	FP	TN	FN	FPR	TPR
Above all (+∞)	0	0	2	3	0	0
0.9	1	0	2	2	0	1/3
0.8	2	1	1	1	1/2	2/3
0.4	2	2	0	1	1	2/3
0.1	3	2	0	0	1	1

There are five points, including the endpoint above all scores. The two score-0.8 examples create one joint move; their order is never an arbitrary ranking tiebreak. The displayed polyline and numeric sweep table derive from the same counts.

Area is a ranking statistic

Trapezoidal area is

AUC=\sum_j(FPR_j-FPR_{j-1})\frac{TPR_j+TPR_{j-1}}2.

For this table it is $1/4+1/3=7/12\approx0.5833333$ . Independently, compare every positive score with every negative score: count one for a strictly higher positive score, half for a tie and zero for a lower score. The three positive scores $0.9,0.8,0.1$ earn $2,1.5,0$ wins against negative scores $0.8,0.4$ . Dividing $3.5$ by six pairs also gives $7/12$ .

This pairwise interpretation checks the sweep without repeating its trapezoid algorithm. AUC summarizes ranking over policies; it does not choose the threshold or establish probability calibration. When either actual class is absent, ROC rates and AUC are undefined with explicit reasons. With all scores tied and both classes present, the diagonal segment has AUC $1/2$ .

Explore and avoid misleading comparisons

Change a selected score and inspect the new distinct thresholds, active counts and AUC. Changing only the current threshold changes precision/recall/F1 and the active point but leaves the entire ROC ranking and AUC unchanged. Changing labels changes both the denominator populations and ranking comparison.

Do not silently map undefined metrics to zero in one table and one in another. Do not compare recall across datasets without stating their label population. Do not assume a high ROC AUC means useful precision at low prevalence; precision depends on the mixture of actual classes. Choose metrics and costs appropriate to the decision, then evaluate on independent data.

Numerical limits, connections and checkpoints

Exact count arithmetic and distinct-score ordering handle this 1–64-row probability contract. Rates and AUC use binary64 fractions; display rounding does not define ties. Both label classes are needed for a meaningful ROC plane. Keyboard/numeric/touch controls, reset, immutable sweep playback and spoken summaries report the same active policy, including the visible above-all sentinel.

Python roc builds the exact score sweep; separate set-count literals and pairwise ranking establish its oracle. Later imbalance/calibration chapters consider dataset prevalence and score meaning. You should be able to name every denominator, justify each undefined state, handle ties without arbitrary ordering and distinguish one threshold’s metric from a whole-ranking statistic.

Execute the actual Python and NumPy

The imported source below is the runnable file used in canonical comparisons. Read the named mechanism from this chapter; the other functions coordinate the same Part experiment. From the book root:

.venv/bin/python -B code/python/p02/classification_reference.py

code/python/p02/classification_reference.py

"""III-13–16: stable logit losses, exact thresholds/ROC and softmax gradients.

Run: .venv/bin/python -B code/python/p02/classification_reference.py
Samples occupy rows; binary W is d×1, multiclass W is d×k.
"""
import json
import math
import numpy as np
from fit_reference import column, design, finite

SCORE_FIXTURE = {"scores": [.9, .8, .8, .4, .1], "labels": [1, 0, 1, 0, 1]}


def class_labels(y, n, k=2):
    column(y, n, "Labels")
    if any(int(v) != v or not 0 <= v < k for v in y):
        raise ValueError(f"Labels must be class indices 0–{k-1}")


def sigmoid(z):
    if isinstance(z, bool) or not isinstance(z, (int, float)) or not math.isfinite(z):
        raise ValueError("Logit must be finite")
    return 1 / (1 + math.exp(-z)) if z >= 0 else math.exp(z) / (1 + math.exp(z))


def bce(z, y):
    sigmoid(z)
    if isinstance(y, bool) or y not in (0, 1):
        raise ValueError("Binary label must be 0 or 1")
    return float(finite(max(z, 0) - z * y + math.log1p(math.exp(-abs(z))), "BCE"))


def learning_rate(rate):
    if isinstance(rate, bool) or not isinstance(rate, (int, float)) or not math.isfinite(rate) or not 0 < rate <= 1:
        raise ValueError("Learning rate must be in (0,1]")


def logistic(X, y, weights, rate):
    A, _ = design(X, y)
    W = column(weights, A.shape[1], "Weights")
    class_labels(y, len(A))
    learning_rate(rate)
    logits = finite(A @ W, "Logits")[:, 0].tolist()
    scores = [sigmoid(z) for z in logits]
    losses = [bce(z, label) for z, label in zip(logits, y)]
    loss = float(finite(sum(losses) / len(A), "Mean BCE"))
    gradient = finite(A.T @ (np.array(scores) - np.array(y)).reshape(-1, 1) / len(A), "Gradient")
    next_weights = finite(W - rate * gradient, "Updated weights")
    n, d = A.shape
    return {"logits": logits, "scores": scores, "losses": losses, "loss": loss,
            "gradient": gradient[:, 0].tolist(), "next": next_weights[:, 0].tolist(),
            "shapes": {"X": [n, d], "W": [d, 1], "y": [n, 1], "gradient": [d, 1]}}


def ratio(top, bottom, reason):
    return {"value": top / bottom, "reason": None} if bottom else {"value": None, "reason": reason}


def confusion(scores, y, threshold):
    column(scores, len(scores), "Scores")
    if not 1 <= len(scores) <= 64 or any(not 0 <= p <= 1 for p in scores):
        raise ValueError("Scores require 1–64 probabilities in [0,1]")
    class_labels(y, len(scores))
    if threshold is not None and (isinstance(threshold, bool) or not isinstance(threshold, (int, float)) or not math.isfinite(threshold) or not 0 <= threshold <= 1):
        raise ValueError("Threshold must be in [0,1] or the explicit above-all None sentinel")
    rows = []
    cells = dict(TP=0, FP=0, TN=0, FN=0)
    for p, label in zip(scores, y):
        prediction = int(threshold is not None and p >= threshold)
        cell = ("TP" if label else "FP") if prediction else ("FN" if label else "TN")
        cells[cell] += 1
        rows.append(dict(score=p, label=label, prediction=prediction, cell=cell))
    tp, fp, tn, fn = (cells[k] for k in ("TP", "FP", "TN", "FN"))
    return {"rows": rows, "tp": tp, "fp": fp, "tn": tn, "fn": fn,
            "precision": ratio(tp, tp + fp, "No predicted positives (TP+FP=0)."),
            "recall": ratio(tp, tp + fn, "No actual positives (TP+FN=0)."),
            "f1": ratio(2 * tp, 2 * tp + fp + fn, "No actual or predicted positives (2TP+FP+FN=0)."),
            "accuracy": (tp + tn) / len(scores)}


def roc(scores, y):
    confusion(scores, y, .5)
    positives, negatives = y.count(1), y.count(0)
    points = []
    for threshold in [None] + sorted(set(scores), reverse=True):
        c = confusion(scores, y, threshold)
        points.append({"threshold": threshold, **{k: c[k] for k in ("tp", "fp", "tn", "fn")},
                       "tpr": ratio(c["tp"], positives, "ROC TPR needs at least one positive."),
                       "fpr": ratio(c["fp"], negatives, "ROC FPR needs at least one negative.")})
    reason = None if positives and negatives else "ROC/AUC requires both actual classes."
    area = None if reason else sum((b["fpr"]["value"] - a["fpr"]["value"]) * (a["tpr"]["value"] + b["tpr"]["value"]) / 2 for a, b in zip(points, points[1:]))
    return {"points": points, "auc": {"value": area, "reason": reason}, "positives": positives, "negatives": negatives}


def softmax(logits):
    if not isinstance(logits, (list, tuple)) or not 2 <= len(logits) <= 6:
        raise ValueError("Softmax requires 2–6 logits")
    column(logits, len(logits), "Logits")
    maximum = max(logits)
    shifted = finite(np.array(logits) - maximum, "Shifted logits")
    exp = np.exp(shifted)
    total = float(np.sum(exp))
    return {"maximum": maximum, "shifted": shifted.tolist(), "probabilities": (exp / total).tolist(), "logSumExpShifted": math.log(total)}


def multiclass(X, y, W, rate, shift=0):
    A, _ = design(X, y)
    weights, _ = design(W, [0.] * len(W))
    n, d = A.shape
    k = weights.shape[1]
    if weights.shape[0] != d or not 2 <= k <= 6:
        raise ValueError("W must be d×k with 2–6 classes")
    class_labels(y, n, k)
    learning_rate(rate)
    if isinstance(shift, bool) or not isinstance(shift, (int, float)) or not math.isfinite(shift):
        raise ValueError("Common shift must be finite")
    logits = finite(A @ weights + shift, "Logits")
    normalized = [softmax(row) for row in logits.tolist()]
    P = np.array([r["probabilities"] for r in normalized])
    Y = np.eye(k)[np.array(y, dtype=int)]
    losses = [r["logSumExpShifted"] - r["shifted"][int(y[i])] for i, r in enumerate(normalized)]
    loss = float(finite(sum(losses) / n, "Mean cross entropy"))
    gradient = finite(A.T @ (P - Y) / n, "Gradient")
    next_weights = finite(weights - rate * gradient, "Updated weights")
    return {"logits": logits.tolist(), "normalized": normalized, "probabilities": P.tolist(), "losses": losses, "loss": loss,
            "gradient": gradient.tolist(), "next": next_weights.tolist(), "predicted": np.argmax(P, axis=1).tolist(), "oneHot": Y.tolist(),
            "shapes": {"X": [n, d], "W": [d, k], "logits": [n, k], "Y": [n, k], "gradient": [d, k], "labels": [n, 1]}}


def demonstration():
    X, y, w = [[1, -2], [1, -1], [1, 0], [1, 1], [1, 2]], [0, 0, 1, 0, 1], [0, 1]
    trace = []
    for i in range(4):
        result = logistic(X, y, w, .2)
        trace.append({"index": i, "weights": w, "derived": result})
        w = result["next"]
    return {"logisticTrace": trace, "threshold": confusion(SCORE_FIXTURE["scores"], SCORE_FIXTURE["labels"], .5),
            "roc": roc(SCORE_FIXTURE["scores"], SCORE_FIXTURE["labels"]),
            "softmax": multiclass([[1, -1], [1, 0], [1, 1]], [2, 1, 0], [[0, 0, 0], [1, 0, -1]], .2),
            "extremeBCE": [[z, label, bce(z, label)] for z in (-1000, 1000) for label in (0, 1)]}


if __name__ == "__main__":
    print(json.dumps(demonstration(), indent=2, allow_nan=False))