Follow a line from prediction to update

A prediction is a claim about an observation

An observation supplies an input $x_i$ and a target $y_i$. An affine predictor uses two shared parameters to produce a number:

The weight $w$ controls how much the prediction changes when the input increases by one. The bias $b$ is the prediction at $x=0$. Changing either parameter changes the same line for every sample. Moving an observation changes the data; it does not directly move the predictor’s parameters.

Select a sample in the stage. Its filled circle is the observed pair $(x_i,y_i)$, and its hollow diamond is $(x_i,\hat y_i)$. The dashed segment is vertical because this loss compares target and prediction at the same input. It does not measure the shortest distance to the line.

Our residual convention is

A positive residual means overprediction; a negative residual means underprediction. The sign is useful even though the squared error $r_i^2$ is nonnegative. Some books reverse the residual convention, so check the definition before copying a gradient.

Samples, columns and shapes

These five unitless observations were explicitly hand-authored for this experiment: inputs $[-2,-1,0,1,2]$ and targets $[-3,-1,1,3,4]$, with stable IDs s1 through s5. They are synthetic toy data, not measurements or evidence about a population. The last target is one below the otherwise exact line $y=2x+1$.

Mathematical vectors are columns. A batch puts one observation in each row of $X$:

The scalar bias broadcasts to all five entries: $\hat y=XW+b\mathbf1$, where $\mathbf1$ is a $5\times1$ column of ones. Thus predictions and residuals also have shape $5\times1$. There is no intercept column in this chapter’s $X$; the bias remains a separate scalar. For $n$ observations, replace five with $n$.

The inspector reports these actual shapes. Stable IDs identify observations when their coordinates change; their IDs are not extra features.

Turn all residuals into one objective

Mean squared error averages the squared vertical residuals:

Squaring prevents positive and negative errors from cancelling and makes larger residuals contribute more. Averaging distinguishes MSE from the sum of squared errors. This definition contains no factor of one half and no square root.

MSE is the training objective in this experiment. A low value says the predictor matches these targets; it does not establish causal influence, reliable extrapolation, or performance on unseen observations. We have no separate validation or test set here.

Work one update by hand

Reset to $w=0.5$, $b=0$ and learning rate $\eta=0.1$. Compute each row before changing a parameter.

The squared errors sum to $20.5$, giving $L=20.5/5=4.1$. Their signed residuals sum to $-4$; the weighted residuals sum to

The gradients below therefore give $dw=(2/5)(-13)=-5.2$ and $db=(2/5)(-4)=-1.6$. Subtract both gradients, evaluated at the same old state:

Press Apply one update and check the new rows:

Their sum is $8.532$, so the new MSE is $1.7064$. Recomputing the gradients at this new state gives $dw=-3.12$ and $db=-1.28$. A second update would use these values, not the old gradients. The first decrease is an observation about this update, not a promise that every permitted learning rate decreases loss.

Derive the analytical gradients

Hold the data fixed. The chain rule and the partial derivatives of the affine predictor give

For a squared residual, the outer derivative contributes $2r_i$. Differentiating the average term by term gives

The gradient is the column $\nabla L=[dw,db]^{\mathsf T}$. The weight gradient includes the input because changing $w$ affects each prediction by $x_i$ times that change. The bias gradient weights every sample equally. For the observation at $x=0$, the weight’s contribution is zero even if its residual is large.

Gradient descent takes the simultaneous update

The implementation calculates these analytical derivatives directly. It does not use automatic differentiation or a fitted-model library. A central finite difference, $[L(w+h,b)-L(w-h,b)]/(2h)$, is a separate numerical check of $dw$; it is not the update rule used by the stage.

Edit and replay the same state

Use the numeric fields to change weight, bias, learning rate or the selected observation’s coordinates. Enter applies a valid value; a rejected entry gives a reason and preserves the model. Sample handles also accept pointer dragging and arrow keys. Each arrow changes a coordinate by $0.1$; holding Shift changes it by $0.5$. These paths all edit the observation used by the plot and table.

Replay 3 updates records the starting state and three successive updates. Frame 1 contains zero replay updates; frame 4 contains three. First, Previous and Next select immutable snapshots. The parameters, sample editor, plot, equations, table and inspector all switch to that exact selected frame. Editing or applying a new update starts from the displayed frame and clears the replay. Reset restores the original five observations and parameters.

Try these investigations:

1. Set the bias to $1$ and keep the weight fixed. Which predictions change? Check every row.
2. Reset, then change only sample s3’s target. Explain why its direct contribution to $dw$ is zero while its contribution to $db$ changes.
3. Move s5 vertically and compare its squared error with its residual. Doubling a residual multiplies its squared error by four.
4. Reset and replay three updates. Return to frame 2, edit the weight, and verify the new state begins from frame 2’s observations and bias.

Counterexamples clarify the limits

Constant inputs are still valid predictor data. If two samples have $x=2$ and targets $1$ and $3$, the predictors $(w,b)=(0,2)$ and $(1,0)$ both predict $2$ for both observations. Both have MSE $1$. The data constrain $2w+b$, so they do not identify a unique weight and bias. A single observation has the same kind of ambiguity. Neither case makes prediction, residuals or gradients undefined.

A perfect fit has zero gradient. For observations $(-1,-1)$ and $(1,3)$ with $w=2$, $b=1$, both residuals are zero. MSE and both gradients are zero, so an update leaves the parameters unchanged.

A valid rate can overshoot. With the single observation $(2,1)$, start at $w=b=0$ and $\eta=1$. MSE is initially $1$, and gradients are $dw=-4$, $db=-2$. The next parameters $(4,2)$ predict $10$, so the next MSE is $(10-1)^2=81$. A positive bounded rate alone is not a convergence certificate. Input scale and curvature matter.

Execute the actual Python reference

This is the source file used to generate the golden calculations. NumPy constructs $X$, $W$ and $y$ as column arrays, then computes predictions, residuals, MSE and analytical gradients from scratch. Running it prints the canonical before/after results and an initial-plus-three-updates trace.

From the book root, with the provisioned NumPy environment:

.venv/bin/python -B code/python/p02/linear_regression.py

"""III-07: affine predictions, MSE and analytical gradient descent using NumPy."""
from copy import deepcopy
import json
import math

import numpy as np


CANONICAL_REGRESSION = {
    "samples": [
        {"id": "s1", "x": -2, "y": -3},
        {"id": "s2", "x": -1, "y": -1},
        {"id": "s3", "x": 0, "y": 1},
        {"id": "s4", "x": 1, "y": 3},
        {"id": "s5", "x": 2, "y": 4},
    ],
    "weight": 0.5,
    "bias": 0,
    "learningRate": 0.1,
}


def _finite(value, label):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise ValueError(f"{label} must be a finite number")
    try:
        valid = math.isfinite(value)
    except OverflowError:
        valid = False
    if not valid:
        raise ValueError(f"{label} must be finite; input or arithmetic overflow is invalid")
    return value


def derive_regression(state):
    samples = state.get("samples") if isinstance(state, dict) else None
    if not isinstance(samples, (list, tuple)) or not 1 <= len(samples) <= 64:
        raise ValueError("Regression requires 1 to 64 samples")
    weight = _finite(state.get("weight"), "Weight")
    bias = _finite(state.get("bias"), "Bias")
    rate = _finite(state.get("learningRate"), "Learning rate")
    if not 0 < rate <= 1:
        raise ValueError("Learning rate must be greater than 0 and at most 1")
    ids = set()
    for sample in samples:
        identifier = sample.get("id") if isinstance(sample, dict) else None
        if not isinstance(identifier, str) or not identifier.strip() or identifier in ids:
            raise ValueError("Sample IDs must be nonempty strings and unique")
        ids.add(identifier)
        for key in ("x", "y"):
            _finite(sample.get(key), f"Sample {identifier} {key}")

    # Mathematical vectors are columns; each row of X is one scalar-feature sample.
    X = np.array([[sample["x"]] for sample in samples], dtype=np.float64)
    weights = np.array([[weight]], dtype=np.float64)
    targets = np.array([[sample["y"]] for sample in samples], dtype=np.float64)
    n = len(samples)
    try:
        with np.errstate(over="raise", invalid="raise", divide="raise", under="ignore"):
            products = X @ weights
            predictions = products + bias
            residuals = predictions - targets
            squared = residuals * residuals
            squared_sum = np.sum(squared)
            residual_sum = np.sum(residuals)
            weighted_sum = (X.T @ residuals)[0, 0]
            mse = float(squared_sum / n)
            dw = float((2 / n) * weighted_sum)
            db = float((2 / n) * residual_sum)
            next_weight = weight - rate * dw
            next_bias = bias - rate * db
    except FloatingPointError as error:
        raise ValueError(f"Regression arithmetic overflowed or produced a non-finite value: {error}") from error
    for values, label in ((products, "Weight × x"), (predictions, "Predictions"),
                          (residuals, "Residuals"), (squared, "Squared errors")):
        if not np.isfinite(values).all():
            raise ValueError(f"{label} overflowed or produced a non-finite value")
    for value, label in ((squared_sum, "Squared-error sum"), (residual_sum, "Residual sum"),
                         (weighted_sum, "Weighted-residual sum"), (mse, "MSE"),
                         (dw, "Weight gradient"), (db, "Bias gradient"),
                         (next_weight, "Updated weight"), (next_bias, "Updated bias")):
        _finite(float(value), label)
    return {
        "rows": [dict(id=sample["id"], x=sample["x"], y=sample["y"],
                      prediction=float(predictions[i, 0]), residual=float(residuals[i, 0]),
                      squaredError=float(squared[i, 0])) for i, sample in enumerate(samples)],
        "mse": mse,
        "gradients": {"weight": dw, "bias": db},
        "next": {"weight": next_weight, "bias": next_bias},
        "shapes": {"X": list(X.shape), "weights": list(weights.shape), "targets": list(targets.shape)},
    }


def regression_step(state):
    next_state = {**deepcopy(state), **derive_regression(state)["next"]}
    derive_regression(next_state)
    return next_state


def regression_trace(state, steps):
    if type(steps) is not int or not 0 <= steps <= 20:
        raise ValueError("Trace steps must be an integer from 0 to 20")
    derive_regression(state)
    state = deepcopy(state)
    frames = [{"index": 0, "state": state}]
    for index in range(1, steps + 1):
        state = regression_step(state)
        frames.append({"index": index, "state": state})
    return frames


if __name__ == "__main__":
    print(json.dumps({
        "before": derive_regression(CANONICAL_REGRESSION),
        "after": derive_regression(regression_step(CANONICAL_REGRESSION)),
        "trace": regression_trace(CANONICAL_REGRESSION, 3),
    }, indent=2, allow_nan=False))

The independent oracle uses exact rational arithmetic and an expanded loss based on data moments. It imports neither NumPy nor this reference. Its literal tests independently confirm the canonical MSE, gradients, first update and following loss; numerical tests also compare NumPy with that oracle and check the gradients by finite differences.

Numerical considerations

State uses binary64 numbers. Given sample coordinates use full round-trip decimal text in the table and accessible names. Computed predictions, residuals, squared errors, MSE and update values are displayed to four decimal places and marked approximately. A displayed approximate zero can therefore hide a small nonzero loss or gradient. Display rounding never changes the values used for an update. Keyboard movement adds its increment to the existing coordinate with ordinary binary64 arithmetic, preserving fine precision without extra decimal truncation; pointer movement explicitly snaps to the stated 0.1 lattice. Python/TypeScript parity is checked with absolute and relative tolerances of $10^{-12}$, rather than exact equality between independently rounded calculations.

The numerical API requires 1–64 observations, unique nonempty IDs, finite inputs and $0<\eta\leq1$. Finite inputs can still overflow during products, squared residuals or reductions. Such arithmetic is rejected with an informative error; an overflowing replay returns no partial history. Subnormal values and underflow follow binary64 arithmetic, and a very small update can round to no parameter change.

The plot has equal x/y scale and a fixed domain from −6 to 6. Predictions outside that plane are clipped visually but remain visible numerically in the table. Pointer movement stops at the observation editor’s boundary; numeric entry instead rejects an out-of-range value. Large touch targets expand with the rendered plot size, and only sample handles capture touch gestures, so the rest of the page remains scrollable.

Check your understanding

Each question grades the chosen answer and explains the misconception behind the alternatives. Use the canonical state unless the question says otherwise.

Checkpoints

You can now identify the observation, prediction and signed residual; account for batch and column shapes; calculate MSE and both analytical gradients; apply one simultaneous update; and inspect immutable replay without confusing an input bound with a convergence guarantee. Closed-form fitting, multiple features and classification belong to later chapters.