Inspect the decision that changes a parameter

Four kinds of quantities

The observed inputs and targets are data. The weight $w$ and bias $b$ are trainable parameters: this algorithm changes them in response to data. The learning rate $\eta$ is an externally chosen hyperparameter controlling the update. The loss is a derived number measuring the current predictions against observed targets.

A hyperparameter can be selected using validation evidence, but that is a different decision from this gradient update. This loop does not differentiate or learn $\eta$. Polynomial degree in the next chapter is another hyperparameter because it chooses the family of predictors.

An objective is the function optimized during fitting. An evaluation metric assesses behavior on a specified set of observations. Here both use mean squared error, but a training loss and validation MSE still have different roles and denominators.

Define the objective before differentiating

For the affine predictor and the residual convention used throughout Part I,

$\hat y_i=wx_i+b,\qquad r_i=\hat y_i-y_i,\qquad L=\frac1n\sum_i r_i^2.$

There is no one-half factor. Adding one would halve the gradients without changing the minimizer, and would require consistent learning-rate interpretation. Reversing the residual sign also requires changing the derivative expression consistently.

The feature batch is $n\times1$, weight matrix $1\times1$ and target column $n\times1$. Bias broadcasts as a separate scalar. The two trainable numbers form $\theta=[w,b]^{\mathsf T}$; that parameter vector should not be confused with the $1\times1$ weight matrix in the affine prediction.

Derive the simultaneous update

The chain rule gives $\partial r_i/\partial w=x_i$ and $\partial r_i/\partial b=1$. Differentiate each squared residual and then average:

$\frac{\partial L}{\partial w}=\frac2n\sum_i r_ix_i,\qquad \frac{\partial L}{\partial b}=\frac2n\sum_i r_i.$

The factor two comes from differentiating $r_i^2$. The factor $1/n$ comes from the average. Neither can be omitted without changing the objective.

Gradient descent proposes

$\theta_{\mathrm{next}}=\theta-\eta\nabla L(\theta).$

Both entries of $\nabla L$ use the same old parameters. Computing $dw$, changing $w$, then recomputing $db$ would implement a different sequence. The sheet derives the whole active object once, then renders its prediction rows, loss, gradient and next parameters together.

Work before and after by hand

The canonical toy batch has $x=[-2,-1,0,1,2]$ and $y=[-3,-1,1,3,4]$. At $w=0.5,b=0$, the residuals are $[2,0.5,-1,-2.5,-3]$ and squared errors are $[4,0.25,1,6.25,9]$.

Their sum $20.5$ gives MSE $4.1$. Also $\sum r_ix_i=-13$ and $\sum r_i=-4$, so $dw=-5.2$ and $db=-1.6$. With $\eta=0.1$, the next weight is $1.02$ and next bias is $0.16$.

The updated predictions are $[-1.88,-0.86,0.16,1.18,2.2]$. Residuals become $[1.12,0.14,-0.84,-1.82,-1.8]$. The squared errors sum to $8.532$, giving new MSE $1.7064$.

A second step must use new gradients: $dw=-3.12$ and $db=-1.28$. The first gradient is not a permanent direction for the whole optimization process. Press one update or replay to inspect these actual returned states.

A permitted learning rate can overshoot

Consider one observation $(x,y)=(2,1)$ at $w=b=0$. MSE is $1$, and gradients are $dw=-4$, $db=-2$. Choosing $\eta=1$ returns $(w,b)=(4,2)$, which predicts $10$. New MSE is $81$.

A positive rate validates the direction’s sign, not the size of the finite step. Input scale and curvature influence useful rates. Small parameters are not automatically stable, and immutable traces do not change convergence.

Execute the actual implementation

The Part I wrapper records full states and calls the frozen affine reference. The second imported file exposes that actual NumPy prediction, residual, MSE and analytical-gradient implementation. No library estimator or second Part I regression engine is substituted. Run .venv/bin/python -B code/python/p01/optimization_loop.py for the initial state and three updates.

"""III-05: reuse the frozen affine loss/gradient/update contract, without another optimizer."""
from copy import deepcopy
import json
from pipeline import reference

CANONICAL = deepcopy(reference.CANONICAL_REGRESSION)

def optimization_loop(state=CANONICAL, steps=3):
    # Every frame is a full state; both parameter changes use the same old gradient.
    return [dict(**frame, derived=reference.derive_regression(frame['state'])) for frame in reference.regression_trace(state, steps)]

if __name__ == '__main__':
    print(json.dumps(optimization_loop(), indent=2, allow_nan=False))

"""III-07: affine predictions, MSE and analytical gradient descent using NumPy."""
from copy import deepcopy
import json
import math

import numpy as np


CANONICAL_REGRESSION = {
    "samples": [
        {"id": "s1", "x": -2, "y": -3},
        {"id": "s2", "x": -1, "y": -1},
        {"id": "s3", "x": 0, "y": 1},
        {"id": "s4", "x": 1, "y": 3},
        {"id": "s5", "x": 2, "y": 4},
    ],
    "weight": 0.5,
    "bias": 0,
    "learningRate": 0.1,
}


def _finite(value, label):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise ValueError(f"{label} must be a finite number")
    try:
        valid = math.isfinite(value)
    except OverflowError:
        valid = False
    if not valid:
        raise ValueError(f"{label} must be finite; input or arithmetic overflow is invalid")
    return value


def derive_regression(state):
    samples = state.get("samples") if isinstance(state, dict) else None
    if not isinstance(samples, (list, tuple)) or not 1 <= len(samples) <= 64:
        raise ValueError("Regression requires 1 to 64 samples")
    weight = _finite(state.get("weight"), "Weight")
    bias = _finite(state.get("bias"), "Bias")
    rate = _finite(state.get("learningRate"), "Learning rate")
    if not 0 < rate <= 1:
        raise ValueError("Learning rate must be greater than 0 and at most 1")
    ids = set()
    for sample in samples:
        identifier = sample.get("id") if isinstance(sample, dict) else None
        if not isinstance(identifier, str) or not identifier.strip() or identifier in ids:
            raise ValueError("Sample IDs must be nonempty strings and unique")
        ids.add(identifier)
        for key in ("x", "y"):
            _finite(sample.get(key), f"Sample {identifier} {key}")

    # Mathematical vectors are columns; each row of X is one scalar-feature sample.
    X = np.array([[sample["x"]] for sample in samples], dtype=np.float64)
    weights = np.array([[weight]], dtype=np.float64)
    targets = np.array([[sample["y"]] for sample in samples], dtype=np.float64)
    n = len(samples)
    try:
        with np.errstate(over="raise", invalid="raise", divide="raise", under="ignore"):
            products = X @ weights
            predictions = products + bias
            residuals = predictions - targets
            squared = residuals * residuals
            squared_sum = np.sum(squared)
            residual_sum = np.sum(residuals)
            weighted_sum = (X.T @ residuals)[0, 0]
            mse = float(squared_sum / n)
            dw = float((2 / n) * weighted_sum)
            db = float((2 / n) * residual_sum)
            next_weight = weight - rate * dw
            next_bias = bias - rate * db
    except FloatingPointError as error:
        raise ValueError(f"Regression arithmetic overflowed or produced a non-finite value: {error}") from error
    for values, label in ((products, "Weight × x"), (predictions, "Predictions"),
                          (residuals, "Residuals"), (squared, "Squared errors")):
        if not np.isfinite(values).all():
            raise ValueError(f"{label} overflowed or produced a non-finite value")
    for value, label in ((squared_sum, "Squared-error sum"), (residual_sum, "Residual sum"),
                         (weighted_sum, "Weighted-residual sum"), (mse, "MSE"),
                         (dw, "Weight gradient"), (db, "Bias gradient"),
                         (next_weight, "Updated weight"), (next_bias, "Updated bias")):
        _finite(float(value), label)
    return {
        "rows": [dict(id=sample["id"], x=sample["x"], y=sample["y"],
                      prediction=float(predictions[i, 0]), residual=float(residuals[i, 0]),
                      squaredError=float(squared[i, 0])) for i, sample in enumerate(samples)],
        "mse": mse,
        "gradients": {"weight": dw, "bias": db},
        "next": {"weight": next_weight, "bias": next_bias},
        "shapes": {"X": list(X.shape), "weights": list(weights.shape), "targets": list(targets.shape)},
    }


def regression_step(state):
    next_state = {**deepcopy(state), **derive_regression(state)["next"]}
    derive_regression(next_state)
    return next_state


def regression_trace(state, steps):
    if type(steps) is not int or not 0 <= steps <= 20:
        raise ValueError("Trace steps must be an integer from 0 to 20")
    derive_regression(state)
    state = deepcopy(state)
    frames = [{"index": 0, "state": state}]
    for index in range(1, steps + 1):
        state = regression_step(state)
        frames.append({"index": index, "state": state})
    return frames


if __name__ == "__main__":
    print(json.dumps({
        "before": derive_regression(CANONICAL_REGRESSION),
        "after": derive_regression(regression_step(CANONICAL_REGRESSION)),
        "trace": regression_trace(CANONICAL_REGRESSION, 3),
    }, indent=2, allow_nan=False))

The independent arithmetic oracle confirms the decimal literals as exact Fractions. The accepted reference also checks analytical gradients using finite differences. Python/TypeScript arithmetic parity uses small stated tolerances rather than assuming independently rounded operations are bit-identical.

Explore exact state and immutable history

Edit a weight with more than four decimal places. The symbolic identity remains $\hat y=wx+b$; any rounded parameter substitution is marked approximately. Given sample coordinates retain round-trip decimal text. A displayed approximate zero can still be a small nonzero value.

Focus an observation and use arrows; Shift increases the step. Pointer dragging explicitly snaps to the stated coordinate lattice, while keyboard addition preserves the existing fine precision under ordinary binary64 rounding. Numeric entry rejects unsupported values without inventing a clamped observation.

Replay stores the initial state and three updates. Selecting a historical frame changes every field, row, plot and equation to that state. Editing starts there and clears replay. Reset restores the original hand-authored batch, not the most recently visited frame.

Degenerate data and connections

A perfect fit has zero residuals and gradients, so the update leaves both parameters unchanged. If every input is $2$, the data constrain $2w+b$, not a unique pair. One-sample and constant-input prediction remain valid; unique coefficient identification is a separate question.

This chapter uses full-batch gradient descent. Mini-batches, momentum and adaptive optimizers change the update procedure, not the need to declare loss normalization and state. The next capacity experiment uses a bounded least-squares solve to separate model-family effects from unfinished optimizer convergence.

Check your understanding

Checkpoints

You can distinguish observations, parameters, hyperparameters and loss; derive both gradients; apply one old-state update; and explain overshoot and nonidentifiability. An inspectable optimization loop returns a new full state and makes its assumptions and failure reasons visible.