A learned rule is still a rule you can inspect

What changed when the machine learned?

Start with the selected observation at $x=0$, whose target is $y=1$. The initial rule predicts $0$. Change the bias to $1$: its prediction becomes $1$. Every other prediction shifts by the same amount. Change the weight instead: predictions change in proportion to their inputs. You are editing two numbers in a family of rules.

A learning algorithm chooses those numbers using examples and a criterion. Here the family is the affine predictor

$f_\theta(x)=wx+b,\qquad \theta=\begin{bmatrix}w\\b\end{bmatrix}.$

The model is this parameterized mapping. Its parameters are $w$ and $b$. The data are observed input–target pairs $(x_i,y_i)$. The algorithm evaluates candidate parameters using a loss, then returns a new parameter vector. Learning need not discover a symbolic law or remember each example; in this experiment, five rows influence two shared coefficients.

The input and target have different jobs. An input is information available when a prediction must be made. A target is the observed quantity the prediction is supposed to match. Targets must not quietly become input features when they would be unavailable at prediction time.

Trace the complete mechanism

For a supervised training batch, the pipeline is:

1. Observe inputs and targets with stable sample identities.
2. Represent inputs as compatible numerical features.
3. Apply the current predictor.
4. Compare predictions with targets and aggregate the error.
5. Use that error to choose changed parameters.

The sheet stores scalar features in a batch $X$ of shape $5\times1$. Each sample occupies a row; a single sample’s feature vector is a $1\times1$ column. The weight matrix is $1\times1$, the target column is $5\times1$, and bias broadcasts separately. A column convention does not mean that every batch must have only one row.

Define $r_i=wx_i+b-y_i$ and the mean squared error

$L(w,b)=\frac1n\sum_{i=1}^n r_i^2.$

Differentiation gives $dw=(2/n)\sum_i r_ix_i$ and $db=(2/n)\sum_i r_i$. The weight gradient contains $x_i$ because the sensitivity of a prediction to $w$ is $x_i$. Both partial derivatives are calculated before either parameter changes. The algorithm then returns $w-\eta dw$ and $b-\eta db$, where the user chooses the positive learning rate $\eta$.

The selected row is a microscope, not a switch to one-row training. The loss and update still use all five observations. A selected sample with $x=0$ contributes nothing directly to $dw$, although its target can change $db$.

Work through the first learned rule

The five inputs are $[-2,-1,0,1,2]$ and targets are $[-3,-1,1,3,4]$. These are explicit hand-authored toy values, with no units or claim about a real population. At $w=0.5,b=0$, predictions are $[-1,-0.5,0,0.5,1]$ and residuals are $[2,0.5,-1,-2.5,-3]$.

The squared errors sum to $20.5$, so $L=4.1$. The residual sum is $-4$ and the residual-times-input sum is $-13$. Consequently $dw=-5.2$ and $db=-1.6$. At $\eta=0.1$,

$w_{\mathrm{next}}=0.5-0.1(-5.2)=1.02,\qquad b_{\mathrm{next}}=0-0.1(-1.6)=0.16.$

The selected input $0$ now predicts $0.16$, with residual $-0.84$. Across the batch the new squared errors sum to $8.532$, giving MSE $1.7064$. The update improves this criterion on these observations. It does not establish performance on future data.

Inference uses the returned parameters

Reset and choose inference. For query $x=1.5$, the original parameters predict $0.75$. There is no observed query target, so an observed residual and query loss are undefined. The sheet displays that reason and disables the supervised update.

After training once, the same query predicts $1.02(1.5)+0.16=1.69$. No additional learning is required to evaluate the mapping. A later measured target could be used to evaluate it; deciding to retrain is a separate operation.

Execute the actual Python

The imported source uses NumPy column arrays for query prediction and calls the accepted affine reference for batch loss, gradient and update. There is one authoritative affine engine, rather than a second optimizer with a slightly different convention. From the book root, run .venv/bin/python -B code/python/p01/pipeline.py; the output contains training, inference and full-state trace frames.

"""III-01: observed targets train parameters; an unlabelled query only predicts."""
from copy import deepcopy
import json
from pathlib import Path
import sys
import numpy as np
from numeric import is_finite_number

ROOT = Path(__file__).resolve().parents[3]
sys.path.insert(0, str(ROOT / 'code/python/p02'))
import linear_regression as reference

PIPELINE_CANONICAL = dict(regression=deepcopy(reference.CANONICAL_REGRESSION), mode='training', sampleId='s3', query=1.5)

def derive_pipeline(state):
    if state['mode'] not in ('training', 'inference') or not is_finite_number(state['query']) or abs(state['query']) > 5:
        raise ValueError('Choose training/inference and a finite query between -5 and 5')
    numerical = reference.derive_regression(state['regression'])
    r = state['regression']
    if abs(r['weight']) > 4 or abs(r['bias']) > 4 or r['learningRate'] < .001 or any(abs(s['x']) > 5 or abs(s['y']) > 5 for s in r['samples']):
        raise ValueError('Predictor or observations are outside the supported sheet range')
    row = next((row for row in numerical['rows'] if row['id'] == state['sampleId']), None)
    if row is None:
        raise ValueError('Select an existing sample ID')
    x = row['x'] if state['mode'] == 'training' else state['query']
    X = np.array([[x]], dtype=np.float64)
    W = np.array([[r['weight']]], dtype=np.float64)
    prediction = float((X @ W + r['bias'])[0, 0])
    training = state['mode'] == 'training'
    return dict(state=deepcopy(state), numerical=numerical, x=x, prediction=prediction,
                target=row['y'] if training else None, residual=row['residual'] if training else None,
                loss=numerical['mse'] if training else None,
                reason=None if training else 'The query has no observed target. Inference produces a prediction, without an error or parameter update.',
                inputShape=list(X.shape), batchShape=numerical['shapes']['X'], weightShape=list(W.shape))

def pipeline_update(state):
    if state['mode'] != 'training':
        raise ValueError('Inference does not update parameters')
    result = {**deepcopy(state), 'regression': reference.regression_step(state['regression'])}
    derive_pipeline(result)
    return result

def pipeline_trace(state):
    derive_pipeline(state)
    if state['mode'] == 'inference':
        explanations = ['1. Receive a query x; no target is observed.', '2. Represent x as a 1 × 1 feature column.', '3. Apply the learned predictor. Parameters remain fixed.']
        states = [state] * 3
    else:
        explanations = ['1. Observe the selected sample’s input and target.', '2. Represent its input as a feature column; samples occupy rows in the batch.', '3. Predict with the current shared weight and bias.', '4. Compare prediction with target; compute the batch loss and gradient.', '5. Subtract one old-state gradient from both parameters simultaneously.']
        states = [state] * 4 + [pipeline_update(state)]
    return [dict(index=i, state=deepcopy(item), explanation=explanations[i]) for i, item in enumerate(states)]

if __name__ == '__main__':
    print(json.dumps(dict(training=derive_pipeline(PIPELINE_CANONICAL), inference=derive_pipeline({**PIPELINE_CANONICAL, 'mode': 'inference'}), trace=pipeline_trace(PIPELINE_CANONICAL)), indent=2, allow_nan=False))

The committed golden before/after/trace is independently checked by exact Fraction arithmetic. Matching two implementations alone would not prove that either used the intended normalization or residual sign.

Explore, replay and diagnose

Try changing the selected target while its input is zero. Then reset, change the selected input, and inspect both gradients through the later optimization chapter. Explain why a data edit and a parameter edit affect different parts of the pipeline.

Replay records four explanatory frames at the old parameters, followed by one updated state. These are mathematical snapshots, not an animation that changes the model between frames. Selecting a frame changes every numeric control, equation and table to that frame. Editing begins from the displayed state and ends replay. Reset restores the original observations and parameters.

Limits and common mistakes

This sheet supports weights and biases in $[-4,4]$, observations and queries in $[-5,5]$, and rates from $0.001$ to $1$. An unsupported update is rejected with the previous valid state retained. A permitted rate does not guarantee decreasing loss.

Numbers use binary64 arithmetic. Given coordinates retain round-trip decimal text; computed results carry an approximation sign. Display rounding never feeds training. Constant inputs and one-sample data can support prediction without identifying a unique weight and bias. This example neither proves causation nor justifies extrapolation.

Generalization needs an evaluation design, introduced in training, validation and testing. More elaborate models change the feature representation or parameter family while preserving the distinction between observations, predictions, objectives and decisions.

Check your understanding

Checkpoints

You can name the observed data, chosen family, learned parameters and update rule; follow the entire supervised pipeline; and apply a returned predictor to a query without inventing an error. Training chooses a mapping from examples. Inference evaluates that mapping with parameters held fixed.