A better fit to known rows can be a worse predictor

Capacity is a family of possible rules

A constant cannot bend or slope. A line can slope but cannot curve. A quadratic can express this toy signal, while a degree-eight polynomial can interpolate nine distinct training inputs. Increasing capacity expands the available rules; it does not certify that the selected rule will predict new observations well.

Underfitting occurs when a fitted family fails to express useful structure under the chosen representation and fitting procedure. Overfitting occurs when fitting details of the training sample, including noise, harms performance on relevant new data. Small training loss is evidence of fit, not by itself evidence against overfitting.

The hand-authored grid has seventeen inputs $x_i=-1+i/8$, for zero-based indices $i=0,\ldots,16$. A known quadratic signal generates its targets:

$f_*(x)=1+2x-x^2,\qquad y_i=f_*(x_i)+\epsilon_i.$

The noise seed drives the shared LCG. Each perturbation is the displayed amplitude times one of $-1,-0.5,0,0.5,1$, chosen from five equally sized integer bins of the RNG output up to discrete rounding. These are synthetic illustrative observations, not measurements of a real population.

Reset trains on even grid indices: IDs p1, p3, …, p17. Odd grid indices validate. There are initially nine training rows, eight validation rows and no test rows. An empty test MSE is undefined. Seeded memberships create ten, three and four rows; the separate split seed changes assignments without regenerating noise.

Fit only the active training IDs

A degree-$d$ polynomial uses the coefficient column $\theta=[\theta_0,\ldots,\theta_d]^{\mathsf T}$:

$\hat y(x)=\sum_{j=0}^{d}\theta_jx^j,\qquad X_{ij}=x_i^{j-1}.$

Here $X$ has a first column of ones. For $n_T$ training observations, shapes are $X:n_T\times(d+1)$, $\theta:(d+1)\times1$, and $y_T:n_T\times1$. Unlike the earlier affine sheet, this design includes the intercept in its coefficient column. Mathematical vectors remain columns; observations occupy matrix rows.

The least-squares objective is

$L_T(\theta)=\frac1{n_T}\lVert X_T\theta-y_T\rVert_2^2.$

Both runtimes use a two-pass modified Gram–Schmidt QR solve for this bounded experiment. Writing $X_T=QR$, orthogonal projection gives $R\theta=Q^{\mathsf T}y_T$, solved by back substitution. It fits the selected family directly, so a poor fit here is not explained by too few gradient-descent steps. This is a Part-owned capacity mechanism, not another shared affine optimizer.

Every membership move rebuilds $X_T$ and $y_T$ from the actual displayed training IDs. Validation and test labels do not enter the coefficient solve.

Derive the honest constant baseline

A baseline asks whether the proposed model improves on a simple declared procedure. Under squared error, the best training-fitted constant solves

$L_T(c)=\frac1{n_T}\sum_{i\in T}(c-y_i)^2,\qquad \frac{dL_T}{dc}=\frac2{n_T}\sum_{i\in T}(c-y_i)=0.$

Therefore $c=\bar y_T$. A model’s losses and its baseline losses must be evaluated on the same held-out rows. Degree zero yields the same constant fit up to numerical rounding.

At reset, $\bar y_T=11/18\approx0.6111$. Degree-zero training MSE is approximately $2.1422$, and validation MSE is approximately $2.0472$. Degree two has coefficients approximately $[1.3128,2.0333,-1.6840]$, training MSE $0.0404$ and validation MSE $0.2068$. These are rounded displays of the actual golden fit, not exact equations using rounded coefficients.

Move p1 from training to validation. Eight rows now fit the model. Their baseline mean is $31/32=0.96875$; the new degree-two coefficients are approximately $[1.3542,2.1548,-1.9048]$. Both model and baseline changed because the fitting sample changed, while p1’s identity and target stayed fixed.

A concrete overfitting counterexample

Keep interleaved memberships and noise amplitude $0.5$, but enter noise seed $0$. Degree two has training MSE approximately $0.0968$ and validation MSE $0.0678125$. Degree eight nearly interpolates the nine training targets, with training error close to binary64 zero, but validation MSE approximately $0.3030743$.

The higher-degree fit improves the training objective while worsening held-out error. Reset seed $42$ gives another outcome: degree eight’s validation MSE is approximately $0.1473$, better than degree two’s $0.2068$ on this particular small split. Do not replace the experiment with a slogan that high degree always loses.

Set noise to zero. Degree two recovers coefficients $[1,2,-1]$ within rounding and predicts the noiseless validation rows almost exactly. In nested least-squares families, optimal training error cannot increase with degree in exact arithmetic; held-out error has no corresponding monotonic guarantee.

Bias and variance describe repeated experiments

Bias describes systematic discrepancy between the average fitted prediction and the true signal. Variance describes how fitted predictions change across different training samples. At a fixed input, under zero-mean observation noise independent of fitting and with variance $\sigma^2$, the expected squared prediction error decomposes as

$\mathbb E[(\hat f(x)-Y)^2] =(\mathbb E[\hat f(x)]-f_*(x))^2 +\operatorname{Var}(\hat f(x))+\sigma^2.$

This identity follows by adding and subtracting $\mathbb E[\hat f(x)]$ and using the zero-mean cross terms. It requires the stated stochastic assumptions. One displayed training/validation gap does not numerically estimate all three terms.

In this lab, changing the noise seed produces another deterministic synthetic realization; changing the split alters the observed fitting sample. Compare how coefficients and curves move at low and high degree while keeping the other choices fixed. This supports qualitative bias/variance reasoning without claiming a population estimate from seventeen rows.

Selection, leakage and honest test use

“Choose degree by validation” evaluates degrees $0$ through $8$ on the same active memberships and chooses the smallest degree whose validation loss improves by more than $10^{-12}$. Undefined candidates are excluded. No defined validation loss means no selected degree. Test loss never participates in selection.

Test targets and residuals stay hidden until reveal. Repeatedly revealing, editing and choosing would turn that set into another validation set; the UI cannot make a repeatedly inspected test untouched again. Use the reveal as a final educational evaluation.

The optional whole-target baseline is deliberately invalid. It averages all seventeen labels, including validation and test, instead of the training labels. Its changing score is displayed as a leakage comparison, never as the honest baseline or a selection criterion. The earlier preprocessing counterexample separately shows whole-dataset feature centering leakage.

Execute the actual NumPy reference

The imported source implements seeded observations, shared Part-local split operations, explicit QR fitting, polynomial evaluation and training-only means. It uses NumPy for arrays rather than an ML estimator wrapper. Run .venv/bin/python -B code/python/p01/capacity_baselines.py.

"""III-06: one active split, polynomial capacity and a training-only mean baseline."""
from copy import deepcopy
import json
import math
import numpy as np
from seed import lcg
from numeric import is_finite_number
from split_generalization import assign_split, derive_split, seed_split, PARTITIONS

CAPACITY_CANONICAL = dict(seed=42, degree=2, noise=.5, splitSeed=42, source='interleaved')
RANK_THRESHOLD = 1e-12
RANK_REASON = 'The polynomial design is numerically rank deficient: a reorthogonalized QR column norm is ≤ 1e−12 times the design Frobenius norm.'

def observations(state):
    if type(state['degree']) is not int or not 0 <= state['degree'] <= 8 or not is_finite_number(state['noise']) or not 0 <= state['noise'] <= 2:
        raise ValueError('Integer degree 0–8 and finite noise 0–2 are required')
    random = lcg(state['seed'])
    rows = []
    for i in range(17):
        x = -1+i/8
        truth = 1+2*x-x*x
        noise = state['noise']*(int(next(random)*5)-2)/2
        rows.append(dict(id=f'p{i+1}', group=f'p{i+1}', x=x, y=truth+noise, truth=truth))
    return rows

def capacity_split(state):
    samples = observations(state)
    source = state.get('source','interleaved')
    if source not in ('interleaved','seeded','manual','distribution-shift'):
        raise ValueError('Unknown capacity assignment provenance')
    split = dict(samples=samples, seed=state.get('splitSeed',42), keepGroups=False,
                 assignments=state.get('assignments', {row['id']:'train' if i%2 == 0 else 'validation' for i,row in enumerate(samples)}),
                 source='manual' if source == 'interleaved' else source)
    derive_split(split)
    return split

def capacity_data(state):
    split = capacity_split(state)
    return [dict(id=row['id'],x=row['x'],y=row['y'],truth=row['truth'],partition=split['assignments'][row['id']]) for row in observations(state)]

def assign_capacity(state, identifier, partition):
    split = assign_split(capacity_split(state),identifier,partition)
    return {**deepcopy(state),'assignments':split['assignments'],'source':'manual'}

def preset_capacity_split(state, source, split_seed=None):
    if source not in ('interleaved','seeded','distribution-shift'):
        raise ValueError('Unknown assignment preset')
    split_seed = state.get('splitSeed',42) if split_seed is None else split_seed
    if type(split_seed) is not int or not 0 <= split_seed <= 0xffffffff:
        raise ValueError('Unsigned split seed required')
    samples = observations(state)
    assignments = seed_split(samples,split_seed,False)['assignments'] if source == 'seeded' else {
        row['id']:('train' if i%2 == 0 else 'validation') if source == 'interleaved' else 'train' if row['x'] <= 0 else 'validation' if row['x'] <= .5 else 'test'
        for i,row in enumerate(samples)}
    result = {**deepcopy(state),'assignments':assignments,'source':source,'splitSeed':split_seed}
    capacity_split(result)
    return result

def polynomial_prediction(x, coefficients):
    if not coefficients or len(coefficients) > 9 or not is_finite_number(x) or not all(is_finite_number(value) for value in coefficients):
        raise ValueError('Finite input and 1–9 coefficients required')
    value = 0.0
    for coefficient in reversed(coefficients):
        value = value*x+coefficient
        if not math.isfinite(value):
            raise ValueError('Polynomial evaluation overflowed')
    return value

def fit_capacity(samples, degree):
    if type(degree) is not int or not 0 <= degree <= 8 or len(samples) > 64 or any(not is_finite_number(row['x']) or not is_finite_number(row['y']) or abs(row['x']) > 1 or abs(row['y']) > 10 for row in samples):
        raise ValueError('Invalid bounded polynomial data or degree')
    if len({row['id'] for row in samples}) != len(samples) or any(not isinstance(row['id'],str) or not row['id'].strip() for row in samples):
        raise ValueError('Unique nonempty sample IDs required')
    if not samples:
        return dict(coefficients=None, reason='The training set is empty: coefficients and baseline are undefined.')
    d = degree+1
    if len({row['x'] for row in samples}) < d:
        return dict(coefficients=None,reason='Too few distinct training inputs for a unique polynomial coefficient vector.')
    # NumPy holds the column matrix; the same explicit reorthogonalized QR policy runs in the browser.
    X = np.vander(np.array([row['x'] for row in samples]), N=d, increasing=True)
    y = np.array([[row['y']] for row in samples],dtype=np.float64)
    cutoff = RANK_THRESHOLD*math.hypot(*X.T.flat)
    Q, R = [], np.zeros((d,d),dtype=np.float64)
    for j in range(d):
        v = X[:,j].copy()
        for _ in range(2):
            for k in range(j):
                projection = sum(float(Q[k][i])*float(v[i]) for i in range(len(samples)))
                R[k,j] += projection
                for i in range(len(samples)):
                    v[i] -= projection*Q[k][i]
        norm = math.hypot(*v)
        if not math.isfinite(norm) or norm <= cutoff:
            return dict(coefficients=None,reason=RANK_REASON)
        R[j,j] = norm
        Q.append(v/norm)
    projected = [sum(float(column[i])*float(y[i,0]) for i in range(len(samples))) for column in Q]
    coefficients = np.zeros(d,dtype=np.float64)
    for j in range(d-1,-1,-1):
        value = projected[j]
        for k in range(j+1,d):
            value -= R[j,k]*coefficients[k]
        coefficients[j] = value/R[j,j]
    if not np.isfinite(coefficients).all():
        raise ValueError('Polynomial fit is non-finite')
    return dict(coefficients=coefficients.tolist(),reason=None)

def derive_capacity(state):
    samples = capacity_data(state)
    split = derive_split(capacity_split(state))
    training = [row for row in samples if row['partition'] == 'train']
    fit, mean = fit_capacity(training,state['degree']), split['baselineMean']
    rows = []
    for row in samples:
        prediction = polynomial_prediction(row['x'],fit['coefficients']) if fit['coefficients'] is not None else None
        rows.append(dict(**row,prediction=prediction,residual=None if prediction is None else prediction-row['y'],
                         squaredError=None if prediction is None else (prediction-row['y'])**2,baseline=mean,
                         baselineSquaredError=None if mean is None else (mean-row['y'])**2))
    errors = []
    for partition in PARTITIONS:
        data = [row for row in rows if row['partition'] == partition]
        errors.append(dict(partition=partition,count=len(data),
                           mse=float(np.mean([row['squaredError'] for row in data])) if fit['coefficients'] is not None and data else None,
                           baselineMSE=float(np.mean([row['baselineSquaredError'] for row in data])) if mean is not None and data else None,
                           reason=f'No {partition} samples: MSE is undefined.' if not data else fit['reason']))
    curve = [dict(x=-1+i/40,prediction=polynomial_prediction(-1+i/40,fit['coefficients']) if fit['coefficients'] is not None else None,
                  truth=1+2*(-1+i/40)-(-1+i/40)**2,baseline=mean) for i in range(81)]
    provenance = f"Hand-authored x grid and quadratic signal; bounded synthetic noise uses seed {state['seed']}. Assignments: {state.get('source','interleaved')}. Interleaved reset trains even grid indices (p1, p3, …); validates odd indices (p2, p4, …). Split seed {state.get('splitSeed',42)} is used only by seeded repartitioning."
    return dict(state=deepcopy(state),samples=samples,rows=rows,coefficients=fit['coefficients'],reason=fit['reason'],baselineMean=mean,errors=errors,curve=curve,
                trainingIds=[row['id'] for row in training],split=split,
                shapes=dict(X=[len(training),state['degree']+1],weights=[state['degree']+1,1],targets=[len(training),1]),provenance=provenance)

def capacity_candidates(state):
    candidates = []
    for degree in range(9):
        result = derive_capacity({**deepcopy(state),'degree':degree})
        candidates.append(dict(degree=degree,trainingMSE=result['errors'][0]['mse'],validationMSE=result['errors'][1]['mse']))
    finite = [row for row in candidates if row['validationMSE'] is not None]
    best = finite[0] if finite else None
    for row in finite[1:]:
        if row['validationMSE'] < best['validationMSE']-1e-12:
            best = row
    return dict(candidates=candidates,selectedDegree=None if best is None else best['degree'],
                reason=None if best else 'No defined validation losses: capacity selection is undefined.')

def capacity_trace(state):
    active = derive_capacity(state)
    return [dict(index=i,state={**deepcopy(state),'degree':degree},
                 explanation=f"Degree {degree}: fit only training IDs {', '.join(active['trainingIds']) or '(empty)'}; preserve the displayed observations and memberships.")
            for i,degree in enumerate((0,2,8))]

if __name__ == '__main__':
    print(json.dumps(dict(fit=derive_capacity(CAPACITY_CANONICAL),selection=capacity_candidates(CAPACITY_CANONICAL)),indent=2,allow_nan=False))

An independent Fraction oracle solves exact moment equations by elimination. NumPy’s SVD least-squares solve is another check for well-conditioned fixture cases. Ordinary arithmetic uses $10^{-11}$ tolerance; capacity goldens use $10^{-8}(1+|v|)$ to account for monomial conditioning and shifted-grid extrapolation. Neither tolerance changes the rank decision.

Numerical limits and undefined states

The solver supports degrees $0$–$8$, at most $64$ observations, $|x|\le1$, $|y|\le10$, and unique nonempty IDs. The lab’s amplitude is $0$–$2$. No training observations means no coefficients or training mean. Fewer than $d+1$ distinct inputs means no unique coefficient vector.

Both runtimes reject a design when a reorthogonalized column norm is at most $10^{-12}\lVert X_T\rVert_F$. This is the specified QR policy, not a claim that every SVD implementation uses the same criterion. For the nearly coincident inputs $0.5-1.3\times10^{-12},0.5,0.5+1.3\times10^{-12}$ at degree one, both return an explicit rank reason. Exact arithmetic can still identify a line, but this bounded binary64 solver declines that unstable fit.

Replay compares degrees $0,2,8$ while preserving observations and memberships. Every control, coefficient, residual, curve and inspector changes to the selected full state; an edit begins there. Numeric entries reject invalid values. Given coordinates stay exact text; computed values are marked approximate. Plot axes use explicitly different scales, and off-plane predictions are clipped visually while their numeric residuals remain in the table.

Feature scaling can improve numerical conditioning, but it changes the representation and must be fitted on training only when learned from data. This lab bounds $x$ to a normalized toy grid; it is not a general-purpose high-degree fitting tool. Detailed cross-validation and calibration belong to Part VIII.

Check your understanding

Checkpoints

You can change memberships and capacity in one authoritative experiment, compare individual residuals and honest baselines, distinguish a training fit from held-out evidence, and recognize nonidentifiability and numerical rank failures. Model capacity, data availability and evaluation design jointly determine what a reported score can mean.