III-29 · High-Dimensional Data and Covariance

Move the cloud, then measure its spread

Separate location from variation by centering every observation and inspecting the products that build a covariance matrix.

Chapters in Part VI
  1. III-29 · High-Dimensional Data and Covariance
  2. III-30 · PCA: Eigenvectors and Explained Variance
  3. III-31 · PCA Projection and Reconstruction

Part VI · inspectable two-feature PCA

Center the cloud; inspect every product

Hand-authored synthetic observations; no random draws or seed. Edits change the displayed cloud. Fit and reconstruction use these same rows.

PCA data, principal lines and reconstruction planeFilled circles are observations. The plus is the mean. Solid and dashed principal lines pass through the mean; hollow squares are reconstructions, connected by residual segments. Equal units on both axes. Drag a point or focus its handle and use arrow keys.
● Observation · + Mean · □ Reconstruction · residual segment · solid PC1 / dashed PC2. Lines have arbitrary signs. Repeated-eigenvalue lines show one conventional basis.

Drag any point; arrows move 0.1, Shift + arrow 0.5. Pointer edits snap to 0.1 and stop at ±5. Numeric entries retain finer precision. Overlapping points remain available in the point selector; its chosen handle is brought to the front. Scroll the page outside the handles.

Covariance and retained space
Observation editor

Axes are unoriented lines: v and −v describe the same component. Signs are display conventions.

Frame 1 / 1. Edits begin at the displayed frame.

Input X / centered XcSamples in rows; observations are column vectors in equations. No labels, intercept or standardization.
4 × 2 / 4 × 2
Mean μ2 × 1; axes pass through this mean in source coordinates.
[0, 0]
Covariance Csample divisor 3.
2 × 2
Basis Vk / scores Zk=0 gives empty scores and mean-only reconstruction.
2 × 1 / 4 × 1
Ambiguity
Axes are unoriented lines: v and −v describe the same component. Signs are display conventions.
λ1
6
λ2
0.66667
Retained variance fraction
0.9
Reconstruction SSE
2
Per-observation MSE · SSE/n
0.5
Per-entry MSE · SSE/(2n)
0.25
Xc=X−1μT,X∈R4×2,μ=[00]X_c=X-\mathbf1\mu^T,\quad X\in\mathbb R^{4\times2},\quad\mu=\begin{bmatrix}0\\0\end{bmatrix}
C=XcTXc3,n−1=3C=\frac{X_c^TX_c}{3},\quad n-1=3
Cvj=λjvj,VTV=I,λ1≥λ2,k=1Cv_j=\lambda_jv_j,\quad V^TV=I,\quad\lambda_1\ge\lambda_2,\quad k=1
Z=XcVk∈R4×1,X^=ZVkT+1μTZ=X_cV_k\in\mathbb R^{4\times1},\quad\hat X=ZV_k^T+\mathbf1\mu^T
SSE=∥X−X^∥F2,MSE=SSE/4\mathrm{SSE}=\lVert X-\hat X\rVert_F^2,\quad\mathrm{MSE}=\mathrm{SSE}/4
Covariance C (2 × 2), ordered directions (2 × 1), and variance ratios
QuantityValueInterpretation
C row 1[3.33333, 2.66667]Variance x; covariance x,y
C row 2[2.66667, 3.33333]Covariance y,x; variance y
PC1[0.70711, 0.70711]λ ≈ 6; fraction 0.9
PC2[−0.70711, 0.70711]λ ≈ 0.66667; fraction 0.1
Actual row-batch values; products accumulate into C; residual = observation − reconstruction
IDx,yCentereddx²; dx·dy; dy²Scores (k)ReconstructionResidualSquared error
a[−2, −1][−2, −1]4; 2; 1[−2.12132][−1.5, −1.5][−0.5, 0.5]0.5
b[−1, −2][−1, −2]1; 2; 4[−2.12132][−1.5, −1.5][0.5, −0.5]0.5
c[1, 2][1, 2]1; 2; 4[2.12132][1.5, 1.5][−0.5, 0.5]0.5
d[2, 1][2, 1]4; 2; 1[2.12132][1.5, 1.5][0.5, −0.5]0.5

Start by dragging a filled observation. The table exposes centered products, the matrix measures the same cloud, and the lines and residuals derive from that single numerical state.

Move one point before naming the matrix

Move point a to x = −1 using the numeric editor, or select its handle and press Right ten times. Its centered x changes, but so do the centered x values of the other rows: the mean belongs to the entire cloud. The plus moves with that mean. A centered coordinate is not simply the original coordinate with its label changed.

Reset. Switch Covariance normalization from sample to population. The cloud, mean, component lines and reconstruction stay geometrically identical. Covariance entries and eigenvalues shrink by 3/4 because these four observations now use divisor 4 instead of 3. The retained fraction and reconstruction SSE do not change.

Try the line, isotropic, constant and singleton clouds. These are transparent hand-authored counterexamples, with no seed or randomness. The singleton has a mean but no sample covariance; switch to population to obtain its zero-spread descriptive covariance. Reset restores the four original observations and one retained component. Touch dragging, keyboard moves and numeric edits enter the same model; a replay edit starts from the frame you see.

Many features, one row per observation

An observation with d features is a mathematical column vector xi∈Rd×1x_i\in\mathbb R^{d\times1}. In a batch we store its transpose in a row:

X=[x1T⋮xnT]∈Rn×d,μ=1n∑i=1nxi∈Rd×1.X=\begin{bmatrix}x_1^T\\\vdots\\x_n^T\end{bmatrix}\in\mathbb R^{n\times d},\qquad \mu=\frac1n\sum_{i=1}^n x_i\in\mathbb R^{d\times1}.

A hundred measurements do not necessarily represent a hundred independent directions of variation. Height in metres and height in centimetres duplicate one degree of freedom. A nearly redundant sensor adds a small new direction. Data can occupy a low-dimensional region of a large feature space.

Our browser and canonical Python deliberately accept only dense n×2 batches, with 1–128 uniquely identified observations. That bounded core lets you inspect every product and direction. The n×d equations here explain the generalization; this Part does not implement an arbitrary-dimensional solver, missing-value imputation, sparse matrices, whitening or standardization.

Center before measuring spread

Subtract the same feature mean from every row:

Xc=X−1μT,(Xc)ij=xij−μj.X_c=X-\mathbf1\mu^T,\qquad (X_c)_{ij}=x_{ij}-\mu_j.

Each centered column sums to zero in exact arithmetic. The operation removes location while preserving pairwise differences. Translating every point by a fixed vector t changes the mean to μ+t and leaves Xc unchanged. It therefore leaves covariance unchanged. Moving only one point changes both location and variation.

Centering is a fitted preprocessing operation. In a predictive workflow, fit μ on training rows, then center validation or future rows with that same μ. Re-centering each future batch changes the coordinate system and leaks batch information into the transform. The opening lab uses one descriptive batch and does not claim a held-out evaluation.

Build covariance from literal products

Let q be the declared divisor: q=n for the empirical population convention, or q=n−1 for the sample convention. For q positive,

C=1qXcTXc∈Rd×d,Cjh=1q∑i=1n(xij−μj)(xih−μh).C=\frac1q X_c^T X_c\in\mathbb R^{d\times d},\qquad C_{jh}=\frac1q\sum_{i=1}^n (x_{ij}-\mu_j)(x_{ih}-\mu_h).

Diagonal entries are feature variances. Off-diagonal entries measure signed co-variation, not a distance and not a causal effect. Products of two centered coordinates are positive when both deviate in the same direction and negative when they deviate in opposite directions. Symmetry follows because multiplication commutes: Cjh=Chj.

For any direction v, vTCv=∥Xcv∥22/q≥0v^TCv=\|X_cv\|_2^2/q\ge0. Covariance is positive semidefinite, so its exact eigenvalues cannot be negative. This derivation connects the matrix directly to variance along directions; it is more informative than memorizing a matrix formula.

Sample normalization is the familiar unbiased covariance estimator under independent identically distributed sampling with finite second moments. Population normalization here describes the displayed empirical cloud. Neither choice turns a four-row synthetic cloud into evidence about a real population. With n greater than one, multiplying C by n/(n−1) changes eigenvalues but not its eigenvectors or explained-variance fractions.

Work the four-row cloud by hand

The original rows are (−2,−1), (−1,−2), (1,2), (2,1). Their column sums are zero, so μ=(0,0) and centering leaves them unchanged.

Rowdx²dx·dydy²
a421
b124
c124
d421
Sum10810

Hence the sample and population matrices are

Cs=[10/38/38/310/3],Cp=[5/2225/2].C_s=\begin{bmatrix}10/3&8/3\\8/3&10/3\end{bmatrix},\qquad C_p=\begin{bmatrix}5/2&2\\2&5/2\end{bmatrix}.

The sum of the diagonal entries, the trace, measures total feature variance: 20/3 under sample normalization and 5 under population normalization. Both describe the same geometry in different normalization units. The sample off-diagonal 8/3 is not a correlation coefficient; it need not lie between −1 and 1.

Units, degeneracy and arithmetic limits

If one feature is measured in metres and another in seconds, covariance entries carry mixed products of units. Euclidean PCA on raw numbers then embeds a choice of metric. Multiplying only one feature by 100 can rotate the preferred direction. Standardizing features is a separate fitted decision; this lab does not silently normalize scales. Read the distance-metric discussion in III-18 before treating variance as importance.

Centered data have rank at most min(d,n−1). A line cloud has rank at most one; a constant cloud has rank zero. At n=1, population covariance is the zero matrix, while sample covariance is undefined because q=0. Empty data do not define a mean and are rejected. The model validates dense shapes, IDs, finite coordinates and k even before these degenerate outcomes.

Execute the actual Python and NumPy

The imported canonical file implements the same bounded n×2 contract. It validates each coordinate before constructing the NumPy array, then forms the means, centered products, covariance, eigenpairs and reconstruction. The oracle separately forms exact Fraction products and compares NumPy SVD and symmetric eigh results; a library wrapper is not the teaching mechanism.

From the book root, a terminal invocation uses the original cloud and prints a JSON result:

.venv/bin/python -B code/python/p06/pca.py

For a pipe, provide a complete request with points (objects containing a unique id and two-entry x), normalization (sample or population), and integral numeric k from 0 to 2. No seed is accepted or needed. An invalid request exits with an error instead of manufacturing a covariance.

code/python/p06/pca.py
"""Inspectable two-feature PCA. X stores observations in rows; directions are columns.

No scaling or whitening. Sample normalization divides by n-1; population by n.
One sample has undefined sample covariance. Zero spread has undefined variance ratios.
"""
import json
import math
import sys
import numpy as np

CLOUD = [{"id": i, "x": x} for i, x in zip("abcd", [[-2, -1], [-1, -2], [1, 2], [2, 1]])]
DEFAULT = dict(points=CLOUD, normalization="sample", k=1)


def finite(value, label):
    if isinstance(value, (bool, np.bool_)) or not isinstance(value, (int, float, np.integer, np.floating)):
        raise ValueError(f"{label} must be a finite real number.")
    try:
        number = float(value)
    except (OverflowError, ValueError) as error:
        raise ValueError(f"{label} must be finite; binary64 conversion overflow is rejected.") from error
    if not math.isfinite(number):
        raise ValueError(f"{label} must be finite; arithmetic overflow is rejected.")
    return number + 0.0


def energy(values):
    total = 0.0
    for value in values:
        square = finite(float(value) * float(value), "Squared component")
        if value != 0 and square == 0:
            raise ValueError("Squared component underflow: rescale the observations.")
        total = finite(total + square, "Sum of squares")
    return total


def normalized_energy(values, divisor):
    total = energy(values)
    value = finite(total / divisor, "Normalized energy")
    if total > 0 and value == 0:
        raise ValueError("Variance underflow: rescale the observations.")
    return value


def pca(request):
    if not isinstance(request, dict):
        raise ValueError("PCA request must be an object.")
    normalization, k, points = request.get("normalization"), request.get("k"), request.get("points")
    if normalization not in ("sample", "population"):
        raise ValueError("Normalization must be sample or population.")
    k = finite(k, "Component count")
    if not k.is_integer() or not 0 <= k <= 2:
        raise ValueError("Component count must be an integer from 0 to 2.")
    k = int(k)
    if not isinstance(points, list) or not 1 <= len(points) <= 128:
        raise ValueError("Use 1 to 128 dense two-feature observations.")
    ids, coordinates = [], []
    for point in points:
        if not isinstance(point, dict) or not isinstance(point.get("id"), str) or not point["id"].strip() or point["id"] in ids:
            raise ValueError("Point IDs must be nonempty unique strings.")
        x = point.get("x")
        if not isinstance(x, list) or len(x) != 2:
            raise ValueError("Each observation must have exactly two dense components.")
        ids.append(point["id"])
        coordinates.append([finite(value, "Observation component") for value in x])
    X = np.asarray(coordinates, dtype=np.float64)  # n×2; boundary checked before degenerate returns
    n = len(points)
    mean = []
    for j in range(2):
        first = float(X[0, j])
        if all(float(x) == first for x in X[:, j]):
            mean.append(first)
        else:
            delta = 0.0
            for x in X[:, j]:
                delta = finite(delta + finite(float(x) - first, "Mean difference") / n, "Mean accumulation")
            mean.append(finite(first + delta, "Mean"))
    centered = np.asarray([[finite(float(x[j]) - mean[j], "Centering") for j in range(2)] for x in X])
    divisor = n - 1 if normalization == "sample" else n
    products = [[finite(float(x[0]) * float(x[0]), "Squared x"), finite(float(x[0]) * float(x[1]), "Cross product"), finite(float(x[1]) * float(x[1]), "Squared y")] for x in centered]
    basic = dict(n=n, divisor=divisor, mean=mean, centered=centered.tolist(), products=products)
    if divisor == 0:
        return dict(basic, status="undefined", reason="Sample covariance needs n ≥ 2; n−1 is zero.")
    a, c = (normalized_energy(centered[:, j], divisor) for j in range(2))
    b = 0.0
    for x in centered:
        b = finite(b + finite(float(x[0]) * float(x[1]), "Cross product"), "Covariance sum")
    b = finite(b / divisor, "Covariance")
    repeated, scale = a == c and b == 0, max(a, c, abs(b))
    v1, large = [1.0, 0.0], 0.0
    if scale > 0:
        A, B, C = a / scale, b / scale, c / scale
        L = (A + C) / 2 + math.hypot((A - C) / 2, B)
        large = finite(scale * L, "Largest eigenvalue")
        if b == 0:
            v1 = [1.0, 0.0] if a >= c else [0.0, 1.0]
        else:
            raw = [L - C, B] if A >= C else [B, L - A]
            norm = math.hypot(*raw)
            if norm == 0:
                raise ValueError("Eigenvector scaling underflow: direction is unresolved in binary64.")
            sign = -1 if raw[0] < 0 or (raw[0] == 0 and raw[1] < 0) else 1
            v1 = [sign * x / norm for x in raw]
    v2 = [-v1[1], v1[0]]
    axes = [v1, v2]  # serialized list of 2×1 columns, not a row-batch basis matrix
    def dot(x, v):
        return finite(finite(float(x[0]) * v[0], "Dot product") + finite(float(x[1]) * v[1], "Dot product"), "Dot product")
    scores = [[dot(x, v) for v in axes] for x in centered]
    small = normalized_energy([z[1] for z in scores], divisor)
    if small > large:  # last-bit ordering in a nearly repeated spectrum
        large, small = small, large
        axes = [v2, v1]
        scores = [[z[1], z[0]] for z in scores]
        v1 = axes[0]
    eigenvalues = [large, small]
    total = finite(large + small, "Total variance")
    ratios = None if total == 0 else [v / total for v in eigenvalues]
    rows = []
    for i, x in enumerate(X):
        z = scores[i]
        # Full-space identity is applied directly, avoiding needless rotate/unrotate roundoff.
        reconstructed = x.tolist() if k == 2 else [finite(mean[j] + (finite(z[0] * v1[j], "Reconstruction product") if k else 0), "Reconstruction") for j in range(2)]
        residual = [finite(float(x[j]) - reconstructed[j], "Residual") for j in range(2)]
        rows.append(dict(id=ids[i], x=x.tolist(), centered=centered[i].tolist(), scores=z[:k], fullScores=z,
                         reconstructed=reconstructed, residual=residual, squaredError=energy(residual)))
    sse = 0.0
    for row in rows:
        sse = finite(sse + row["squaredError"], "Reconstruction SSE")
    return dict(basic, status="defined", covariance=[[a, b], [b, c]], axes=axes, eigenvalues=eigenvalues,
                totalVariance=total, ratios=ratios, retained=None if ratios is None else sum(ratios[:k]),
                repeated=repeated, nearRepeated=not repeated and abs(large - small) <= 64 * sys.float_info.epsilon * total,
                zeroVariance=total == 0, rows=rows, sse=sse, mse=sse / n, entryMse=sse / (2 * n))


if __name__ == "__main__":
    try:
        request = DEFAULT if sys.stdin.isatty() else json.load(sys.stdin)
        print(json.dumps(pca(request), allow_nan=False))
    except (ValueError, TypeError, OverflowError) as error:
        print(json.dumps({"error": str(error)}), file=sys.stderr)
        sys.exit(2)

Five graded investigations

X contains 7 observations and 5 features. What are the shapes of μ and C?
Add the same vector (3,−1) to every point. What changes?
For the original four rows, what is sample covariance between x and y?
One finite observation is available. Which covariance statement is valid?
Why can a 100-feature centered dataset with only 4 rows be rank deficient?

What you should now be able to explain

You can separate translation from spread, write the n×d and d×d shapes, build every covariance entry from centered products, choose a normalization explicitly, and explain why small or redundant batches are rank deficient. III-30 uses this same covariance to find directions; III-31 measures what projection discards.