III-30 · PCA: Eigenvectors and Explained Variance
Find the line with the most spread
Use covariance eigengeometry to choose orthogonal directions, then separate explained variation from prediction quality.
Part VI · inspectable two-feature PCA
Spread chooses a line, not an arrow
Hand-authored synthetic observations; no random draws or seed. Edits change the displayed cloud. Fit and reconstruction use these same rows.
Drag any point; arrows move 0.1, Shift + arrow 0.5. Pointer edits snap to 0.1 and stop at ±5. Numeric entries retain finer precision. Overlapping points remain available in the point selector; its chosen handle is brought to the front. Scroll the page outside the handles.
Axes are unoriented lines: v and −v describe the same component. Signs are display conventions.
Frame 1 / 1. Edits begin at the displayed frame.
- Input X / centered XcSamples in rows; observations are column vectors in equations. No labels, intercept or standardization.
- 4 × 2 / 4 × 2
- Mean μ2 × 1; axes pass through this mean in source coordinates.
- [0, 0]
- Covariance Csample divisor 3.
- 2 × 2
- Basis Vk / scores Zk=0 gives empty scores and mean-only reconstruction.
- 2 × 1 / 4 × 1
- Ambiguity
- Axes are unoriented lines: v and −v describe the same component. Signs are display conventions.
- λ1
- 6
- λ2
- 0.66667
- Retained variance fraction
- 0.9
- Reconstruction SSE
- 2
- Per-observation MSE · SSE/n
- 0.5
- Per-entry MSE · SSE/(2n)
- 0.25
| Quantity | Value | Interpretation |
|---|---|---|
| C row 1 | [3.33333, 2.66667] | Variance x; covariance x,y |
| C row 2 | [2.66667, 3.33333] | Covariance y,x; variance y |
| PC1 | [0.70711, 0.70711] | λ ≈ 6; fraction 0.9 |
| PC2 | [−0.70711, 0.70711] | λ ≈ 0.66667; fraction 0.1 |
| ID | x,y | Centered | dx²; dx·dy; dy² | Scores (k) | Reconstruction | Residual | Squared error |
|---|---|---|---|---|---|---|---|
| a | [−2, −1] | [−2, −1] | 4; 2; 1 | [−2.12132] | [−1.5, −1.5] | [−0.5, 0.5] | 0.5 |
| b | [−1, −2] | [−1, −2] | 1; 2; 4 | [−2.12132] | [−1.5, −1.5] | [0.5, −0.5] | 0.5 |
| c | [1, 2] | [1, 2] | 1; 2; 4 | [2.12132] | [1.5, 1.5] | [−0.5, 0.5] | 0.5 |
| d | [2, 1] | [2, 1] | 4; 2; 1 | [2.12132] | [1.5, 1.5] | [0.5, −0.5] | 0.5 |
Ask the cloud for a direction
In the original cloud, PC1 follows the line y=x. Drag point a upward and inspect the off-diagonal covariance, the solid line and both eigenvalues. The direction changes because its variance is recomputed from the edited observations; this is not a decorative rotation.
Reset. Select the isotropic cloud. Its spread is equal in every direction. The lab shows a coordinate basis and explicitly states that it is conventional; the picture does not establish a unique horizontal first component. With the constant cloud, both eigenvalues are zero and the explained-variance fractions are undefined. “0/0” is not a retained fraction of zero or one.
Select line cloud, then retain one component. The cloud lies in a one-dimensional span. In exact arithmetic one eigenvalue is positive and the other is zero. Numerical output may contain a very small residual from a nonexact floating direction; a tolerance used in a test is not permission to erase a positive eigenvalue in the model.
PCA chooses an orthonormal coordinate system
For centered row-batch Xc and covariance C=XcᵀXc/q, a unit column direction v produces projected scalar scores z=Xcv. Their variance under the same divisor q is
The unit constraint is essential: multiplying a direction by ten multiplies its score variance by a hundred without finding a better line. PCA maximizes this variance over unit directions. It then repeats inside orthogonal complements, so later components do not duplicate earlier directions.
For real symmetric C, an orthonormal eigenbasis exists:
V has d×d shape, with directions in its columns. Covariance is positive semidefinite because every directional variance is nonnegative. These directions describe variation in the supplied metric, not class boundaries, causes or supervised feature importance. PCA needs no target labels.
Derive why the maximum is an eigenvector
Use a Lagrange multiplier for the unit constraint:
At a stationary direction, Cv=λv, and multiplying by vᵀ shows its variance is λ. To find the maximum, write a unit vector in the orthonormal eigenbasis: v=Va with aᵀa=1. Then
The bound is attained by a leading eigenvector. Restricting to vectors orthogonal to v1 removes its coordinate and leaves λ2 as the next maximum. Distinct eigenvalues determine distinct lines, up to sign. If the largest eigenvalue repeats, any unit vector in its eigenspace attains the same maximum.
Work the original eigenpairs literally
For the sample covariance from III-29,
Multiplying gives Cv1=6v1 and Cv2=(2/3)v2. Their dot product is zero and each norm is one. The eigenvalue sum is 6+2/3=20/3, equal to the covariance trace. Population normalization multiplies both eigenvalues by 3/4: 4.5 and 0.5. The directions are unchanged.
Define explained-variance fractions only when the total is positive:
Here r1=6/(20/3)=0.9 and r2=0.1. One component retains 90% of the empirical feature variation; it does not give 90% prediction accuracy. Zero components retain zero when there is positive spread; all components retain one. For a zero matrix, the denominator is zero and every ratio is undefined.
| Covariance convention | λ1 | λ2 | Retained with k=1 |
|---|---|---|---|
| Sample, q=3 | 6 | 2/3 | 0.9 |
| Population, q=4 | 4.5 | 0.5 | 0.9 |
| Isotropic preset, population | 2 | 2 | 0.5; direction ambiguous |
| Constant preset | 0 | 0 | Undefined; no spread |
Signs and repeated eigenvalues are not bugs
If Cv=λv, then C(−v)=λ(−v). Flipping a component flips its scores but leaves the reconstructed points unchanged because both factors change sign. The browser chooses a reproducible display sign for the leading eigenvector and obtains the perpendicular second vector. Another library may choose different signs; raw vector equality is the wrong parity test. Compare the projector vvᵀ or the absolute inner product for a simple eigenvalue.
For C=αI, all directions have the same variance. A one-component reconstruction depends on the chosen line, but its total discarded energy is the same for every line. More generally, if a repeated eigenspace straddles the retained/discarded boundary, there is no unique rank-k projector. If you retain an entire repeated eigenspace, its projector is unique even though the basis within it can rotate. Our oracle compares orthogonality, eigen relations and optimal SSE in the ambiguous case, without requiring NumPy to return the browser’s basis.
Near a repeated spectrum, a tiny perturbation can rotate the vectors substantially. The lab warns when the computed gap is within 64 machine epsilons of total variance. This is a conditioning warning; it does not merge eigenvalues, choose a unique axis, or zero out variation. Exact computed equality is labeled separately.
Calculate the small eigenvalue without subtracting large equals
For a symmetric two-by-two covariance , the formal eigenvalues are
The plus expression is well behaved after scaling by the largest entry magnitude and using hypot. The minus expression can lose all small positive variation through cancellation. For diagonal C=diag(0.5,5×10⁻¹⁷), both large terms can round to 0.25, making their difference zero.
This kernel uses the scaled plus formula to obtain a stable leading direction, choosing the longer of two algebraic eigenvector formulas. It obtains the minor eigenvalue from centered data directly:
This sums nonnegative small quantities instead of subtracting two large quantities. No tolerance clamps a small positive value to zero. Last-bit ordering is restored if directional energy rounds above the computed leading value in a nearly repeated spectrum. Input dimensions, IDs and all finite arithmetic are checked before any zero-spread return.
Execute the actual Python and NumPy
The imported reference is actual source. pca(request) returns means, centered rows, covariance, eigenvectors, eigenvalues and fractions, followed by projection and reconstruction for the chosen k. axes serializes a list of direction columns; form a matrix with these vectors as columns before performing XcV.
.venv/bin/python -B code/python/p06/pca.py
That terminal command uses the original hand-authored request. Pipes require the full JSON request described in III-29. NumPy integer or finite integral numeric k values, including JSON 1.0, are accepted; booleans, nonintegral numbers and nonfinite values are rejected. Inspect oracle/p06/independent.py for the separate SVD/eigh method rather than interpreting Python–TypeScript agreement alone as independence.
"""Inspectable two-feature PCA. X stores observations in rows; directions are columns.
No scaling or whitening. Sample normalization divides by n-1; population by n.
One sample has undefined sample covariance. Zero spread has undefined variance ratios.
"""
import json
import math
import sys
import numpy as np
CLOUD = [{"id": i, "x": x} for i, x in zip("abcd", [[-2, -1], [-1, -2], [1, 2], [2, 1]])]
DEFAULT = dict(points=CLOUD, normalization="sample", k=1)
def finite(value, label):
if isinstance(value, (bool, np.bool_)) or not isinstance(value, (int, float, np.integer, np.floating)):
raise ValueError(f"{label} must be a finite real number.")
try:
number = float(value)
except (OverflowError, ValueError) as error:
raise ValueError(f"{label} must be finite; binary64 conversion overflow is rejected.") from error
if not math.isfinite(number):
raise ValueError(f"{label} must be finite; arithmetic overflow is rejected.")
return number + 0.0
def energy(values):
total = 0.0
for value in values:
square = finite(float(value) * float(value), "Squared component")
if value != 0 and square == 0:
raise ValueError("Squared component underflow: rescale the observations.")
total = finite(total + square, "Sum of squares")
return total
def normalized_energy(values, divisor):
total = energy(values)
value = finite(total / divisor, "Normalized energy")
if total > 0 and value == 0:
raise ValueError("Variance underflow: rescale the observations.")
return value
def pca(request):
if not isinstance(request, dict):
raise ValueError("PCA request must be an object.")
normalization, k, points = request.get("normalization"), request.get("k"), request.get("points")
if normalization not in ("sample", "population"):
raise ValueError("Normalization must be sample or population.")
k = finite(k, "Component count")
if not k.is_integer() or not 0 <= k <= 2:
raise ValueError("Component count must be an integer from 0 to 2.")
k = int(k)
if not isinstance(points, list) or not 1 <= len(points) <= 128:
raise ValueError("Use 1 to 128 dense two-feature observations.")
ids, coordinates = [], []
for point in points:
if not isinstance(point, dict) or not isinstance(point.get("id"), str) or not point["id"].strip() or point["id"] in ids:
raise ValueError("Point IDs must be nonempty unique strings.")
x = point.get("x")
if not isinstance(x, list) or len(x) != 2:
raise ValueError("Each observation must have exactly two dense components.")
ids.append(point["id"])
coordinates.append([finite(value, "Observation component") for value in x])
X = np.asarray(coordinates, dtype=np.float64) # n×2; boundary checked before degenerate returns
n = len(points)
mean = []
for j in range(2):
first = float(X[0, j])
if all(float(x) == first for x in X[:, j]):
mean.append(first)
else:
delta = 0.0
for x in X[:, j]:
delta = finite(delta + finite(float(x) - first, "Mean difference") / n, "Mean accumulation")
mean.append(finite(first + delta, "Mean"))
centered = np.asarray([[finite(float(x[j]) - mean[j], "Centering") for j in range(2)] for x in X])
divisor = n - 1 if normalization == "sample" else n
products = [[finite(float(x[0]) * float(x[0]), "Squared x"), finite(float(x[0]) * float(x[1]), "Cross product"), finite(float(x[1]) * float(x[1]), "Squared y")] for x in centered]
basic = dict(n=n, divisor=divisor, mean=mean, centered=centered.tolist(), products=products)
if divisor == 0:
return dict(basic, status="undefined", reason="Sample covariance needs n ≥ 2; n−1 is zero.")
a, c = (normalized_energy(centered[:, j], divisor) for j in range(2))
b = 0.0
for x in centered:
b = finite(b + finite(float(x[0]) * float(x[1]), "Cross product"), "Covariance sum")
b = finite(b / divisor, "Covariance")
repeated, scale = a == c and b == 0, max(a, c, abs(b))
v1, large = [1.0, 0.0], 0.0
if scale > 0:
A, B, C = a / scale, b / scale, c / scale
L = (A + C) / 2 + math.hypot((A - C) / 2, B)
large = finite(scale * L, "Largest eigenvalue")
if b == 0:
v1 = [1.0, 0.0] if a >= c else [0.0, 1.0]
else:
raw = [L - C, B] if A >= C else [B, L - A]
norm = math.hypot(*raw)
if norm == 0:
raise ValueError("Eigenvector scaling underflow: direction is unresolved in binary64.")
sign = -1 if raw[0] < 0 or (raw[0] == 0 and raw[1] < 0) else 1
v1 = [sign * x / norm for x in raw]
v2 = [-v1[1], v1[0]]
axes = [v1, v2] # serialized list of 2×1 columns, not a row-batch basis matrix
def dot(x, v):
return finite(finite(float(x[0]) * v[0], "Dot product") + finite(float(x[1]) * v[1], "Dot product"), "Dot product")
scores = [[dot(x, v) for v in axes] for x in centered]
small = normalized_energy([z[1] for z in scores], divisor)
if small > large: # last-bit ordering in a nearly repeated spectrum
large, small = small, large
axes = [v2, v1]
scores = [[z[1], z[0]] for z in scores]
v1 = axes[0]
eigenvalues = [large, small]
total = finite(large + small, "Total variance")
ratios = None if total == 0 else [v / total for v in eigenvalues]
rows = []
for i, x in enumerate(X):
z = scores[i]
# Full-space identity is applied directly, avoiding needless rotate/unrotate roundoff.
reconstructed = x.tolist() if k == 2 else [finite(mean[j] + (finite(z[0] * v1[j], "Reconstruction product") if k else 0), "Reconstruction") for j in range(2)]
residual = [finite(float(x[j]) - reconstructed[j], "Residual") for j in range(2)]
rows.append(dict(id=ids[i], x=x.tolist(), centered=centered[i].tolist(), scores=z[:k], fullScores=z,
reconstructed=reconstructed, residual=residual, squaredError=energy(residual)))
sse = 0.0
for row in rows:
sse = finite(sse + row["squaredError"], "Reconstruction SSE")
return dict(basic, status="defined", covariance=[[a, b], [b, c]], axes=axes, eigenvalues=eigenvalues,
totalVariance=total, ratios=ratios, retained=None if ratios is None else sum(ratios[:k]),
repeated=repeated, nearRepeated=not repeated and abs(large - small) <= 64 * sys.float_info.epsilon * total,
zeroVariance=total == 0, rows=rows, sse=sse, mse=sse / n, entryMse=sse / (2 * n))
if __name__ == "__main__":
try:
request = DEFAULT if sys.stdin.isatty() else json.load(sys.stdin)
print(json.dumps(pca(request), allow_nan=False))
except (ValueError, TypeError, OverflowError) as error:
print(json.dumps({"error": str(error)}), file=sys.stderr)
sys.exit(2)Five graded investigations
Connect the spectrum to the next task
You can derive the variance maximum, verify eigenpair shapes, compute retained fractions, and distinguish sign, repeated-basis and near-repeat ambiguity. Book II’s eigenvectors and SVD supply the geometry. III-31 now uses these directions to compress data and measure the actual reconstruction error; a large retained fraction remains an unsupervised quantity.