III-27 · Initialization, Convergence, Scaling and Failure Cases
A stable center can still be the wrong destination
Compare two fixed points on the same rectangle, inspect an empty cluster, and change the units that determine nearest-center decisions.
Distance → assignment → mean
Hand-authored, unitless toy observations. Labels identify groups; they have no numerical order.
Filled circles: observations. Hollow diamonds: centroids. Dashed links use held labels. Drag snaps to 0.1; arrow keys preserve full input precision. Only handles capture touch. The selector brings an overlapping handle forward; Enter or Space does the same for a focused handle.
Frame 1 / 1 · edit begins at the displayed state.
- X · samples in rowsEach mathematical feature vector is a 2 × 1 column.
- 4 × 2
- Centroids / squared distances / assignments
- 2 × 2 / 4 × 2 / 4 × 1
- Held-assignment WCSSNearest-label preview; press Assign to commit.
- ≈ 8
- Next nearest-assignment WCSSAllows labels to change; same current centroids.
- ≈ 8
- Mean update · current → proposed WCSSHeld assignments; empty centroids retained. Inspect before applying Update means.
- ≈ 8 → ≈ 4
- Phase and stopNot stopped; replay is capped at 40 half-steps.
- ready · 0 mean updates
- Empty-cluster policyEmpty mean is undefined. Retained coordinates are a policy, not a computed empty mean.
- Retain previous centroid
- Input provenance / initializationSeed affects partial Fisher–Yates observation selection only. Current centroids may be edited.
- Hand-authored observations / seed 7
| Observation | x₁ | x₂ | d² to c0 | d² to c1 | Held label | Nearest / tie |
|---|---|---|---|---|---|---|
| s1 | -3 | 0 | ≈ 0 | ≈ 16 | 0 | 0 / unique |
| s2 | -1 | 0 | ≈ 4 | ≈ 4 | 0 | 0 / tie |
| s3 | 1 | 0 | ≈ 16 | ≈ 0 | 1 | 1 / unique |
| s4 | 3 | 0 | ≈ 36 | ≈ 4 | 1 | 1 / unique |
| Cluster | IDs / n | Sum | Mean / policy | Current centroid |
|---|---|---|---|---|
| c0 | s1, s2 / 2 | [-4, 0] | ≈ [-2, 0] | ≈ [-3, 0] |
| c1 | s3, s4 / 2 | [4, 0] | ≈ [2, 0] | ≈ [1, 0] |
Stop does not mean best
Load Horizontal local minimum, press Assign nearest, and inspect the proposed means. Press Update means. The centers remain (0,−1) and (0,1), and WCSS stays 16. The sheet honestly reports a fixed point. Now load Vertical groups and perform the same two operations: centers (−2,0) and (2,0) also stay fixed, but WCSS is 4.
Both results use exactly the four points (−2,−1), (−2,1), (2,−1), (2,1). The difference is the starting centers, not the dataset or a fabricated convergence trace. Drag a center away from either configuration and inspect how the nearest-center boundary changes. A manually moved center is a new experiment; monotonicity applies to Lloyd steps on fixed data and scales.
Derive the two costs
For horizontal groups, each point is two units horizontally from its group center. Each squared distance is 4; four points give . For vertical groups, every point is one unit vertically from its center, so .
For the same rectangle with positive reference scales , these fixed groupings have costs
With , vertical groups are better. With , horizontal groups cost 0.16 while vertical groups cost 4. This is a change of the model’s metric. It need not preserve what “nearby” means.
For this four-row example only, an independent oracle enumerates all seven unlabeled nonempty two-cluster partitions. The singleton/triple splits have WCSS , the diagonal pairing costs 20, horizontal costs 16, and vertical costs 4. Thus 4 is the exact global two-cluster minimum for these unit-scale observations. Lloyd’s stop flag does not perform that enumeration, and no global guarantee is exported to larger data.
Monotonicity has a scope
The assignment step minimizes each row’s cost given the centers; the nonempty mean update minimizes each cluster’s cost given its membership. Retaining an empty center changes no assigned cost. Therefore each half-step has non-increasing WCSS in exact arithmetic.
This explains a descent property, not a promise of a unique solution. At the horizontal rectangle configuration, every point is strictly closer to its assigned horizontal center, and those centers are already the means. There is no improving Lloyd half-step even though another grouping is better. Ties, empty centers and coincident data can also make assignments or centers non-unique.
The implementation stops only when every updated center exactly equals its old binary64 coordinates. It reports that condition rather than declaring mathematical convergence to infinite precision. Replay is capped at 40 half-steps; a trace reaching that cap without the stop flag is unfinished, not converged. The pure trace helper supports up to 100 half-steps. It returns no partial history if an arithmetic failure occurs.
A seed is a reproducible choice of rows
Our initialization samples k distinct observation indices without replacement using a partial Fisher–Yates permutation. It does not implement k-means++, estimate a global optimum, or forbid coincident coordinates. At step i, draw a number u from the frozen platform LCG, choose index j=i+floor(u(n−i)), exchange positions i and j, and retain the first k observation rows as initial centers.
The integer generator is
Seeds are unsigned 32-bit integers. There is one draw per selected center. Seed 0 is valid. At seed 0 the first two generator integers are 1013904223 and 1196435762. For the original four-point row order, the first selected index is 0; the second is 1. Centers are therefore (−3,0) and (−1,0). Seed 7 selects indices 0 and 3, giving centers (−3,0) and (3,0).
Change Seed and press Initialize from seed to make that choice concrete. Merely typing a seed does not replace already edited centers. Changing k deliberately initializes k new centers using the current seed. Repeat seeds to compare runs; a favorable objective is evidence about this objective on these data, not about a natural category structure.
Row order is part of the deterministic experiment. Reordering observations while keeping the same seed can choose different coordinates. Stable unique IDs preserve identity; IDs themselves are not features and are not used as random keys. The independent oracle evaluates a closed-form modular geometric series for the generator rather than translating its recurrence.
Units change squared importance
Suppose an observation’s first coordinate is 2 metres from a center. With reference scale 1 metre, its normalized difference is 2 and squared contribution 4. Express the same difference as 200 centimetres and the same reference as 100 centimetres: the ratio is still 2. Changing both together preserves the metric. Converting the coordinate but leaving its numerical reference at 1 instead makes that feature’s squared contribution 10,000 times larger.
The sheet keeps raw coordinates in the plot and divides by the explicit Scale x/y in the numerical model. Larger reference scales reduce a feature’s importance. If scales are estimated from real data, estimate them on the training design and carry the same transformation into later observations. Outliers, constant features, missing values and inappropriate feature units need deliberate preprocessing; this toy sheet does not infer a pipeline.
An objective expressed under one scale cannot be compared numerically with a differently scaled objective as if it were the same unit. Comparing k on the same representation also needs care because training WCSS cannot increase when additional unconstrained centers are allowed.
Empty clusters and repeated observations stay visible
Load Empty cluster. Two points at (−1,0) and (1,0) and two centers at (0,0) create exact ties. Both rows choose label 0, whose mean is (0,0). Cluster 1 has no rows. It keeps its old center; its mean is explicitly undefined. WCSS remains 2, and the mean update stops without inventing a second group.
Load Coincident observations. Three distinct IDs all occupy (1,1), with k=3 and three coincident initial centers. Every distance is zero, every tie chooses label 0, and two clusters stay empty. k≤n is necessary for this initialization, but it does not guarantee k nonempty groups when distinct rows share coordinates. Removing duplicate IDs from the dataset would change its empirical weights; coordinate coincidence alone cannot justify removing them.
| Situation | Defined numerical result | Interpretation |
|---|---|---|
| k=1 | One coordinatewise mean | WCSS summarizes total scatter |
| k>n or k=0 | Rejected with reason | No silent clamping of k |
| Duplicate coordinates, distinct IDs | Separate weighted observations | Can produce empty centers |
| Empty cluster | Undefined mean; retain old center | No hidden reseeding |
| Horizontal rectangle fixed point | WCSS 16 | Worse than another fixed point |
| Nonzero squared-distance underflow | Rejected; rescale | No manufactured zero-distance tie |
Evaluate partitions, not arbitrary label names
Assignments [0,0,1,1] and [1,1,0,0] describe the same grouping. They have identical within/between membership relations, even though a literal elementwise “accuracy” would call every label different. Our same_partition checks whether every pair of observations is together in both assignments. It is invariant to a one-to-one permutation of occupied labels.
This helper is a partition-equivalence predicate, not an external accuracy score, adjusted Rand index or semantic ground truth. It accepts nonnegative cluster labels. DBSCAN’s −1 noise marker is handled separately; grouping all noise points into one true cluster would introduce a different evaluation convention. Real external labels require an explicitly defined metric and treatment of noise or unmatched groups.
Numerical caveats and connections
The interactive coordinates stay within −5…5 and reference scales within 0.1…10; these are educational bounds, not model theory. Numeric edits reject values beyond them without replacing the last valid state. Numeric, pointer, touch, keyboard and reset paths share the same validated model. Editing during replay starts from the displayed frozen state and clears the old trace.
III-26 established the two subproblems. III-28 will question whether squared distance to a mean is even the right definition of a cluster. Book II’s local/global optima and numerical stopping chapters explain why reproducibility, stability and optimality are separate questions.
Read the canonical reference
initialize, lloyd_trace, same_partition and the shape/arithmetic validators below are the actual Python source, not a prose copy. The default executable prints the first mean update; fixture inputs can be sent as JSON on stdin to select other operations.
.venv/bin/python -B code/python/p05/clustering.py
"""Canonical from-scratch NumPy clustering. X is n×2; centers are k×2.
No target column, fitting library, hidden reseeding, or optimum guarantee.
"""
import json
import math
import sys
from copy import deepcopy
import numpy as np
def number(v, name, lo=None, hi=None, integer=False):
try:
valid = isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(float(v))
except (OverflowError, TypeError):
valid = False
if not valid:
raise ValueError(f"{name} must be a finite number.")
v = float(v)
if (lo is not None and v < lo) or (hi is not None and v > hi) or (integer and v != math.floor(v)):
raise ValueError(f"{name} is outside its supported range.")
return int(v) if integer else v
def vector(v, name):
if not isinstance(v, (list, tuple)) or len(v) != 2:
raise ValueError(f"{name} needs a dense two-feature row.")
return [number(x, name) for x in v]
def observations(points):
if not isinstance(points, list) or not 1 <= len(points) <= 32:
raise ValueError("Use 1–32 observations; empty data have no objective.")
ids, rows = set(), []
for p in points:
if not isinstance(p, dict) or not isinstance(p.get("id"), str) or not p["id"].strip() or p["id"] in ids:
raise ValueError("Observation IDs must be unique nonempty strings.")
ids.add(p["id"])
rows.append(vector(p.get("x"), "Observation"))
return np.asarray(rows, dtype=np.float64)
def scales_vector(scales):
values = vector(scales, "Scales")
if any(x <= 0 for x in values):
raise ValueError("Reference scales must be positive.")
return values
def squared_distance(a, b, scales=(1, 1)):
a, b, scales = vector(a, "First vector"), vector(b, "Second vector"), scales_vector(scales)
total = 0.0
for j in range(2):
difference = number(a[j]-b[j], "Coordinate difference")
delta = number(difference/scales[j], "Scaled difference")
square = number(delta*delta, "Squared difference")
if difference != 0 and (delta == 0 or square == 0):
raise ValueError("Distance underflow: rescale coordinates or reference scales so distinct points remain distinguishable.")
total = number(total+square, "Squared distance")
return total
def distance_matrix(X, centers, scales):
# Explicit scalar boundary checks keep NumPy warning settings from changing semantics.
return np.asarray([[squared_distance(x.tolist(), c.tolist(), scales) for c in centers] for x in X], dtype=np.float64)
def inspect_lloyd(state):
X = observations(state["points"])
scales = scales_vector(state["scales"])
centers = state["centers"]
if not isinstance(centers, list):
raise ValueError("Centroids must be a dense array.")
number(len(centers), "k", 1, len(X), True)
C = np.asarray([vector(c, "Centroid") for c in centers], dtype=np.float64)
phase, labels = state["phase"], state["labels"]
if phase not in ("ready", "assign", "update") or type(state["stopped"]) is not bool:
raise ValueError("Invalid phase or stop flag.")
number(state["iteration"], "Iteration", 0, 1000, True)
if phase == "ready":
if labels is not None or state["stopped"] or state["iteration"] != 0:
raise ValueError("Invalid ready state.")
elif not isinstance(labels, list) or len(labels) != len(X):
raise ValueError("Assignments must match rows.")
if state["stopped"] and phase != "update":
raise ValueError("Stop is reported only after mean update.")
if labels is not None:
labels = [number(v, "Assignment", 0, len(C)-1, True) for v in labels]
D = distance_matrix(X, C, scales)
nearest = np.argmin(D, axis=1).tolist() # first index on computed equality
held = nearest if labels is None else labels
clusters = []
for j, c in enumerate(C):
indices = [i for i, label in enumerate(held) if label == j]
sums = [0.0, 0.0]
for i in indices:
for d in range(2):
sums[d] = number(sums[d]+float(X[i, d]), "Cluster sum")
mean = [number(v/len(indices), "Mean") for v in sums] if indices else c.tolist()
clusters.append({"index":j, "ids":[state["points"][i]["id"] for i in indices], "count":len(indices), "sums":sums, "mean":mean, "empty":not indices})
def objective(assignment, matrix=D):
total = 0.0
for i, label in enumerate(assignment):
total = number(total+float(matrix[i, label]), "WCSS")
return total
return {"distances":D.tolist(), "nearest":nearest, "labels":held, "clusters":clusters,
"objective":objective(held), "nearestObjective":objective(nearest),
"proposedObjective":objective(held, distance_matrix(X, np.asarray([c["mean"] for c in clusters]), scales)),
"ties":[sum(v == row[nearest[i]] for v in row) > 1 for i, row in enumerate(D.tolist())]}
def lloyd_step(state):
active = inspect_lloyd(state)
result = deepcopy(state)
if state["stopped"]:
return result
if state["phase"] != "assign":
result.update(labels=active["nearest"], phase="assign")
else:
centers = [c["mean"] for c in active["clusters"]]
result.update(centers=centers, phase="update", iteration=state["iteration"]+1, stopped=centers == state["centers"])
inspect_lloyd(result)
return result
def lloyd_trace(state, half_steps=40):
number(half_steps, "Half steps", 1, 100, True)
frames = [deepcopy(state)]
inspect_lloyd(state)
for _ in range(half_steps):
if frames[-1]["stopped"]:
break
frames.append(lloyd_step(frames[-1]))
return frames
def initialize(points, k, seed):
observations(points)
k = number(k, "k", 1, len(points), True)
state = number(seed, "Seed", 0, 2**32-1, True)
order = list(range(len(points)))
for i in range(k):
state = (1664525*state+1013904223) % 2**32
j = i+math.floor((state/2**32)*(len(points)-i))
order[i], order[j] = order[j], order[i]
return [vector(points[i]["x"], "Selected centroid") for i in order[:k]]
def same_partition(a, b):
if not isinstance(a, list) or not isinstance(b, list) or len(a) != len(b):
raise ValueError("Partitions need equal row counts.")
a = [number(x, "Label", 0, 31, True) for x in a]
b = [number(x, "Label", 0, 31, True) for x in b]
return all((a[i] == a[j]) == (b[i] == b[j]) for i in range(len(a)) for j in range(i))
def hierarchical(points, linkage, scales=(1, 1)):
X = observations(points)
scales_vector(scales)
if linkage not in ("single", "complete", "average"):
raise ValueError("Choose single, complete or average linkage.")
D = np.sqrt(distance_matrix(X, X, scales)).tolist()
groups = [{"id":i, "members":[i]} for i in range(len(X))]
merges = []
while len(groups) > 1:
best = None
for i in range(len(groups)):
for j in range(i+1, len(groups)):
values = [D[a][b] for a in groups[i]["members"] for b in groups[j]["members"]]
if linkage == "single":
height = min(values)
elif linkage == "complete":
height = max(values)
else:
total = 0.0
for d in values:
total = number(total+d, "Linkage sum")
height = number(total/len(values), "Average linkage")
if best is None or height < best[0]:
best = height, i, j
height, i, j = best
left, right = groups[i], groups[j]
group = {"id":len(X)+len(merges), "members":sorted(left["members"]+right["members"])}
merges.append({"left":deepcopy(left), "right":deepcopy(right), "group":deepcopy(group), "height":height})
groups = sorted([g for g in groups if g["id"] not in (left["id"], right["id"])]+[group], key=lambda g:g["id"])
return {"distances":D, "merges":merges}
def dbscan(points, epsilon, min_points, scales=(1, 1)):
X = observations(points)
epsilon = number(epsilon, "Epsilon", 0)
min_points = number(min_points, "Minimum points", 1, len(X), True)
limit = number(epsilon*epsilon, "Squared epsilon")
D = distance_matrix(X, X, scales).tolist()
neighborhoods = [[i for i, d in enumerate(row) if d <= limit] for row in D]
core = [len(ns) >= min_points for ns in neighborhoods]
labels, visited, count = [-1]*len(X), [False]*len(X), 0
for i in range(len(X)):
if visited[i]:
continue
visited[i] = True
if not core[i]:
continue
labels[i] = count
queue, queued = list(neighborhoods[i]), set(neighborhoods[i])
q = 0
while q < len(queue):
j = queue[q]
if not visited[j]:
visited[j] = True
if core[j]:
for v in neighborhoods[j]:
if v not in queued:
queue.append(v); queued.add(v)
if labels[j] == -1:
labels[j] = count
q += 1
count += 1
return {"distances":D, "neighborhoods":neighborhoods, "core":core, "labels":labels, "count":count,
"kinds":["core" if core[i] else "noise" if label < 0 else "border" for i, label in enumerate(labels)]}
def dispatch(request):
operation = request["operation"]
if operation == "lloyd":
state = request["state"]
before = inspect_lloyd(state)
after_state = lloyd_step(state)
after = inspect_lloyd(after_state)
return {"distances":before["distances"], "labels":before["labels"], "means":[c["mean"] for c in before["clusters"]],
"empty":[c["empty"] for c in before["clusters"]], "objective":before["objective"],
"proposedObjective":before["proposedObjective"], "afterCenters":after_state["centers"], "afterObjective":after["objective"]}
if operation == "hierarchical":
result = hierarchical(request["points"], request["linkage"], request.get("scales", [1, 1]))
return {"merges":[{"left":m["left"]["members"], "right":m["right"]["members"], "height":m["height"]} for m in result["merges"]]}
if operation == "dbscan":
result = dbscan(request["points"], request["epsilon"], request["minPoints"], request.get("scales", [1, 1]))
return {k:result[k] for k in ("neighborhoods", "core", "labels", "kinds", "count")}
if operation == "initialize":
return {"centers":initialize(request["points"], request["k"], request["seed"])}
raise ValueError("Unknown clustering operation.")
if __name__ == "__main__":
raw = "" if sys.stdin.isatty() else sys.stdin.read()
default = {"operation":"lloyd", "state":{"points":[{"id":f"s{i+1}", "x":[x,0]} for i, x in enumerate([-3,-1,1,3])], "centers":[[-3,0],[1,0]], "scales":[1,1], "labels":[0,0,1,1], "phase":"assign", "iteration":0, "stopped":False}}
try:
print(json.dumps(dispatch(json.loads(raw) if raw.strip() else default), allow_nan=False, indent=2))
except (ValueError, KeyError, TypeError, OverflowError) as error:
print(f"Clustering input error: {error}", file=sys.stderr)
raise SystemExit(2)Five graded investigations
Checkpoints
You can reproduce initialization from a seed and row order, explain both rectangle costs, distinguish monotonic descent from global optimization, convert units consistently and inspect empty or coincident clusters without manufacturing a mean. An honest experiment states its data, metric, tie rule, stopping condition, numerical limits and evaluation convention together.