III-27 · Initialization, Convergence, Scaling and Failure Cases

A stable center can still be the wrong destination

Compare two fixed points on the same rectangle, inspect an empty cluster, and change the units that determine nearest-center decisions.

Chapters in Part V
  1. III-26 · k-Means
  2. III-27 · Initialization, Convergence, Scaling and Failure Cases
  3. III-28 · Hierarchical and Density-Based Clustering

Distance → assignment → mean

Hand-authored, unitless toy observations. Labels identify groups; they have no numerical order.

Feature plane with draggable centroidsAxes use raw coordinates and equal scale, −5 to 5. Group numbers and the tables supplement color. Arrow keys move a handle by 0.1; Shift uses 0.5.s1; coordinates -3, 0; noise; cluster 0 s1:0s2; coordinates -1, 0; noise; cluster 0 s2:0s3; coordinates 1, 0; noise; cluster 1 s3:1s4; coordinates 3, 0; noise; cluster 1 s4:1c0c1

Filled circles: observations. Hollow diamonds: centroids. Dashed links use held labels. Drag snaps to 0.1; arrow keys preserve full input precision. Only handles capture touch. The selector brings an overlapping handle forward; Enter or Space does the same for a focused handle.

Frame 1 / 1 · edit begins at the displayed state.

X · samples in rowsEach mathematical feature vector is a 2 × 1 column.
4 × 2
Centroids / squared distances / assignments
2 × 2 / 4 × 2 / 4 × 1
Held-assignment WCSSNearest-label preview; press Assign to commit.
≈ 8
Next nearest-assignment WCSSAllows labels to change; same current centroids.
≈ 8
Mean update · current → proposed WCSSHeld assignments; empty centroids retained. Inspect before applying Update means.
≈ 8 → ≈ 4
Phase and stopNot stopped; replay is capped at 40 half-steps.
ready · 0 mean updates
Empty-cluster policyEmpty mean is undefined. Retained coordinates are a policy, not a computed empty mean.
Retain previous centroid
Input provenance / initializationSeed affects partial Fisher–Yates observation selection only. Current centroids may be edited.
Hand-authored observations / seed 7
dij2=∑r=12((xir−cjr)/sr)2,ai=arg min⁡jdij2d_{ij}^2=\sum_{r=1}^2((x_{ir}-c_{jr})/s_r)^2,\quad a_i=\operatorname*{arg\,min}_j d_{ij}^2
μj=1∣Cj∣∑i∈Cjxi,J=∑idi,ai2\mu_j=\frac{1}{|C_j|}\sum_{i\in C_j}x_i,\quad J=\sum_i d_{i,a_i}^2
Actual scaled squared distances. Lowest centroid index wins exact ties. Preview labels; ≈ marks display rounding.
Observationx₁x₂d² to c0d² to c1Held labelNearest / tie
s1-30≈ 0≈ 1600 / unique
s2-10≈ 4≈ 400 / tie
s310≈ 16≈ 011 / unique
s430≈ 36≈ 411 / unique
Held membership → sums → proposed mean. Empty clusters retain previous coordinates; their mean is undefined.
ClusterIDs / nSumMean / policyCurrent centroid
c0s1, s2 / 2[-4, 0]≈ [-2, 0]≈ [-3, 0]
c1s3, s4 / 2[4, 0]≈ [2, 0]≈ [1, 0]

Load Vertical groups and Horizontal local minimum on the same rectangle. Apply real assignments and mean updates, compare objectives, then test seeds, scales, empty clusters and coincident observations.

Stop does not mean best

Load Horizontal local minimum, press Assign nearest, and inspect the proposed means. Press Update means. The centers remain (0,−1) and (0,1), and WCSS stays 16. The sheet honestly reports a fixed point. Now load Vertical groups and perform the same two operations: centers (−2,0) and (2,0) also stay fixed, but WCSS is 4.

Both results use exactly the four points (−2,−1), (−2,1), (2,−1), (2,1). The difference is the starting centers, not the dataset or a fabricated convergence trace. Drag a center away from either configuration and inspect how the nearest-center boundary changes. A manually moved center is a new experiment; monotonicity applies to Lloyd steps on fixed data and scales.

Derive the two costs

For horizontal groups, each point is two units horizontally from its group center. Each squared distance is 4; four points give JH=16J_H=16. For vertical groups, every point is one unit vertically from its center, so JV=4J_V=4.

For the same rectangle with positive reference scales sx,sys_x,s_y, these fixed groupings have costs

JH=16sx2,JV=4sy2.J_H=\frac{16}{s_x^2},\qquad J_V=\frac4{s_y^2}.

With sx=sy=1s_x=s_y=1, vertical groups are better. With sx=10,sy=1s_x=10,s_y=1, horizontal groups cost 0.16 while vertical groups cost 4. This is a change of the model’s metric. It need not preserve what “nearby” means.

For this four-row example only, an independent oracle enumerates all seven unlabeled nonempty two-cluster partitions. The singleton/triple splits have WCSS 40/340/3, the diagonal pairing costs 20, horizontal costs 16, and vertical costs 4. Thus 4 is the exact global two-cluster minimum for these unit-scale observations. Lloyd’s stop flag does not perform that enumeration, and no global guarantee is exported to larger data.

Monotonicity has a scope

The assignment step minimizes each row’s cost given the centers; the nonempty mean update minimizes each cluster’s cost given its membership. Retaining an empty center changes no assigned cost. Therefore each half-step has non-increasing WCSS in exact arithmetic.

This explains a descent property, not a promise of a unique solution. At the horizontal rectangle configuration, every point is strictly closer to its assigned horizontal center, and those centers are already the means. There is no improving Lloyd half-step even though another grouping is better. Ties, empty centers and coincident data can also make assignments or centers non-unique.

The implementation stops only when every updated center exactly equals its old binary64 coordinates. It reports that condition rather than declaring mathematical convergence to infinite precision. Replay is capped at 40 half-steps; a trace reaching that cap without the stop flag is unfinished, not converged. The pure trace helper supports up to 100 half-steps. It returns no partial history if an arithmetic failure occurs.

A seed is a reproducible choice of rows

Our initialization samples k distinct observation indices without replacement using a partial Fisher–Yates permutation. It does not implement k-means++, estimate a global optimum, or forbid coincident coordinates. At step i, draw a number u from the frozen platform LCG, choose index j=i+floor(u(n−i)), exchange positions i and j, and retain the first k observation rows as initial centers.

The integer generator is

zt+1=(1664525zt+1013904223) mod 232,ut+1=zt+1/232.z_{t+1}=(1664525z_t+1013904223)\bmod2^{32}, \qquad u_{t+1}=z_{t+1}/2^{32}.

Seeds are unsigned 32-bit integers. There is one draw per selected center. Seed 0 is valid. At seed 0 the first two generator integers are 1013904223 and 1196435762. For the original four-point row order, the first selected index is 0; the second is 1. Centers are therefore (−3,0) and (−1,0). Seed 7 selects indices 0 and 3, giving centers (−3,0) and (3,0).

Change Seed and press Initialize from seed to make that choice concrete. Merely typing a seed does not replace already edited centers. Changing k deliberately initializes k new centers using the current seed. Repeat seeds to compare runs; a favorable objective is evidence about this objective on these data, not about a natural category structure.

Row order is part of the deterministic experiment. Reordering observations while keeping the same seed can choose different coordinates. Stable unique IDs preserve identity; IDs themselves are not features and are not used as random keys. The independent oracle evaluates a closed-form modular geometric series for the generator rather than translating its recurrence.

Units change squared importance

Suppose an observation’s first coordinate is 2 metres from a center. With reference scale 1 metre, its normalized difference is 2 and squared contribution 4. Express the same difference as 200 centimetres and the same reference as 100 centimetres: the ratio is still 2. Changing both together preserves the metric. Converting the coordinate but leaving its numerical reference at 1 instead makes that feature’s squared contribution 10,000 times larger.

The sheet keeps raw coordinates in the plot and divides by the explicit Scale x/y in the numerical model. Larger reference scales reduce a feature’s importance. If scales are estimated from real data, estimate them on the training design and carry the same transformation into later observations. Outliers, constant features, missing values and inappropriate feature units need deliberate preprocessing; this toy sheet does not infer a pipeline.

An objective expressed under one scale cannot be compared numerically with a differently scaled objective as if it were the same unit. Comparing k on the same representation also needs care because training WCSS cannot increase when additional unconstrained centers are allowed.

Empty clusters and repeated observations stay visible

Load Empty cluster. Two points at (−1,0) and (1,0) and two centers at (0,0) create exact ties. Both rows choose label 0, whose mean is (0,0). Cluster 1 has no rows. It keeps its old center; its mean is explicitly undefined. WCSS remains 2, and the mean update stops without inventing a second group.

Load Coincident observations. Three distinct IDs all occupy (1,1), with k=3 and three coincident initial centers. Every distance is zero, every tie chooses label 0, and two clusters stay empty. k≤n is necessary for this initialization, but it does not guarantee k nonempty groups when distinct rows share coordinates. Removing duplicate IDs from the dataset would change its empirical weights; coordinate coincidence alone cannot justify removing them.

SituationDefined numerical resultInterpretation
k=1One coordinatewise meanWCSS summarizes total scatter
k>n or k=0Rejected with reasonNo silent clamping of k
Duplicate coordinates, distinct IDsSeparate weighted observationsCan produce empty centers
Empty clusterUndefined mean; retain old centerNo hidden reseeding
Horizontal rectangle fixed pointWCSS 16Worse than another fixed point
Nonzero squared-distance underflowRejected; rescaleNo manufactured zero-distance tie

Evaluate partitions, not arbitrary label names

Assignments [0,0,1,1] and [1,1,0,0] describe the same grouping. They have identical within/between membership relations, even though a literal elementwise “accuracy” would call every label different. Our same_partition checks whether every pair of observations is together in both assignments. It is invariant to a one-to-one permutation of occupied labels.

This helper is a partition-equivalence predicate, not an external accuracy score, adjusted Rand index or semantic ground truth. It accepts nonnegative cluster labels. DBSCAN’s −1 noise marker is handled separately; grouping all noise points into one true cluster would introduce a different evaluation convention. Real external labels require an explicitly defined metric and treatment of noise or unmatched groups.

Numerical caveats and connections

The interactive coordinates stay within −5…5 and reference scales within 0.1…10; these are educational bounds, not model theory. Numeric edits reject values beyond them without replacing the last valid state. Numeric, pointer, touch, keyboard and reset paths share the same validated model. Editing during replay starts from the displayed frozen state and clears the old trace.

III-26 established the two subproblems. III-28 will question whether squared distance to a mean is even the right definition of a cluster. Book II’s local/global optima and numerical stopping chapters explain why reproducibility, stability and optimality are separate questions.

Read the canonical reference

initialize, lloyd_trace, same_partition and the shape/arithmetic validators below are the actual Python source, not a prose copy. The default executable prints the first mean update; fixture inputs can be sent as JSON on stdin to select other operations.

.venv/bin/python -B code/python/p05/clustering.py
code/python/p05/clustering.py
"""Canonical from-scratch NumPy clustering. X is n×2; centers are k×2.
No target column, fitting library, hidden reseeding, or optimum guarantee.
"""
import json
import math
import sys
from copy import deepcopy
import numpy as np


def number(v, name, lo=None, hi=None, integer=False):
    try:
        valid = isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(float(v))
    except (OverflowError, TypeError):
        valid = False
    if not valid:
        raise ValueError(f"{name} must be a finite number.")
    v = float(v)
    if (lo is not None and v < lo) or (hi is not None and v > hi) or (integer and v != math.floor(v)):
        raise ValueError(f"{name} is outside its supported range.")
    return int(v) if integer else v


def vector(v, name):
    if not isinstance(v, (list, tuple)) or len(v) != 2:
        raise ValueError(f"{name} needs a dense two-feature row.")
    return [number(x, name) for x in v]


def observations(points):
    if not isinstance(points, list) or not 1 <= len(points) <= 32:
        raise ValueError("Use 1–32 observations; empty data have no objective.")
    ids, rows = set(), []
    for p in points:
        if not isinstance(p, dict) or not isinstance(p.get("id"), str) or not p["id"].strip() or p["id"] in ids:
            raise ValueError("Observation IDs must be unique nonempty strings.")
        ids.add(p["id"])
        rows.append(vector(p.get("x"), "Observation"))
    return np.asarray(rows, dtype=np.float64)


def scales_vector(scales):
    values = vector(scales, "Scales")
    if any(x <= 0 for x in values):
        raise ValueError("Reference scales must be positive.")
    return values


def squared_distance(a, b, scales=(1, 1)):
    a, b, scales = vector(a, "First vector"), vector(b, "Second vector"), scales_vector(scales)
    total = 0.0
    for j in range(2):
        difference = number(a[j]-b[j], "Coordinate difference")
        delta = number(difference/scales[j], "Scaled difference")
        square = number(delta*delta, "Squared difference")
        if difference != 0 and (delta == 0 or square == 0):
            raise ValueError("Distance underflow: rescale coordinates or reference scales so distinct points remain distinguishable.")
        total = number(total+square, "Squared distance")
    return total


def distance_matrix(X, centers, scales):
    # Explicit scalar boundary checks keep NumPy warning settings from changing semantics.
    return np.asarray([[squared_distance(x.tolist(), c.tolist(), scales) for c in centers] for x in X], dtype=np.float64)


def inspect_lloyd(state):
    X = observations(state["points"])
    scales = scales_vector(state["scales"])
    centers = state["centers"]
    if not isinstance(centers, list):
        raise ValueError("Centroids must be a dense array.")
    number(len(centers), "k", 1, len(X), True)
    C = np.asarray([vector(c, "Centroid") for c in centers], dtype=np.float64)
    phase, labels = state["phase"], state["labels"]
    if phase not in ("ready", "assign", "update") or type(state["stopped"]) is not bool:
        raise ValueError("Invalid phase or stop flag.")
    number(state["iteration"], "Iteration", 0, 1000, True)
    if phase == "ready":
        if labels is not None or state["stopped"] or state["iteration"] != 0:
            raise ValueError("Invalid ready state.")
    elif not isinstance(labels, list) or len(labels) != len(X):
        raise ValueError("Assignments must match rows.")
    if state["stopped"] and phase != "update":
        raise ValueError("Stop is reported only after mean update.")
    if labels is not None:
        labels = [number(v, "Assignment", 0, len(C)-1, True) for v in labels]
    D = distance_matrix(X, C, scales)
    nearest = np.argmin(D, axis=1).tolist()  # first index on computed equality
    held = nearest if labels is None else labels
    clusters = []
    for j, c in enumerate(C):
        indices = [i for i, label in enumerate(held) if label == j]
        sums = [0.0, 0.0]
        for i in indices:
            for d in range(2):
                sums[d] = number(sums[d]+float(X[i, d]), "Cluster sum")
        mean = [number(v/len(indices), "Mean") for v in sums] if indices else c.tolist()
        clusters.append({"index":j, "ids":[state["points"][i]["id"] for i in indices], "count":len(indices), "sums":sums, "mean":mean, "empty":not indices})
    def objective(assignment, matrix=D):
        total = 0.0
        for i, label in enumerate(assignment):
            total = number(total+float(matrix[i, label]), "WCSS")
        return total
    return {"distances":D.tolist(), "nearest":nearest, "labels":held, "clusters":clusters,
            "objective":objective(held), "nearestObjective":objective(nearest),
            "proposedObjective":objective(held, distance_matrix(X, np.asarray([c["mean"] for c in clusters]), scales)),
            "ties":[sum(v == row[nearest[i]] for v in row) > 1 for i, row in enumerate(D.tolist())]}


def lloyd_step(state):
    active = inspect_lloyd(state)
    result = deepcopy(state)
    if state["stopped"]:
        return result
    if state["phase"] != "assign":
        result.update(labels=active["nearest"], phase="assign")
    else:
        centers = [c["mean"] for c in active["clusters"]]
        result.update(centers=centers, phase="update", iteration=state["iteration"]+1, stopped=centers == state["centers"])
    inspect_lloyd(result)
    return result


def lloyd_trace(state, half_steps=40):
    number(half_steps, "Half steps", 1, 100, True)
    frames = [deepcopy(state)]
    inspect_lloyd(state)
    for _ in range(half_steps):
        if frames[-1]["stopped"]:
            break
        frames.append(lloyd_step(frames[-1]))
    return frames


def initialize(points, k, seed):
    observations(points)
    k = number(k, "k", 1, len(points), True)
    state = number(seed, "Seed", 0, 2**32-1, True)
    order = list(range(len(points)))
    for i in range(k):
        state = (1664525*state+1013904223) % 2**32
        j = i+math.floor((state/2**32)*(len(points)-i))
        order[i], order[j] = order[j], order[i]
    return [vector(points[i]["x"], "Selected centroid") for i in order[:k]]


def same_partition(a, b):
    if not isinstance(a, list) or not isinstance(b, list) or len(a) != len(b):
        raise ValueError("Partitions need equal row counts.")
    a = [number(x, "Label", 0, 31, True) for x in a]
    b = [number(x, "Label", 0, 31, True) for x in b]
    return all((a[i] == a[j]) == (b[i] == b[j]) for i in range(len(a)) for j in range(i))


def hierarchical(points, linkage, scales=(1, 1)):
    X = observations(points)
    scales_vector(scales)
    if linkage not in ("single", "complete", "average"):
        raise ValueError("Choose single, complete or average linkage.")
    D = np.sqrt(distance_matrix(X, X, scales)).tolist()
    groups = [{"id":i, "members":[i]} for i in range(len(X))]
    merges = []
    while len(groups) > 1:
        best = None
        for i in range(len(groups)):
            for j in range(i+1, len(groups)):
                values = [D[a][b] for a in groups[i]["members"] for b in groups[j]["members"]]
                if linkage == "single":
                    height = min(values)
                elif linkage == "complete":
                    height = max(values)
                else:
                    total = 0.0
                    for d in values:
                        total = number(total+d, "Linkage sum")
                    height = number(total/len(values), "Average linkage")
                if best is None or height < best[0]:
                    best = height, i, j
        height, i, j = best
        left, right = groups[i], groups[j]
        group = {"id":len(X)+len(merges), "members":sorted(left["members"]+right["members"])}
        merges.append({"left":deepcopy(left), "right":deepcopy(right), "group":deepcopy(group), "height":height})
        groups = sorted([g for g in groups if g["id"] not in (left["id"], right["id"])]+[group], key=lambda g:g["id"])
    return {"distances":D, "merges":merges}


def dbscan(points, epsilon, min_points, scales=(1, 1)):
    X = observations(points)
    epsilon = number(epsilon, "Epsilon", 0)
    min_points = number(min_points, "Minimum points", 1, len(X), True)
    limit = number(epsilon*epsilon, "Squared epsilon")
    D = distance_matrix(X, X, scales).tolist()
    neighborhoods = [[i for i, d in enumerate(row) if d <= limit] for row in D]
    core = [len(ns) >= min_points for ns in neighborhoods]
    labels, visited, count = [-1]*len(X), [False]*len(X), 0
    for i in range(len(X)):
        if visited[i]:
            continue
        visited[i] = True
        if not core[i]:
            continue
        labels[i] = count
        queue, queued = list(neighborhoods[i]), set(neighborhoods[i])
        q = 0
        while q < len(queue):
            j = queue[q]
            if not visited[j]:
                visited[j] = True
                if core[j]:
                    for v in neighborhoods[j]:
                        if v not in queued:
                            queue.append(v); queued.add(v)
            if labels[j] == -1:
                labels[j] = count
            q += 1
        count += 1
    return {"distances":D, "neighborhoods":neighborhoods, "core":core, "labels":labels, "count":count,
            "kinds":["core" if core[i] else "noise" if label < 0 else "border" for i, label in enumerate(labels)]}


def dispatch(request):
    operation = request["operation"]
    if operation == "lloyd":
        state = request["state"]
        before = inspect_lloyd(state)
        after_state = lloyd_step(state)
        after = inspect_lloyd(after_state)
        return {"distances":before["distances"], "labels":before["labels"], "means":[c["mean"] for c in before["clusters"]],
                "empty":[c["empty"] for c in before["clusters"]], "objective":before["objective"],
                "proposedObjective":before["proposedObjective"], "afterCenters":after_state["centers"], "afterObjective":after["objective"]}
    if operation == "hierarchical":
        result = hierarchical(request["points"], request["linkage"], request.get("scales", [1, 1]))
        return {"merges":[{"left":m["left"]["members"], "right":m["right"]["members"], "height":m["height"]} for m in result["merges"]]}
    if operation == "dbscan":
        result = dbscan(request["points"], request["epsilon"], request["minPoints"], request.get("scales", [1, 1]))
        return {k:result[k] for k in ("neighborhoods", "core", "labels", "kinds", "count")}
    if operation == "initialize":
        return {"centers":initialize(request["points"], request["k"], request["seed"])}
    raise ValueError("Unknown clustering operation.")


if __name__ == "__main__":
    raw = "" if sys.stdin.isatty() else sys.stdin.read()
    default = {"operation":"lloyd", "state":{"points":[{"id":f"s{i+1}", "x":[x,0]} for i, x in enumerate([-3,-1,1,3])], "centers":[[-3,0],[1,0]], "scales":[1,1], "labels":[0,0,1,1], "phase":"assign", "iteration":0, "stopped":False}}
    try:
        print(json.dumps(dispatch(json.loads(raw) if raw.strip() else default), allow_nan=False, indent=2))
    except (ValueError, KeyError, TypeError, OverflowError) as error:
        print(f"Clustering input error: {error}", file=sys.stderr)
        raise SystemExit(2)

Five graded investigations

The horizontal rectangle run stops with WCSS 16. What follows?
A feature and its reference scale are both multiplied by 100. What happens to its squared-distance contribution?
With seed 0 and the original four-row order, which initial centers does k=2 select?
Three rows coincide and k=3. Must all three clusters be occupied?
Are [0,0,1,1] and [1,1,0,0] the same partition?

Checkpoints

You can reproduce initialization from a seed and row order, explain both rectangle costs, distinguish monotonic descent from global optimization, convert units consistently and inspect empty or coincident clusters without manufacturing a mean. An honest experiment states its data, metric, tie rule, stopping condition, numerical limits and evaluation convention together.