III-41 · Composition and Learned Representations
Compose two layers and locate what nonlinearity changes
Trace hidden coordinates, prove the affine collapse and construct an untrained XOR counterexample before asking how weights could be learned.
Part IX · transparent forward pass
Inspect both layers, then remove the activation
Hand-chosen illustrative parameters, never trained. One sample is a column; no batches, labels, fitting, loss, seed or classification threshold. Edits are local.
Computations use binary64. SVG and table labels are rounded for display; full-precision stored values are in “Complete current forward record”.
Forward operation 1 / 14. Inspect the input and parameter shapes. These parameters are hand-chosen, not trained. Complete values remain visible; replay highlights an operation, not training.
- Input provenanceSingle sample; columns stored as flat arrays. No seed, loss, optimizer or class threshold.
- Hand-chosen, untrained values
- x shape
- 2 × 1
- W1 shape
- 2 × 2
- b1 shape
- 2 × 1
- z1 shape
- 2 × 1
- a1 shape
- 2 × 1
- W2 shape
- 1 × 2
- b2 shape
- 1 × 1
- z2 shape
- 1 × 1
- ActivationElementwise at the hidden layer only; output is z2.
- relu
- Hidden 1
- dot 1; b 0; z 1; a 1
- Hidden 2
- dot -1; b 0; z -1; a 0
- Output 1
- dot 1; b 0; z 1; a 1
| Row | Products | Running sums | Dot | b | z | a |
|---|---|---|---|---|---|---|
| Hidden 1 | 1; 0 | 1; 1 | 1 | 0 | 1 | 1 |
| Hidden 2 | -1; 0 | -1; -1 | -1 | 0 | -1 | 0 |
| Output 1 | 1; 0 | 1; 1 | 1 | 0 | 1 | 1 |
Counterfactual affine collapse: hidden activation removed; equivalence is not asserted.
- W_eff = W2 W1o × d = 1 × 2
- [[0,0]]
- b_eff = W2 b1 + b2o × 1 = 1 × 1
- [0]
- Collapsed affine output
- 0
- Actual − collapsedA difference at one point alone cannot prove general equivalence.
- 1
Computed binary64 collapsed affine: 0 + (0) ≈ 0. Actual − collapsed: 1.
| x | z1 | h1 = a1[1] | h2 = a1[2] | z2 | XOR target |
|---|---|---|---|---|---|
| (0,0) | 0; 0 | 0 | 0 | 0 | 0 |
| (0,1) | -1; 1 | 0 | 1 | 1 | 1 |
| (1,0) | 1; -1 | 1 | 0 | 1 | 1 |
| (1,1) | 0; 0 | 0 | 0 | 0 | 0 |
Complete current forward record
{
"state": {
"input": {
"kind": "network",
"x": [
1,
0
],
"W1": [
[
1,
-1
],
[
-1,
1
]
],
"b1": [
0,
0
],
"W2": [
[
1,
1
]
],
"b2": [
0
],
"activation": "relu"
},
"phase": 0
},
"numerical": {
"kind": "network",
"hidden": [
{
"products": [
1,
0
],
"running": [
1,
1
],
"dot": 1,
"b": 0,
"z": 1
},
{
"products": [
-1,
0
],
"running": [
-1,
-1
],
"dot": -1,
"b": 0,
"z": -1
}
],
"a1": [
1,
0
],
"output": [
{
"products": [
1,
0
],
"running": [
1,
1
],
"dot": 1,
"b": 0,
"z": 1
}
],
"collapse": {
"status": "defined",
"W": [
[
0,
0
]
],
"b": [
0
],
"rows": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"difference": [
1
]
},
"shapes": {
"x": [
2,
1
],
"W1": [
2,
2
],
"b1": [
2,
1
],
"z1": [
2,
1
],
"a1": [
2,
1
],
"W2": [
1,
2
],
"b2": [
1,
1
],
"z2": [
1,
1
]
}
},
"binary": [
{
"x": [
0,
0
],
"status": "defined",
"forward": {
"kind": "network",
"hidden": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
},
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"a1": [
0,
0
],
"output": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"collapse": {
"status": "defined",
"W": [
[
0,
0
]
],
"b": [
0
],
"rows": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"difference": [
0
]
},
"shapes": {
"x": [
2,
1
],
"W1": [
2,
2
],
"b1": [
2,
1
],
"z1": [
2,
1
],
"a1": [
2,
1
],
"W2": [
1,
2
],
"b2": [
1,
1
],
"z2": [
1,
1
]
}
}
},
{
"x": [
0,
1
],
"status": "defined",
"forward": {
"kind": "network",
"hidden": [
{
"products": [
0,
-1
],
"running": [
0,
-1
],
"dot": -1,
"b": 0,
"z": -1
},
{
"products": [
0,
1
],
"running": [
0,
1
],
"dot": 1,
"b": 0,
"z": 1
}
],
"a1": [
0,
1
],
"output": [
{
"products": [
0,
1
],
"running": [
0,
1
],
"dot": 1,
"b": 0,
"z": 1
}
],
"collapse": {
"status": "defined",
"W": [
[
0,
0
]
],
"b": [
0
],
"rows": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"difference": [
1
]
},
"shapes": {
"x": [
2,
1
],
"W1": [
2,
2
],
"b1": [
2,
1
],
"z1": [
2,
1
],
"a1": [
2,
1
],
"W2": [
1,
2
],
"b2": [
1,
1
],
"z2": [
1,
1
]
}
}
},
{
"x": [
1,
0
],
"status": "defined",
"forward": {
"kind": "network",
"hidden": [
{
"products": [
1,
0
],
"running": [
1,
1
],
"dot": 1,
"b": 0,
"z": 1
},
{
"products": [
-1,
0
],
"running": [
-1,
-1
],
"dot": -1,
"b": 0,
"z": -1
}
],
"a1": [
1,
0
],
"output": [
{
"products": [
1,
0
],
"running": [
1,
1
],
"dot": 1,
"b": 0,
"z": 1
}
],
"collapse": {
"status": "defined",
"W": [
[
0,
0
]
],
"b": [
0
],
"rows": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"difference": [
1
]
},
"shapes": {
"x": [
2,
1
],
"W1": [
2,
2
],
"b1": [
2,
1
],
"z1": [
2,
1
],
"a1": [
2,
1
],
"W2": [
1,
2
],
"b2": [
1,
1
],
"z2": [
1,
1
]
}
}
},
{
"x": [
1,
1
],
"status": "defined",
"forward": {
"kind": "network",
"hidden": [
{
"products": [
1,
-1
],
"running": [
1,
0
],
"dot": 0,
"b": 0,
"z": 0
},
{
"products": [
-1,
1
],
"running": [
-1,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"a1": [
0,
0
],
"output": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"collapse": {
"status": "defined",
"W": [
[
0,
0
]
],
"b": [
0
],
"rows": [
{
"products": [
0,
0
],
"running": [
0,
0
],
"dot": 0,
"b": 0,
"z": 0
}
],
"difference": [
0
]
},
"shapes": {
"x": [
2,
1
],
"W1": [
2,
2
],
"b1": [
2,
1
],
"z1": [
2,
1
],
"a1": [
2,
1
],
"W2": [
1,
2
],
"b2": [
1,
1
],
"z2": [
1,
1
]
}
}
}
],
"traceEvent": null
}Change the function between the layers
At the opening input x=(1,0), the hidden preactivations are (1,−1). ReLU turns them into a1=(1,0), and the output row sums these coordinates to z2=1. Choose identity: a1 becomes (1,−1) and the output becomes 0. The matrices did not change. The affine-collapse comparison is 0 for either selection; its label explains why it is an identity comparison in one case and a counterfactual in the other.
Use the four input buttons to inspect (0,0), (0,1), (1,0), (1,1). The table keeps all four cases visible, including both hidden activations and the current output when the calculation is representable. Each case is evaluated separately: a failing counterfactual row says Unavailable with its numerical reason and withholds its derived values, while the valid active pass remains visible. Selecting such a row through an input button reports rejection and preserves the last valid active state. With the original ReLU parameters the outputs are 0,1,1,0. Edit any entry of W1, b1, W2 or b2: the table reflects the new function, so it may stop matching the XOR targets. Restore hand-chosen XOR restores the original matrices and ReLU. Reset additionally restores operation 1 and the opening input.
Replay walks through first-layer terms, hidden activations, second-layer terms and final output. Every contribution, running sum, bias and tensor shape remains inspectable. This is an explanation of dependency order, not a sequence of learning updates. An edit during replay forks the displayed input and parameters and closes playback; historical frames remain detached and immutable.
Composition changes the coordinates supplied to the next layer
For a single sample, define
Activation applies separately to each hidden coordinate. The final output is z2; there is no output activation or classification threshold. The first layer can be understood as h copies of the neuron from III-40. Each row of W1 contains one hidden neuron’s weights. Its activated result becomes one coordinate of the next layer’s input.
The contracts are x d×1, W1 h×d, b1,z1,a1 h×1, W2 o×h, and b2,z2 o×1. Inner dimensions determine which products exist. Bias vectors must match the corresponding number of rows; they are not arbitrary broadcastable objects. The explorer fixes d=h=2,o=1. The pure model supports other positive compatible dimensions and rejects incompatible, empty or ragged shapes. It supports no batches; columns are flat arrays in JSON, not observation rows.
A representation here simply means the coordinates a1 supplied to the second layer. A learned representation would require those coordinates’ defining parameters to have been fitted from data. Their interpretation depends on the objective and evidence, not the word “hidden.” Our hand-chosen features do not demonstrate semantic understanding, learned categories or generalization.
Prove that two affine maps collapse
For identity activation a1=z1. Substitution and distributivity yield
Thus W_eff=W2W1 has shape o×d and b_eff=W2b1+b2 has shape o×1. No matter how wide the intermediate layer is, the resulting input-to-output function is affine. Repeating the argument collapses further affine-only layers too. Such factorizations may impose rank constraints or change optimization, but they do not add a new nonlinear input-output function family.
With nonidentity φ, replacing a1 by z1 changes the function in general. The comparison in the explorer explicitly removes φ: it is a counterfactual affine collapse, not an asserted equivalent representation of the nonlinear network. They may agree at some inputs, and some special parameters make them agree everywhere. One equality or inequality at a single input should not be mistaken for a universal proof.
Work an identity composition completely
Consider the independent literal example
Hidden row 1 has products (2,−2), dot 0, and z1[1]=1. Hidden row 2 has products (−2,−0.5), dot −2.5, and z1[2]=−4.5. With identity, the output products are (3,9); the running sums are 3 then 12 and z2=12.5.
The effective weight row is (3·1+(−2)(−1),3·2+(−2)0.5)=(5,5). Its bias is 3·1+(−2)(−2)+0.5=7.5. Direct collapse gives 5·2+5(−1)+7.5=12.5. All these numbers are exactly representable dyadic values. Enter these parameters in the explorer to reproduce each intermediate.
Switch to ReLU without changing them. The hidden coordinates become (1,0) and z2=3.5. The counterfactual remains 12.5. The difference −9 comes from replacing the negative hidden coordinate by zero, before applying its second-layer weight −2. Applying ReLU only after the final sum would be a different network.
Construct XOR with two hand-chosen features
The opening parameters are
They define h1=max(0,x1−x2), h2=max(0,x2−x1), and z2=h1+h2=|x1−x2|. On binary inputs this equals XOR. Outside the four binary corners it is an absolute-difference function, not a guaranteed binary output or probability.
| x | z1 | h1 | h2 | Output z2 |
|---|---|---|---|---|
| (0,0) | (0,0) | 0 | 0 | 0 |
| (0,1) | (−1,1) | 0 | 1 | 1 |
| (1,0) | (1,−1) | 1 | 0 | 1 |
| (1,1) | (0,0) | 0 | 0 | 0 |
Now independently rule out a scalar affine function that matches these four values. Suppose f(x)=c1x1+c2x2+b. The (0,0) case forces b=0. The (1,0) and (0,1) cases force c1=c2=1. Then f(1,1)=2, contradicting the required 0. This contradiction does not rely on the implementation, or on merely observing the collapse in the browser. It concerns exact scalar outputs; it does not sneak in a thresholding rule.
For these matrices W_eff=(0,0) and b_eff=0. Identity composition therefore outputs zero for every input, while ReLU separates the two signs of a difference. We chose the features using our knowledge of XOR. The example establishes representational possibility, not that a trainer discovers those parameters or that the coordinates have learned a semantic concept.
Arithmetic order is part of the experiment
The affine-collapse proof is exact algebra over the reals. Binary64 rounds each product and addition. A scalar example makes the limit visible: choose x=1, W1=1, b1=10¹⁶, W2=1, b2=−10¹⁶ and identity. The staged calculation rounds 1+10¹⁶ to 10¹⁶, then cancels to 0. The collapsed bias cancels to 0 first, and the direct affine output is 1. Algebraic equivalence does not promise identical rounded intermediates. These scales exceed the UI’s ±1000 editing bound but are covered by the authored pure-model tests.
The oracle uses exact Fraction sums for these small rational examples, with Decimal activation values for sigmoid. Exact integer/dyadic fixtures are checked exactly; transcendental quantities use the documented 32ε small-case allowance from III-40, with relative-only checks for small nonzero tails. There is no arbitrary tolerance for shape acceptance, XOR targets or the contradiction. Near cancellation, input precision and intermediate rounding can dominate; no small universal relative error guarantee is implied.
Read the executable mechanism
The same actual Python file supplies both chapters. NumPy handles column construction and elementwise product arrays; explicit sequential sums expose the computation order. It returns hidden products, running, dot, b, z, the activated vector a1, output rows, effective matrices/biases, direct collapsed rows and actual-minus-collapsed differences. The separate oracle computes exact arithmetic without deriving expectations through this module. The owner executes its self-tests, committed-literal freshness check and actual Python/TypeScript parity tests.
"""III-40–41: inspected binary64 forward passes, without fitting or thresholds.
Vectors are columns stored as flat lists; NumPy constructs d×1 columns. Each
row's products are accumulated left to right to match the visible trace.
"""
import json
import math
import sys
import numpy as np
def finite(value, label):
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(f"{label} must be a finite number")
try:
value = float(value)
except (OverflowError, ValueError) as error:
raise ValueError(f"{label} is not representable in binary64") from error
if not math.isfinite(value):
raise ValueError(f"{label} must be finite; arithmetic overflow is rejected")
return value
def vector(value, label):
if not isinstance(value, list) or not value:
raise ValueError(f"{label} must be a nonempty dense column stored as a flat list")
return [finite(v, f"{label}[{i}]") for i, v in enumerate(value)]
def matrix(value, label):
if not isinstance(value, list) or not value:
raise ValueError(f"{label} must be a nonempty matrix")
rows = [vector(row, f"{label}[{i}]") for i, row in enumerate(value)]
if any(len(row) != len(rows[0]) for row in rows):
raise ValueError(f"{label} must be rectangular")
return rows
def activation(phi):
if not isinstance(phi, str) or phi not in ("identity", "relu", "sigmoid"):
raise ValueError("Choose identity, relu or sigmoid")
def activate(z, phi):
z = finite(z, "Preactivation")
activation(phi)
if phi == "identity":
return z
if phi == "relu":
return max(0.0, z)
# This exponential cannot overflow. Tails may round to exactly zero or one.
e = math.exp(-abs(z))
return 1 / (1 + e) if z >= 0 else e / (1 + e)
def affine(x, W, b):
column = np.array(x, dtype=np.float64).reshape(-1, 1)
weights = np.array(W, dtype=np.float64)
try:
with np.errstate(over="raise", invalid="raise", under="ignore"):
terms = weights * column.T
except FloatingPointError as error:
raise ValueError("Product arithmetic overflow") from error
rows = []
for i, terms_row in enumerate(terms):
products = [finite(float(p), "Product") for p in terms_row]
if any(w != 0 and v != 0 and p == 0 for w, v, p in zip(W[i], x, products)):
raise ValueError("A nonzero product underflowed to zero. Rescale the inputs.")
dot, running = 0.0, []
for p in products:
dot = finite(dot + p, "Running sum")
running.append(dot)
rows.append(dict(products=products, running=running, dot=dot, b=b[i], z=finite(dot + b[i], "Bias addition")))
return rows
def forward(request):
if not isinstance(request, dict) or request.get("kind") not in ("neuron", "network"):
raise ValueError("Expected a neuron or network request")
x = vector(request.get("x"), "x")
phi = request.get("activation")
activation(phi)
d = len(x)
if request["kind"] == "neuron":
w, b = vector(request.get("w"), "w"), finite(request.get("b"), "b")
if len(w) != d:
raise ValueError("w and x must both have shape d × 1")
row = affine(x, [w], [b])[0]
return dict(kind="neuron", row=row, a=activate(row["z"], phi), shapes=dict(x=[d, 1], w=[d, 1], b=[1, 1], z=[1, 1], a=[1, 1]))
W1, b1 = matrix(request.get("W1"), "W1"), vector(request.get("b1"), "b1")
W2, b2 = matrix(request.get("W2"), "W2"), vector(request.get("b2"), "b2")
h, o = len(W1), len(W2)
if len(W1[0]) != d or len(b1) != h or len(W2[0]) != h or len(b2) != o:
raise ValueError("Require x d × 1, W1 h × d, b1 h × 1, W2 o × h, b2 o × 1")
hidden = affine(x, W1, b1)
a1 = [activate(row["z"], phi) for row in hidden]
output = affine(a1, W2, b2)
try:
W = [[affine([row[j] for row in W1], [weights], [0])[0]["z"] for j in range(d)] for weights in W2]
b = [row["z"] for row in affine(b1, W2, b2)]
rows = affine(x, W, b)
difference = [finite(row["z"] - collapsed["z"], "Collapse difference") for row, collapsed in zip(output, rows)]
collapse = dict(status="defined", W=W, b=b, rows=rows, difference=difference)
except ValueError as error:
collapse = dict(status="unrepresentable", reason=str(error))
return dict(kind="network", hidden=hidden, a1=a1, output=output, collapse=collapse,
shapes=dict(x=[d, 1], W1=[h, d], b1=[h, 1], z1=[h, 1], a1=[h, 1], W2=[o, h], b2=[o, 1], z2=[o, 1]))
if __name__ == "__main__":
raw = sys.stdin.read().strip()
request = json.loads(raw) if raw else dict(kind="neuron", x=[2, -1], w=[0.5, 2], b=0.25, activation="relu")
try:
print(json.dumps(forward(request), allow_nan=False))
except ValueError as error:
print(str(error), file=sys.stderr)
sys.exit(1)Mistakes, connections and the next book
More affine-only layers do not create nonlinear expressiveness. More hidden coordinates do not prove that useful features were learned. A hidden activation is not the same as an output threshold. ReLU at zero is a valid forward value even though derivative-based learning needs a convention at the corner. A sigmoid hidden feature can saturate; a tiny forward sensitivity matters later to derivatives, not to the shape of matrix multiplication.
Book III has distinguished model families, fitting objectives, evaluation and numerical limits. Book IV must connect a forward output to a task-specific loss, use the chain rule to relate changes in that loss to earlier weights, and use optimization to update parameters. Book II’s actual chain-rule, gradient and objective prerequisites are linked below through the frozen resolver. Training-only data boundaries, held-out evaluation and numerical diagnostics remain necessary when features are learned. This closing chapter implements no derivatives, backward pass, trainer or later architecture.
Five graded investigations
What you can now explain
You can trace a two-layer column-vector computation, prove the affine collapse, identify the hidden nonlinearity’s effect, construct and independently justify XOR, and distinguish a fixed representation from learned features. Book III ends here. Learning those weights requires optimization and derivatives; the conceptual bridge is ready for Book IV without implementing its training machinery.