Keep the affine predictor and add one explicit function

{
  "state": {
    "input": {
      "kind": "neuron",
      "x": [
        2,
        -1
      ],
      "w": [
        0.5,
        2
      ],
      "b": 0.25,
      "activation": "relu"
    },
    "phase": 0
  },
  "numerical": {
    "kind": "neuron",
    "row": {
      "products": [
        1,
        -2
      ],
      "running": [
        1,
        -1
      ],
      "dot": -1,
      "b": 0.25,
      "z": -0.75
    },
    "a": 0,
    "shapes": {
      "x": [
        2,
        1
      ],
      "w": [
        2,
        1
      ],
      "b": [
        1,
        1
      ],
      "z": [
        1,
        1
      ],
      "a": [
        1,
        1
      ]
    }
  },
  "binary": [],
  "traceEvent": null
}

Start with a negative preactivation

The opening neuron has x=(2,−1), w=(0.5,2) and b=0.25. Its contributions are 1 and −2, the running sums are 1 then −1, and the preactivation is −0.75. ReLU returns 0. Change Activation to identity: the contributions and preactivation stay fixed, while the output becomes −0.75. Choose sigmoid: the output is about 0.320821. Changing the output function changes the mechanism even though no weight changes.

Now change x 2 from −1 to 1. The second contribution becomes 2; the dot is 3 and z=3.25. ReLU now passes that positive value through. Enter b=−3 to reach z=0, where both identity and ReLU return 0 but sigmoid returns 0.5. Each x coordinate, each weight and b has a numeric editor. Enter applies a draft; arrows move by 0.1 and Shift + arrows by 0.5. Invalid drafts keep the previous numerical state. Buttons and selectors also work by keyboard and touch.

Replay forward pass starts at the current input and walks through every product, running sum, bias addition and activation. The complete calculation stays visible throughout: replay explains a forward pass rather than simulating training. Editing during replay begins at the displayed input and ends replay; reset restores the original negative example, ReLU and operation 1. Expand the complete record to inspect full-precision values and the detached stored trace event.

The neuron contains the old predictor

In III-07 Linear Regression, the frozen scalar predictor is ŷ=wx+b. Here the dot product generalizes that same affine mechanism to d features:

For d=1 and identity activation, a=wx+b is exactly that scalar affine prediction in real arithmetic. The earlier chapter’s predictor remains unchanged; this Part implements the vector extension with its own validated kernel. In III-07 a fitted weight could result from minimizing MSE. Here the parameters are supplied by us and the explorer never updates them from data.

An artificial neuron is a numerical function, not a biological cell simulation. Each weight scales one coordinate; the sum combines contributions; the bias shifts the preactivation; the activation transforms that scalar. A zero weight removes that feature’s direct contribution. A zero input removes its contribution for that sample, even when the weight is nonzero. Neither fact means a parameter was learned to be irrelevant.

Define values and shapes separately

The mathematical vectors x and w are d×1 columns. Their transpose product wᵀx is 1×1. The bias b, preactivation z and activation a are scalar values, represented as 1×1 in the shape inspector. The JSON interface stores columns as flat arrays; the canonical Python creates a NumPy column explicitly. It accepts one sample only, without implicit broadcasting or a batch convention.

The input x is an observation; w and b are parameters; φ is an explicitly selected function. These roles do not swap when we edit them. There is no target, loss, decision threshold, optimizer or evaluation metric in a bare forward pass. A sigmoid value in [0,1] could be used as a probability in an appropriately fitted and evaluated binary model, as in III-13; merely putting sigmoid here does not establish calibration or define a class decision.

Derive the three output functions

Identity, ReLU and sigmoid are

Identity leaves the affine function unchanged. ReLU is piecewise linear: negative z gives 0, positive z gives z, and z=0 gives 0. It is continuous, but its derivative is not uniquely defined at zero. Forward evaluation at that point is perfectly defined. A later optimizer must state a derivative convention; this chapter implements none.

For very negative z, directly evaluating e⁻ᶻ can overflow. Multiply the numerator and denominator by eᶻ to obtain eᶻ/(1+eᶻ). The stable implementation therefore uses

Every exponential argument is nonpositive. Real sigmoid is strictly between 0 and 1 for finite z; binary64 can round its positive tail to 1 and underflow its negative tail to 0. Those rounded endpoint outputs are disclosed, not treated as exact probabilities of certainty.

Work the products by hand

For the opening x, w and b,

The outputs are identity −0.75, ReLU 0, and sigmoid approximately 0.3208213008. With x=(0,0) and b=2, both products and the dot vanish; identity and ReLU output 2 while sigmoid outputs approximately 0.880797078. The bias survives zero input. Zero neuron instead sets both inputs and bias to zero, retains the weights, and selects ReLU.

A compact set of further cases is useful:

Use the two extreme buttons to inspect ±1000 with zero input and the extreme bias. Small changes far into a saturated sigmoid tail can leave the displayed output unchanged. Saturation concerns the function’s sensitivity and floating representation; it is separate from choosing a class threshold. ReLU’s negative side is exactly flat even in real arithmetic.

Explore the actual canonical reference

The code below is imported from the actual executable Python file. forward(request) constructs NumPy products, then accumulates them in the same left-to-right order shown in the inspector. Its default CLI request is the opening neuron; a complete JSON request on stdin selects other values. The network branch will be developed in III-41. Committed expectations use independent rational sums and high-precision Decimal sigmoid; agreement between two translations alone is insufficient.

"""III-40–41: inspected binary64 forward passes, without fitting or thresholds.

Vectors are columns stored as flat lists; NumPy constructs d×1 columns. Each
row's products are accumulated left to right to match the visible trace.
"""
import json
import math
import sys

import numpy as np


def finite(value, label):
    if isinstance(value, bool) or not isinstance(value, (int, float)):
        raise ValueError(f"{label} must be a finite number")
    try:
        value = float(value)
    except (OverflowError, ValueError) as error:
        raise ValueError(f"{label} is not representable in binary64") from error
    if not math.isfinite(value):
        raise ValueError(f"{label} must be finite; arithmetic overflow is rejected")
    return value


def vector(value, label):
    if not isinstance(value, list) or not value:
        raise ValueError(f"{label} must be a nonempty dense column stored as a flat list")
    return [finite(v, f"{label}[{i}]") for i, v in enumerate(value)]


def matrix(value, label):
    if not isinstance(value, list) or not value:
        raise ValueError(f"{label} must be a nonempty matrix")
    rows = [vector(row, f"{label}[{i}]") for i, row in enumerate(value)]
    if any(len(row) != len(rows[0]) for row in rows):
        raise ValueError(f"{label} must be rectangular")
    return rows


def activation(phi):
    if not isinstance(phi, str) or phi not in ("identity", "relu", "sigmoid"):
        raise ValueError("Choose identity, relu or sigmoid")


def activate(z, phi):
    z = finite(z, "Preactivation")
    activation(phi)
    if phi == "identity":
        return z
    if phi == "relu":
        return max(0.0, z)
    # This exponential cannot overflow. Tails may round to exactly zero or one.
    e = math.exp(-abs(z))
    return 1 / (1 + e) if z >= 0 else e / (1 + e)


def affine(x, W, b):
    column = np.array(x, dtype=np.float64).reshape(-1, 1)
    weights = np.array(W, dtype=np.float64)
    try:
        with np.errstate(over="raise", invalid="raise", under="ignore"):
            terms = weights * column.T
    except FloatingPointError as error:
        raise ValueError("Product arithmetic overflow") from error
    rows = []
    for i, terms_row in enumerate(terms):
        products = [finite(float(p), "Product") for p in terms_row]
        if any(w != 0 and v != 0 and p == 0 for w, v, p in zip(W[i], x, products)):
            raise ValueError("A nonzero product underflowed to zero. Rescale the inputs.")
        dot, running = 0.0, []
        for p in products:
            dot = finite(dot + p, "Running sum")
            running.append(dot)
        rows.append(dict(products=products, running=running, dot=dot, b=b[i], z=finite(dot + b[i], "Bias addition")))
    return rows


def forward(request):
    if not isinstance(request, dict) or request.get("kind") not in ("neuron", "network"):
        raise ValueError("Expected a neuron or network request")
    x = vector(request.get("x"), "x")
    phi = request.get("activation")
    activation(phi)
    d = len(x)
    if request["kind"] == "neuron":
        w, b = vector(request.get("w"), "w"), finite(request.get("b"), "b")
        if len(w) != d:
            raise ValueError("w and x must both have shape d × 1")
        row = affine(x, [w], [b])[0]
        return dict(kind="neuron", row=row, a=activate(row["z"], phi), shapes=dict(x=[d, 1], w=[d, 1], b=[1, 1], z=[1, 1], a=[1, 1]))
    W1, b1 = matrix(request.get("W1"), "W1"), vector(request.get("b1"), "b1")
    W2, b2 = matrix(request.get("W2"), "W2"), vector(request.get("b2"), "b2")
    h, o = len(W1), len(W2)
    if len(W1[0]) != d or len(b1) != h or len(W2[0]) != h or len(b2) != o:
        raise ValueError("Require x d × 1, W1 h × d, b1 h × 1, W2 o × h, b2 o × 1")
    hidden = affine(x, W1, b1)
    a1 = [activate(row["z"], phi) for row in hidden]
    output = affine(a1, W2, b2)
    try:
        W = [[affine([row[j] for row in W1], [weights], [0])[0]["z"] for j in range(d)] for weights in W2]
        b = [row["z"] for row in affine(b1, W2, b2)]
        rows = affine(x, W, b)
        difference = [finite(row["z"] - collapsed["z"], "Collapse difference") for row, collapsed in zip(output, rows)]
        collapse = dict(status="defined", W=W, b=b, rows=rows, difference=difference)
    except ValueError as error:
        collapse = dict(status="unrepresentable", reason=str(error))
    return dict(kind="network", hidden=hidden, a1=a1, output=output, collapse=collapse,
                shapes=dict(x=[d, 1], W1=[h, d], b1=[h, 1], z1=[h, 1], a1=[h, 1], W2=[o, h], b2=[o, 1], z2=[o, 1]))


if __name__ == "__main__":
    raw = sys.stdin.read().strip()
    request = json.loads(raw) if raw else dict(kind="neuron", x=[2, -1], w=[0.5, 2], b=0.25, activation="relu")
    try:
        print(json.dumps(forward(request), allow_nan=False))
    except ValueError as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)

Mistakes and numerical limits

Do not activate each contribution separately and then sum it: φ(Σwⱼxⱼ+b) is generally different from Σφ(wⱼxⱼ)+b. Do not discard a negative weight: its sign is part of the dot product. Do not omit the bias or silently change units. Do not report a ReLU zero as a class label, nor claim that a nonlinear function has learned from data without a fitting procedure.

Exact integer and dyadic worked examples need no approximation tolerance. Independent and unit comparisons check every expected zero exactly. Other nonexact sigmoid values compare to Decimal using relative 32ε plus an absolute 32ε for magnitudes at least 10⁻⁸, where ε=2⁻⁵². Tiny nonzero tails use relative-only comparison. Endpoint cases declared exact use equality; nonexact derived sums, including the sum of separately rounded sigmoid complements near 1, use the allowance rather than an endpoint rule. Browser numerical projections use the stated allowance, including an absolute 32ε at expected zero, independently of these exact oracle/unit assertions. This small-case allowance covers exponential implementation and final rounding; it is not a universal error guarantee or a way to accept a fabricated zero tail. SVG quantities retain full-precision data attributes; labels round for readability.

Five graded investigations

What you can now explain

You can carry the frozen affine predictor into a vector dot product, account for every term and bias, distinguish preactivation from activation, and evaluate stable sigmoid at extreme inputs. III-41 composes these mechanisms. To learn weights rather than choose them, Book IV will need a loss, derivatives and optimization: the Book II chain-rule and gradient-descent prerequisites explain why output sensitivity must reach every parameter. This Part stops at forward computation.