Seeing Why Neural Networks Work — Four Live Proofs

There's a gap between understanding neural networks at the level of "matrix multiplications and nonlinearities" and genuinely feeling why they work the way they do. I spent a while on the wrong side of that gap.

The ERA v5 Session 1 assignment asks you to prove four core claims from first principles. Instead of writing Python notebooks, I built a live webapp where every experiment trains in real-time in your browser — the neural nets are implemented from scratch in vanilla JavaScript: forward pass, backprop, SGD, the full thing. No TensorFlow, no PyTorch, no abstraction layer between you and the math.

Here's what each experiment proves and why the result is surprising enough to be worth building.

The Setup

Generate 300 2D points: 150 form an inner ring (class 0), 150 form an outer ring (class 1). The two rings are concentric — no straight line can separate them. This is the simplest possible dataset that is not linearly separable.

Train two models on exactly this data:

• Model A (Linear): input → Linear(2→1) → Sigmoid. One layer, no hidden units.
• Model B (ReLU): input → Linear(2→32) → ReLU → Linear(32→1) → Sigmoid. One hidden layer with 32 units.

Same optimizer, same learning rate, same number of epochs. Only the architecture differs.

The Money Shot

After 1000 epochs, the linear model has learned to draw a straight line through the data. It's the best possible straight line — but a straight line can't encircle a ring. Accuracy hovers around 55%: barely better than random.

The ReLU model's decision boundary wraps around the inner ring. It doesn't find one line; it pieces together many local linear regions to approximate a curve. Accuracy: ~99%.

The mathematical reason: a composition of linear functions is still linear. f(g(x)) = (W₂W₁)x = Wₑₓₓ. You can add as many linear layers as you want; they all collapse to a single matrix. ReLU breaks that collapse.

Three Models, Same Data

Train three models on the ring dataset:

• 1-Layer Linear: 2 → 1 → sigmoid
• 5-Layer Linear: 2 → 4 → 4 → 4 → 4 → 1 → sigmoid, no activations between layers
• 5-Layer ReLU: Same shape, but ReLU after each hidden layer

The Collapse Proof

For the 5-layer linear model, after training is done, I multiply all five weight matrices together:

The boundary plots for 1-layer-linear and 5-layer-linear are visually identical: both are straight lines. The 5-layer ReLU model solves the task. Same training, same data, same number of layers. The only difference is whether ReLU breaks the linear chain between layers.

The Toy Language

Build a tiny synthetic grammar with 8 tokens and two sentence templates:

• Animals (3): cat, dog, cow
• Fruits (2): apple, mango
• Verbs (3): eat, chase, see
• Template: [animal] [verb] [animal | fruit]

Generate 1,200 (input, target) pairs by extracting consecutive token pairs from generated sentences. Train an embedding model: lookup(token) → Linear(16→8) → softmax → predict next token. The loss is cross-entropy. No similarity labels. No clustering objective. Just next-token prediction.

Emergent Structure

After training, take the 8×16 embedding table and project it to 2D with PCA (implemented from scratch with power iteration). Plot the result.

Animals cluster together. Fruits cluster together. Verbs cluster together. No one told the model this. It fell out of the loss function.

The nearest-neighbor for every animal is another animal. The nearest-neighbor for every verb is another verb. The demo updates the neighbor table live as training progresses — you can watch the structure emerge over ~300 epochs.

The Setup

Train the same large model — 2 → 64 → 64 → 1 with ReLU (~4,400 parameters) — on the ring data at three dataset sizes: 20, 200, and 2000 points. Keep a fixed test set of 500 points. Run 600 epochs and track train loss and test loss for all three runs simultaneously.

What You See

With 20 points and 4,400 parameters, the model has ~220× more parameters than data points. It can effortlessly memorize every training point — essentially learning the location of each dot rather than the ring structure. Train loss goes to near zero. Test loss stays high because the memorized solution doesn't generalize.

With 2000 points, the model can no longer memorize every individual point. The sheer variety of positions forces it to learn the underlying structure: inner ring vs outer ring. Train and test loss converge.

The Decision Boundary Visual

The demo also shows the decision boundaries side by side for all three training sizes. At n=20 the boundary is chaotic — fragments and islands of color fitting the 20 training points exactly. At n=2000 it's a clean circle. Same model. Different data.

---

How It's Built

Every neural network in this demo is implemented from scratch in ~600 lines of vanilla JavaScript. No TensorFlow.js. No ml5.js. Just:

• Matrix operations: matmul, transpose, hadamard, scalar ops — all plain arrays
• Forward pass: sequential layer execution with cache for backprop
• Backpropagation: full chain rule, handling ReLU, sigmoid, and linear activations
• SGD: full-batch gradient descent (no mini-batching for simplicity)
• PCA: covariance matrix + power iteration for top-2 eigenvectors
• Decision boundary: batch-evaluate a 70×70 grid and colorize via ImageData

The training runs asynchronously using requestAnimationFrame so the UI updates live without freezing the browser.

// The core of the backward pass — full chain rule from output to input
for (let i = L - 1; i >= 0; i--) {
  const prevA = i === 0 ? cache[0] : cache[i].A;
  const dW    = mm(tr(prevA), dZ);   // gradient for weights
  const db    = sumRows(dZ);          // gradient for biases

  if (i > 0) {
    const dAprev = mm(dZ, tr(W[i]));       // backprop through weights
    const Zprev  = cache[i].Z;             // cached pre-activations
    dZ = (layers[i-1].act === 'relu')
       ? had(dAprev, reluGrad(Zprev))      // ReLU: zero out negative slopes
       : dAprev;                           // linear: identity pass-through
  }
  W[i] = msub(W[i], smul(dW, lr));
  b[i] = vsub(b[i], vscale(db, lr));
}

The reason to implement it from scratch rather than using a library: you can't really claim to understand backpropagation until you've debugged a sign error in a chain-rule computation at 1am. Libraries abstract that experience away. Sometimes you want the pain.

---

References & Further Reading

• Hornik et al. (1989) — Multilayer feedforward networks are universal approximators. The theoretical basis for why ReLU networks can approximate any continuous function.
• Mikolov et al. (2013) — Distributed Representations of Words and Phrases and their Compositionality (word2vec). The paper that made the embedding-as-context-prediction connection explicit.
• Zhang et al. (2017) — Understanding deep learning requires rethinking generalization. The empirical paper that showed over-parameterized networks memorize random labels, forcing a rethink of classical generalization theory.
• Goodfellow et al. — Deep Learning, Chapter 6 (Feedforward Networks) and Chapter 7 (Regularization). The standard textbook treatment.

---

All source code at github.com/harshvardhanraju/ERAv5