Under the hood

How I grow new neurons (function-preserving widening)

When Big AI Brain earns a new stage, it doesn't start over with a bigger, empty brain. It grows the one it has. Here's the trick, and the actual code that does it.

The problem

A neural network has a fixed number of neurons. Want a bigger one? Normally you build a new, bigger network and train it from scratch, throwing away everything the small one learned. For a brain that's supposed to grow up in public, that would be like wiping its memory every birthday.

The trick

In 2016, researchers Tianqi Chen, Ian Goodfellow and Jonathon Shlens published a method they nicknamed Net2Net. The idea is almost embarrassingly simple:

1. Copy a neuron. Pick an existing neuron and make an exact duplicate, with the same incoming connections. The copy "sees" exactly what the original sees.
2. Split its voice. The original used to send its signal forward with some strength. Now there are two of them, so each sends half. Half plus half equals the same total.
3. Nothing changes, yet. Right after growing, the network gives exactly the same answers as before. That's the "function-preserving" part. Nothing was forgotten.
4. Then they drift apart. A tiny bit of random noise breaks the tie, and during the next study session the twins learn different jobs. Now the brain has more room to think.

The real code

This is a trimmed version of the function inside Big AI Brain. W_in holds the connections coming into the hidden layer, W_out the connections going out.

def widen(W_in, W_out, extra, rng):
    src = rng.integers(0, W_in.shape[1], size=extra)   # which neurons to copy
    counts = np.bincount(src, minlength=W_in.shape[1]) + 1
    W_in  = np.concatenate([W_in, W_in[:, src]], axis=1)  # copy their inputs
    W_out = W_out / counts[:, None]                        # share each output...
    W_out = np.concatenate([W_out, W_out[src, :]], axis=0) # ...between original and copies
    return W_in, W_out   # same answers, more neurons

Big AI Brain starts with 384 neurons in each hidden layer and grows 96 more every time it earns a stage, up to 1,536. The counter on the homepage is live.

← All lessons