Each Akida NSoC has effectively 1.2 million neurons
and 10 billion synapses, representing orders of
magnitude better efficiency than other neural
processing devices on the market.
How all those neurons and synapses fit into that chip??
The individual components may be quite small. The technology appears to be a variant of spiking neural network, where instead of RAM/multiply-add units for storing data and performing the matrix multipications digitally, the technology is effectively analog - albeit using pulse frequency modulation (hence the "spiking" name) rather than a continuously varying voltage.
The basic operation of a neuron can be modeled as the dot product (a∙b) of a array of pulse modulated inputs (a) and an array of pre-programmed memristors (b). The current delivered by each input is given by a
ib
i, where a
i is the pulse frequency of input i, and b
i is the programmed conductivity of memristor i. By connecting all inputs to a summing capacitor, the dot product operation is complete. So, you can have a neuron with 1k inputs consisting of a capacitor, sense amp/pulse frequency modulator and 1000 memristor cells which receive inputs from 1000 other neurons. The memristor cells can be manufactured on the scale of flash memory cells, and multilayer manufacturing (similar to 3D NAND flash) has also been demonstrated, so 10 G memristor cells on one chip doesn't seem completely unreasonable.