Thanks to a mathematical breakthrough achieved at CWI, AI applications like speech recognition, gesture recognition and electrocardiogram (ECG) classification can become a hundred to a thousand times more energy efficient. This means it will be possible to put much more elaborate AI in chips, enabling applications to run locally on a smartphone or smartwatch instead of in the cloud.
Since 2012, the field of artificial intelligence (AI) has made great steps forward thanks to a technique called deep learning. Deep learning has led to numerous practical applications: Apple uses it in the voice recognition of Siri, Facebook automatically tags photos with it and Google Translate translates texts with it, to name just a few.
Deep learning is based on information processing by large artificial neural networks that have dozens or even hundreds of layers. It doesn’t come for free, however. Since 2012, training the largest deep neural networks has become 300,000 times more computationally intensive. Every few months, training costs have doubled. The training of text generator GPT-3, for example, which amazed the world in 2020 by writing human-style texts in all kinds of styles, cost the annual consumption of 300 Dutch households (1 GWh, 4.1 million euros on the electricity bill).
Even the new national supercomputer Snellius, which the Netherlands bought in 2021, still uses one megawatt. Compare this energy consumption with that of the human brain, which is around twenty watts, a factor of 50,000 lower. Although the supercomputer can do scientific calculations at a speed no human can even come close to, the human brain can perform many cognitive tasks the supercomputer can’t do at all. And even much smaller neural networks often consume too much energy to apply them in mobile applications, as we’d like to do more and more.
The result is that AI algorithms typically run ‘in the cloud’ where the data is sent from our smartphones to the data center to be processed. Apart from the energy considerations, this means that there’s also a serious privacy concern with the current approach.
The development of AI that’s efficient enough to run on local hardware has therefore become increasingly important. My colleagues and I take inspiration from the human brain and in 2021, we published a significant breakthrough in computing with so-called spiking neural networks (SNNs). We’ve demonstrated that thanks to this breakthrough, dedicated chips can recognize speech, gestures and electrocardiograms (ECGs) twenty to a thousand times more energy-efficiently than traditional AI techniques.
Classical neural networks use signals that are easy to handle mathematically. Because these signals are continuous, it’s relatively straightforward to calculate how a change in the network will affect the overall computation. SNNs calculate with pulses, which is much more like what happens in the brain and takes less energy but has the disadvantage that the signals are discontinuous and it’s much harder to determine how local changes affect the overall network behavior. However, we’ve found a mathematical solution to that problem. In our new algorithm, the neurons in the network need to communicate much less with each other and, in addition, each neuron needs to do fewer calculations.
We tested our computer algorithm on three benchmarks that consisted of test sets of about ten gestures, a series of words and a continuous ECG signal. SNNs trained with our algorithm performed at least as well as traditional deep neural networks. However, as the spiking neurons in the network communicate only very sparingly, we achieve a highly improved energy efficiency: when computing the required number of network calculations, we gain a factor of a hundred to a thousand over the traditional networks.
A key trick for this efficiency is that spiking neurons themselves maintain a small piece of memory: internal states regulate a neuron’s behavior, like whether it will emit a spike, and these states are updated at each timestep. The parameters with which the states are updated can in turn be learned during training. The result is that each spiking neuron learns to integrate the signals it receives over the timescales of the task. Looking at it differently, this finding also suggests that neurons in the brain may similarly tune the corresponding biological parameters, as there’s a functional benefit.
Unfortunately, current AI chips are designed for computing classical neural networks. To run spiking neural networks efficiently in the real world, a new type of chips is needed. To exploit the theoretical energy efficiency of SNNs, dedicated chips have to be built that facilitate spiking neurons at the hardware level. Techniques like in-memory computing and novel materials will play a key role there. Based on our algorithms, our research partner Imec has produced a special neuromorphic chip with 336 pulsed neurons: the μBrain chip. When we run our algorithm on it, we gain a factor of twenty to forty in real-world energy consumption, depending on the desired level of accuracy.
The energy gain in practice is always lower than in theory because of the conversion of digital to analog signals and vice versa, and because of the reading of data. But twenty to forty times still makes a massive difference: for detecting heart defects, it means that you can implant an ECG-recording chip and it will run for a year on a single battery.
Neuromorphic chips are on the verge of practical, everyday applications. In the coming years, they’ll contain more and more pulsed neurons, which will further expand the application possibilities of AI in wearable systems. For example, at the end of September 2021, Intel introduced the neuromorphic chip Loihi 2, which already contains one million pulsed neurons.
The human brain is living proof that energy-efficient intelligence is possible. It’s a great challenge for researchers to build energy-efficient AI through improvements in both hardware, for example through analog neuromorphic chips, and software, for example with more efficient neural networks, like our SNNs. At the same time, the close correspondence between spiking neural networks and their biological counterparts may lead to new insights into biology, such as where to look for the biological equivalent of our flexible parameters.
Main picture credit: Dirk Gillissen