Neural networks (NN) are the heart of any deep learning system. NN derives its function from biological neurons from the human brain. These neurons arranged in layers process information parallelly by comparing with the reference information fed earlier in the network system. Since these eventually “learn” from the information, they form the core structure of deep learning (DL).
The study of NNs has seen many developments, be it in the form of applications or even in the design of NN hardware models. Researchers are constantly trying to minimise the time for learning process in NN as well as the energy consumption in its working. In this article, we will discuss the application of analog electronic circuits in creating NNs and its benefits along energy efficiency as well as implementation in DL.
The Need For Analog Devices
NNs, if implemented on a large scale, require electrical energy on an equally giant level. Even though there are powerful hardware such as graphical processing units (GPU), they fail to quickly switch data and forth from memory and processing units to work for NNs. This has necessitated the use of traditional hardware devices such as analog circuits because they score more points over modern devices in terms of cost and energy, with only negligible data loss.
Another benefit would be the little effect on accuracy in analog devices unlike in modern digital devices which has binary operations for computations which would drastically affect if there are slight changes. Analog devices have continuous signals, which means processing activity does not stop in-between. This makes it ideal for NN processing since the power consumption is very less and fluctuations in the processing stage are minimal.
A large NN architecture consists of millions of neurons that are interspersed in layers. Every neuron gets its input from different neurons preceding from it, and deliver the output. This neuron interaction is evaluated in terms of ‘synaptic weight’. It means that the weights represent values of each node and vectors, and decide the relationship between input data and output data in the NNs. All of this process means higher and complex computations. Analog circuits reduce these overhead by effectively manipulating weight data.
Developments in analog NNs can be traced back to the ‘60s where it was the only choice of electronic framework for fields such as DL and NN. It all started with the computer simulation of a single neuron to analyse its electrical characteristics. Gradually representative models of neurons and neuron layers were built by referencing these simulations. These were improved over time in terms of response and input. The first instance of analog NNs was developed by academics at Harvard University, where the researchers brought out a continuous time analog NN with delays. The delay was due to the neuronal oscillations observed in the networks. This study acted as a base for the coming studies on analog implementations of NNs.
In the late ‘90s and ‘00s, there were a number of studies exploring stabilisation of these NNs for energy utilisation and integration with sophisticated electronic circuits. Ultimately, the potential in analog circuits was spurred due to these research developments.
IBM’s Novel Approach To Analog Electronics
In a latest study by IBM, the researchers developed analog circuits which utilise the concept of non-volatile memory and parallel computing. Stefano Ambrogio, the lead researcher in the study elaborates on the technique as follows:
“In our paper, we describe how analog non-volatile memories (NVM) can efficiently accelerate the “backpropagation” algorithm at the heart of many recent AI advances. These memories allow the “multiply-accumulate” operations used throughout these algorithms to be parallelised in the analog domain, at the location of weight data, using underlying physics. Instead of large circuits to multiply and add digital numbers together, we simply pass a small current through a resistor into a wire and then connect many such wires together to let the currents build up. This lets us perform many calculations at the same time, rather than one after the other. And instead of shipping digital data on long journeys between digital memory chips and processing chips, we can perform all the computation inside the analog memory chip.”
The study implements phase-change memory (PCM) devices, Complementary Metal-Oxide Semiconductor (CMOS) capacitors on the hardware aspect and simulations on the software aspect to even out variations in the output and achieve accuracy in the NN. The core idea was to focus on the NN weight distribution in detail which was necessary for building the analog circuits. The chips designed by the team showed a computation energy output of 28,065 billion operations per second which is staggeringly higher than the efficiencies achieved in the GPUs available in today’s market.
Conclusion
The above developments show how analog hardware could possibly be the forerunner of ML and AI hardware. Not just rejigging them, but also provide a faster and powerful hardware for future applications like DL. Above all, these hardware provide an avenue for being energy-efficient.
