As you read this news piece, you are getting help from the ambient light that hits the photoreceptors in your eyes (the sensing step – remember this) and gets converted into electrical signals (the computing step – also remember this) so that your brain (the visual cortex) can make sense of the letters that appear in this article. Biologically inspired machine vision exploits the same principle to progress to the point where artificial systems can “see”.
For instance, when you use your smartphone to take a picture, you have probably noticed that it can identify objects in a scene before you press the record button. Your smartphone camera is actually considered to be a modern image sensor. Such semiconductor-based image sensors capture the surrounding visual information and then pass it along to the processing units. It is then the job of the processing units to decode the optical signals and convert them into digital output. This movement of data between the sensor and processing units not only requires high power consumption, but also results in high computational latency – in the order of milliseconds. But what if a machine could see in nanoseconds?
Researchers in Austria have designed a network of image sensors in which images are encoded as bright pixels with varying optical intensities. They have demonstrated that by tuning the sensitivity of the sensors, in terms of pixel brightness, their new device is capable of self-computation and therefore bypasses the need to relay the signals to higher-level processing units (Nature 10.1038/s41586-020-2038-x).
Design and working principle
Inspired by the natural interconnected architecture of the brain, Lukas Mennel and colleagues from Thomas Mueller’s group at the Institute of Photonics, Vienna University of Technology, implemented an artificial neural network (ANN) in their image sensor to overcome the high-latency computing issue. They put together rows and columns of photodiodes – tiny, light-sensitive semiconductors, each having a few layers of tungsten diselenide – sequentially on a chip to create a photosensor network.
Neurons, the interconnected elements of the brain, are connected to each other by synapses, with the strength of the synaptic connections playing a critical role in neuronal information processing. Aware of this concept, Mueller’s research group designed the network such that each semiconductor’s response to light can be strengthened or weakened by applying an external voltage. A change in the voltage thus results in a change in the connection (synaptic) strength. This tuning capability then set the stage for the researchers to take advantage of machine learning algorithms such as ANNs.
Combining sensors and machine learning
The researchers implemented two types of ANNs: a classifier and an autoencoder. A classifier learns to classify images into different categories after a series of training processes (supervised learning), while an autoencoder recognizes a characteristic component or structure of an image from input data, without extra information (unsupervised learning).
In their design, the responsivities of the photodiodes under optical illumination were set by gate voltages, which themselves were results of either supervised or unsupervised learning processes. Changes in the optical intensity affect the output of the 3×3 array of pixels, enabling the device to self-sense and self-compute in nanoseconds.
“We have presented an ANN vision sensor for ultrafast recognition and encoding of optical images. The device concept is easily scalable and provides various training possibilities for ultrafast machine vision applications,” the authors conclude. This is indeed a promising technology, however, there is more to be done for it to be used in practical applications. For instance, imaging under dim light would be difficult. Redesigning the device to improve its semiconductors’ light absorption could increase the range of light intensities that it can detect.
