According to new research from Los Alamos National Laboratory, artificial neural networks can become unstable if they do not ‘sleep’.
Los Alamos National Laboratory computer scientist Yijing Watkins and her research team found that when they exposed their artificial neural networks to states that are analogous to the waves that living brains experience during sleep, stability was restored. “It was as though we were giving the neural networks the equivalent of a good night’s rest,” said Watkins.
Unsupervised instability
The research team initially aimed to develop neural networks that closely approximate how biological systems learn to see. The group struggled with stabilising simulated neural networks undergoing unsupervised dictionary training, which involves classifying objects without having prior examples to compare them to.
Los Alamos computer scientist and study co-author Garrett Kenyon said: “The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilise biologically realistic, spiking neuromorphic processors or when trying to understand biology itself. The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”
Mimicking human cycles
The team decided to expose the networks to an artificial analogue of sleep to stabilise them. They experimented with various types of noise, the best results were found when the team used waves of Gaussian noise, which includes a wide range of frequencies and amplitudes. They hypothesise that the noise mimics the input received by biological neurons during slow-wave sleep.
The groups’ next goal is to implement their algorithm on Intel’s Loihi neuromorphic chip. Researchers aim to allow Loihi to sleep from time to time, enabling it to stably process information from a silicon retina camera in real time. If the findings confirm the need for sleep in artificial brains, we can probably expect the same to be true of androids and other intelligent machines that may come about in the future.
