What's Next? Unlocking the Brain's Ability to Predict Visual Experience using Artificial Intelligence
This image was created with the assistance of DALL·E 3
Have you ever wondered how your brain effortlessly translates the complex world into your rich visual experience? Thanks to a recent breakthrough in neuroscience and artificial intelligence, we understand that the brain can predict what we are going to see even before we see it!
A recent discovery from researchers at Western University and the Salk Institute found that waves of brain activity drive what we will see next. Beyond enhancing our understanding of the fundamentals of human perception, this research continues to show how artificial intelligence can inspire further study of ourselves.
Research estimates that 50% of the brain is directly or indirectly involved with vision. Part of the reason the brain focuses so heavily on vision is because it is so essential yet so complex. If you imagine for a minute a sunset, we take for granted that the brain instantly produces our experience of the redness of the sky, the circular sun, the birds flying in the sky, and the clouds moving across the horizon. The brain effortlessly combines these colours, shapes, objects, and motion into a single coherent experience!
But there's more to vision than just processing images; our brain also must predict what comes next. Picture yourself as a baseball player at bat – and the pitcher has just thrown a ball toward you. If you are a professional player, this ball is coming at you at a speed of around 42 meters/second. Before you can even process the original position of the ball, it is already over halfway toward you. Thankfully, your brain makes a prediction about how the ball is moving, which is done mostly without our awareness. However, much about how the brain accomplishes this remains a mystery.
The researchers at Western University and the Salk Institute have focused on understanding how we accomplish this amazing feat of prediction. They hypothesized that viewing an initial image (e.g., the pitcher throwing a ball) produces a wave of activity in the brain region responsible for vision, and this wave helps constrain what we are likely to see next. Notably, they believed that an often-overlooked connection in the visual part of the brain is necessary for these waves to occur.
But first – some context. The story of vision begins at the eyes, which convert light to signals that are understood by the brain. These signals travel down the optic nerve and to the back of the brain. This is where the magic happens.
At the back of your brain lies the visual cortex. The visual cortex consists of billions of tiny information processors – called neurons. Neurons can be thought of as tiny light switches, turning on and off. Neurons communicate with other neurons by turning off or on the lights of their neighbours, enabling connections to be formed between multiple neurons. These different patterns of light form the basis of everything you see!
Neurons connect with each other in two main ways. There are feedforward connections where neurons transmit signals up the brain. The second type is a recurrent connection, which go across the brain. Importantly, only recurrent connections can produce wave-like activity. This wave enables visual prediction by controlling whether neurons in the visual cortex turn on or off.
Schematic showing both feedforward (arrows moving up) and recurrent (arrows moving between) some neurons (dots). The neurons can either be turned on (yellow) or off (black).
The authors simulated the visual cortex using a novel type of artificial intelligence, called a complex-value neural network (cv-NN). The cv-NN is designed to mathematically represent complicated images in our environment. Put simply, the cv-NN takes the representation of those images and tries to predict what image or set of images are likely to come next.
To test their hypotheses the researchers made three separate cv-NNs. The first one simulates feedforward connections, the second simulates recurrent connections like the ones we have in our visual cortex, and the third was a control condition, which has recurrent connections but are unlike those in the visual cortex. They wanted to see which approach could best predict future images.
As the authors predicted, the cv-NN with recurrent connections like the ones in the visual cortex was the only one that could successfully predict future images. The authors tested it on simple images (such as a circle moving across a screen) and complex (black and white silhouettes of people walking), both confirming their hypothesis.
Despite this being a simulation, the authors took several steps to make their experiment closely align with our real brain. In fact, the prediction of the next image often worked best when the cv-NN mimicked known factors of the visual cortex, such as the overall proportion of connections as well as the speed of those connections.
This study furthers our understanding of the necessary relationship between vision and prediction. It highlights the central role of recurrent connections, a previously understudied research topic. Furthermore, the implications move far beyond simply predicting when to hit a baseball, but to many of the visual experiences that you find important, whether it is safely driving your car to work or taking in a beautiful sunset.
Finally, the results are yet another example of the beautiful duality of neuroscience and artificial intelligence. Where artificial intelligence tells us about essential aspects of our experience and how the structure of our brains can lead to the development of novel artificial intelligence. In sum, while we cannot know what the future of neuroscience looks like, you should predict good things to come!
Original article:
Benigno, G. B., Budzinski, R. C., Davis, Z. W., Reynolds, J. H., & Muller, L. (2023). Waves traveling over a map of visual space can ignite short-term predictions of sensory input. Nature Communications, 14(1), https://doi.org/10.1038/s41467-023-39076-2