Figure 1: This is not a video game—it is an example of a complex virtual-reality environment being used in research settings Image source: Creating immersive virtual reality experiences using human emotion
Virtual reality and electrodes implanted into the brain may feel like a plotline in Cyberpunk 2077; however, while you may be enjoying your favorite video game, researchers in the lab of Dr. Julio Martinez-Trujillo are using the same technology to answer questions about how the brain creates visual working memory.
Working memory is how the brain stores and uses information in the short-term to carry out a task. Working memory is an essential part of our daily lives: it helps us sort and store relevant information, allowing us to make quick everyday decisions. Imagine you are going to meet your friend for a picnic. You see them at the top of the hill, but they duck out of view to grab sandwiches from the car. Your visual working memory works with your attention to help you focus on reaching the top of the hill, instead of getting distracted by the dog playing fetch nearby. Working memory also allows you to store a spatial “picture” of where your friend was just standing. Equipped with this mental picture and with your attention focused, you can reach the top of the hill and start setting up the picnic blanket, even though you can no longer see your friend.
While researchers have a good understanding of how the brain represents working memory in simple tasks, it has been difficult to understand how neurons encode working memory in more complex tasks like the picnic example above. Most real-world environments have a lot of stimuli, or “noise”, for the brain to sort through- the mix of sounds in a busy café, the layers of hills on the horizon, or a bird flying overhead. When you are using working memory to perform a complex task in this environment, your brain is working hard behind the scenes to organize and prioritize information.
The researchers in this study wanted to see the patterns of electrical signals neurons create during tasks with visual targets that move in and out of view, like your friend in the example. They believed that the electrical firing patterns of neurons might be more complicated in this kind of task, because the brain would have to store the memory of your friend’s previous location and actively compete with other, more current visual noise.
However, recreating environments that mimic complex real-world scenarios like this is hard to do in a laboratory. This poses a problem for researchers: while simple experimental tasks can be helpful to understanding basic processes in the brain, if they do not accurately represent the conditions we experience in the real world, it can be hard to test how our brains respond to complicated real-world problems. In 2024, a collaboration of mathematicians and neuroscientists at the University of Western Ontario had a creative approach to tackling this problem: using the same programming software used in video games, they created a 3D virtual reality environment to run their experiments in. In this case, a “virtual reality” environment is a digital environment that can be seen and moved around in three dimensions. Most modern video games use these environments.
During the experiment, the monkeys were placed in a dark room and shown a 3D virtual reality environment, with a red box (a “cue”) in the middle of the screen. Once the monkeys got a good look at the cue, it disappeared from the environment. After a short delay during which the monkeys could look at any point in the environment but could not move, the monkeys were tasked with moving in the VR space to where the cue used to be. If they managed to reach the area in a specified timeframe, they were given a fruit juice reward. To see how neurons were firing throughout this task, researchers used microelectrode arrays – grids of electrodes that can record electrical activity of individual neurons in a specific area of the brain. These arrays were implanted into the monkeys to record activity in the lateral prefrontal cortex, a brain region important to working memory and decision-making. The electrodes have no impact on the monkey’s ability to think and behave normally. They act as observers, taking note of what processes are happening in the brain in real time.
Figure 2: Images of the virtual environment used in this study, with and without the cue. With this environment, researchers were able to add trees, mountains, and hills to the scene to add visual “noise”. It also allowed the monkeys to look and move anywhere in a 360-degree radius. Source: Fig. 1: Experimental design for exploration of sequential activity. | Nature Communications | Nature Communications
Using this technology, the research team found two interesting things. First, they discovered that different neurons were firing more often than expected during different parts of the task. Some neurons fired more upon seeing the cue, while others fired more when remembering where the cue was located or getting to the target area. This may mean that different neurons represent different stages of working memory.
Second, the researchers created “clusters” of these neurons and compared them to the visual distance between the monkeys and the cue area they were trying to reach as they moved around the space. They found that these clusters of firing neurons mapped onto different viewpoints from the monkey’s perspective. These two findings indicate that, instead of the brain representing working memory in one consistent, unchanging “mental map”, this map may change depending on where the monkey is in space and time.
Previous research using simpler working memory tasks does not demonstrate these complicated neural patterns. By using VR and electrode implants, these scientists were able to record a more realistic and previously unseen representation of how our brain may be processing working memory in complex, real-world scenarios.
Not only is virtual reality a fun and immersive way to experience the latest video game, but it is also a very useful new technology for researchers. The technology used in this study allowed researchers to better understand how our neurons work when we are using working memory in a 3D space. Bringing virtual environments into the laboratory, where researchers are often limited in their ability to create realistic testing scenarios, may be a game-changer.
Original Article: Busch, A., Roussy, M., Luna, R., Leavitt, M. L., Mofrad, M. H., Gulli, R. A., Corrigan, B., Mináč, J., Sachs, A. J., Palaniyappan, L., Muller, L., & Martinez-Trujillo, J. C. (2024). Neuronal activation sequences in lateral prefrontal cortex encode visuospatial working memory during Virtual Navigation. Nature Communications, 15(1). https://doi.org/10.1038/s41467-024-48664-9