In our day-to-day lives, we’re constantly making a slew of decisions from immediate matters to prospects on the far horizon. But the evolutionary nuts-and-bolts of how our brains weigh these numerous daily decisions and what role is played by the neurotransmitter serotonin has been shrouded in mystery.
Now, a new study led by an interdisciplinary uOttawa Faculty of Medicine team delivers fascinating findings on this big topic and potentially unravels a hidden aspect of what our nervous system’s complex serotonin system is really doing inside the enigmatic organ in our skulls.
Published in the journal Nature, this study from a highly impactful international collaboration was considered by one of the expert reviewers who evaluated the work to offer “broad implications across neuroscience, psychology, and psychiatry, enhancing our understanding of serotonin’s role in mood regulation, learning, and motivated behavior.”
The team’s innovative work merges ideas from reinforcement learning (RL) theory – used in neuroscience to better understand learning, behavior, and decision-making – with recent hard-won insights into the filtering properties of the brain’s dorsal raphe nucleus. That’s a region of the mammalian brainstem rich in serotonin-producing neurons.
Serotonin is often painted as the brain’s “pleasure chemical.” Antidepressant drugs such as selective serotonin reuptake inhibitors (SSRIs) famously target the serotonin system as part of a multi-billion-dollar industry. However, serotonin’s precise role in the nervous system is ambiguous and perplexing: It’s implicated in everything from mood and movement regulation to appetite and sleep-wake cycles. The fact that it’s activated by pain, pleasure and surprise has long been a brain research puzzle.
With this study, the uOttawa-led researchers put forth a unifying perspective on serotonin they dub a “prospective code for value” – a biological code for how the brain places a value for future rewards. This code essentially explains why serotonin neurons are activated in the brain in response to both rewards and punishments, with a preference for surprising rewards.
“Our work asks the question: What does serotonin tell the brain? In a nutshell, we find that its message closely matches the expectation of future rewards,” says senior author Dr. Richard Naud, associate professor at the Faculty of Medicine’s Department of Cellular and Molecular Medicineand the uOttawa Department of Physics.
Co-author Dr. Jean-Claude Béïque, professor in the Department of Cellular and Molecular Medicine, puts the main findings like this: “Your brain needs to compute the expected value of the actions you contemplate and undertake as you interact with a changing world, asking ‘What’s the value of this decision versus that decision in that particular environment?’ That’s a hard problem. So what we think serotonin actually does in the brain is encode the expected value of a particular environment or course of actions in order to ultimately guide everyday decisions.”
Drs. Béïque and Naud are both members of the uOttawa Brain and Mind Research Institute’s Centre for Neural Dynamics and Artificial Intelligence.
The Initial Spark
The germ of the idea began years ago at the uOttawa Faculty of Medicine when first author Emerson Harkin, then a PhD student in Dr. Naud’s lab, started to simulate reinforcement-learning models while working on the biophysical properties of serotonin neurons. Dr. Harkin, who finished his PhD work at uOttawa in late 2023 and was awarded his degree in March 2024, says the core idea came “half-serendipitously.”
After investing a lot of time and effort studying the electrical properties of the brain cells that produce serotonin, he and his uOttawa Faculty of Medicine supervisors started looking closely at findings from other labs that focused on measuring the activity of serotonin neurons in animals experiencing rewards and punishments. While the overall picture was extremely puzzling, he says it was then they realized they might be chasing something promising.
“Nobody seemed to have considered the possibility that serotonin neurons might be activated by changes in the animal’s surroundings, like the start of a signal that reward will arrive soon or the end of a punishment,” says Dr. Harkin, referring to lab experiments with mouse models. “When we looked at these previous results through the lens of what we had seen under our microscopes and with our electrodes, a lot of results that had previously seemed puzzling or contradictory suddenly started to fit together.”
Now in Germany doing a postdoc at the Max Planck Institute for Biological Cybernetics, Dr. Harkin asserts that the findings show a big part of what the serotonin system does is send a “message to the rest of the brain saying: “Here’s our best guess about how good your near future will be, and here’s how quickly that guess is improving.”
Dr. Naud, a computational neuroscience expert who drilled down on the theory’s deeply complex math, explains the initial idea evolved quite a bit over the last couple of years as the collaborative team finetuned the work with “lots of reading, lots of discussion, lots of thinking.”
Next Steps
Looking forward, the research team aims to study the role of serotonin on behavior to try and figure out what the rest of the brain does with the neurotransmitter’s messages. Dr. Naud says perhaps finding ways of employing reinforcement-learning theory on frameworks can help them do this.
Interestingly, Dr. Naud, whose neuroscience work often has implications for theories of learning and memory that can inform future developments in Artificial Intelligence (AI), believes the team’s study just published in Nature appears to show how special the nervous system really is.
“In a way I feel that the team’s findings show that the brain doesn't work the way machines do. If we perturb the signaling of rewards in the machine it would do many things that the perturbation in brains don't do,” he says.