DeepMind has found that a machine learning technique called distributional reinforcement learning also provides a new explanation for how the reward pathways in the brain work. These are the pathways that govern the brain’s response to pleasurable events and are mediated by neurons that release the brain chemical dopamine.
“Dopamine in the brain is a type of surprise signal,” says Will Dabney. “When things turn out better than expected, more dopamine gets released.”
DeepMind team found that individual dopamine neurons actually seem to vary based on a different level of optimism or pessimism. “They all end up signaling at different levels of surprise,” says Dabney. “More like a choir all singing different notes, harmonizing together.”
The finding drew inspiration from a process known as distributional reinforcement learning, which is one of the techniques AI has used to master games such as Go. Reinforcement learning uses rewards to reinforce the behavior. It requires an understanding about how a current action leads to a future reward. For example, a dog may learn the command “sit” because it is rewarded with a treat when it does so.
“When someone plays the lottery, for example, they expect to win or they expect to lose, but they don’t expect this halfway average outcome that doesn’t necessarily really occur,” he says.
When the future is uncertain, the possible outcomes can instead be represented as a probability distribution: some are positive, others negative. AIs that use distributional reinforcement learning algorithms are able to predict the full spectrum of possible rewards.
To test whether the brain’s dopamine reward pathways also work via a distribution, the team recorded responses from individual dopamine neurons in mice. The mice were trained to perform a task and were given rewards of varying and unpredictable sizes.
The researchers found that different dopamine cells showed reliably different levels of surprise.
