Playing Atari with Six Neurons
Giuseppe Cuccu - University of Fribourg
May 17, 2019, 2 p.m. - April 16, 2019, 3 p.m.
McConnell 103
Hosted by: Jérôme Waldispühl
Deep reinforcement learning, applied to vision-based problems like Atari games, maps pixels directly to actions; internally, the deepneural network bears the responsibility of both extracting usefulinformation and making decisions based on it. By separating the image processing from decision-making, one could better understandthe complexity of each task, as well as potentially find smaller policyrepresentations that are easier for humans to understand and may generalize better. To this end, we propose a new method for learningpolicies and compact state representations separately but simultaneously for policy approximation in reinforcement learning. State representations are generated by an encoder based on two novel algorithms: Increasing Dictionary Vector Quantization makes the encoder capable of growing its dictionary size over time, to address new observations as they appear in an open-ended online-learning context; Direct Residuals Sparse Coding encodes observations by disregarding reconstruction error minimization, and aiming instead for highest information inclusion. The encoder autonomously selects observations online to train on, in order to maximize code sparsity. As the dictionary size increases, the encoder produces increasingly larger inputs for the neural network: this is addressed by a variation of the Exponential Natural Evolution Strategies algorithm which adapts its probability distribution dimensionality along the run. We test our system on a selection of Atari games using tiny neural networks of only 6 to 18 neurons (depending on the game’s controls). These are still capable of achieving results comparable—and occasionally superior—to state-of-the-art techniques which use two orders of magnitude more neurons.
Giuseppe Cuccu is a Senior Researcher at the eXascale InfoLab, University of Fribourg, Switzerland. He obtained my Ph.D. in 2018 from the University of Fribourg under the supervision of Philippe Cudré-Mauroux. Prior to that he worked for different start-ups (and founded one), and as a Research Assistant with the Dalle Molle Institute for Artificial Intelliggence in Lugano, Switzerland. He completed both my bachelor and master (summa cum laude) at the University of Milano-Bicocca, Italy.
