Available on ORBi since
06 August 2022
Article (Scientific journals)
Recurrent networks, hidden states and beliefs in partially observable environments
Lambrechts, Gaspard  ; Bolland, Adrien  ; Ernst, Damien 
2022 • In Transactions on Machine Learning Research
Peer reviewed
 

Files


Full Text
belief-rnn-tmlr.pdf
Publisher postprint (3 MB) Creative Commons License - Attribution

All documents in ORBi are protected by a user license.

Send to



Details



Keywords :
partially observable environments; reinforcement learning; recurrent neural network; hidden state; belief; POMDP; RL; RNN
Abstract :
[en] Reinforcement learning aims to learn optimal policies from interaction with environments whose dynamics are unknown. Many methods rely on the approximation of a value function to derive near-optimal policies. In partially observable environments, these functions depend on the complete sequence of observations and past actions, called the history. In this work, we show empirically that recurrent neural networks trained to approximate such value functions internally filter the posterior probability distribution of the current state given the history, called the belief. More precisely, we show that, as a recurrent neural network learns the Q-function, its hidden states become more and more correlated with the beliefs of state variables that are relevant to optimal control. This correlation is measured through their mutual information. In addition, we show that the expected return of an agent increases with the ability of its recurrent architecture to reach a high mutual information between its hidden states and the beliefs. Finally, we show that the mutual information between the hidden states and the beliefs of variables that are irrelevant for optimal control decreases through the learning process. In summary, this work shows that in its hidden states, a recurrent neural network approximating the Q-function of a partially observable environment reproduces a sufficient statistic from the history that is correlated to the relevant part of the belief for taking optimal actions.
Disciplines :
Computer science
Author, co-author :
Lambrechts, Gaspard ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Bolland, Adrien ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Ernst, Damien ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Language :
English
Title :
Recurrent networks, hidden states and beliefs in partially observable environments
Publication date :
August 2022
Journal title :
Transactions on Machine Learning Research
Peer reviewed :
Peer reviewed
Tags :
CÉCI : Consortium des Équipements de Calcul Intensif
Funders :
FRIA - Fonds pour la Formation à la Recherche dans l'Industrie et dans l'Agriculture [BE]
Commentary :
A more concise version of this article is available at https://hdl.handle.net/2268/294541.

Statistics


Number of views
238 (39 by ULiège)
Number of downloads
94 (7 by ULiège)

Bibliography


Similar publications



Contact ORBi