Article (Scientific journals)
Warming up recurrent neural networks to maximise reachable multistability greatly improves learning
Lambrechts, Gaspard; De Geeter, Florent; Vecoven, Nicolas et al.
2023In Neural Networks, 166, p. 645-669
Peer Reviewed verified by ORBi
 

Files


Full Text
1-s2.0-S0893608023003817-main.pdf
Publisher postprint (3.19 MB)
Download

All documents in ORBi are protected by a user license.

Send to



Details



Keywords :
Recurrent neural network; Multistability; Initialisation procedure; Long-term memory; Warmup; Long time dependencies
Abstract :
[en] Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called “warmup” to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.
Disciplines :
Computer science
Author, co-author :
Lambrechts, Gaspard  ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
De Geeter, Florent   ;  Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Systèmes et modélisation
Vecoven, Nicolas 
Ernst, Damien  ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Drion, Guillaume ;  Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
 These authors have contributed equally to this work.
Language :
English
Title :
Warming up recurrent neural networks to maximise reachable multistability greatly improves learning
Publication date :
August 2023
Journal title :
Neural Networks
ISSN :
0893-6080
eISSN :
1879-2782
Publisher :
Elsevier, United Kingdom
Volume :
166
Pages :
645-669
Peer reviewed :
Peer Reviewed verified by ORBi
Available on ORBi :
since 03 June 2021

Statistics


Number of views
550 (112 by ULiège)
Number of downloads
190 (32 by ULiège)

Scopus citations®
 
0
Scopus citations®
without self-citations
0
OpenCitations
 
0

Bibliography


Similar publications



Contact ORBi