[en] Computing the Bayesian posterior of a neural network is a challenging task
due to the high-dimensionality of the parameter space. Anchored ensembles
approximate the posterior by training an ensemble of neural networks on
anchored losses designed for the optima to follow the Bayesian posterior.
Training an ensemble, however, becomes computationally expensive as its number
of members grows since the full training procedure is repeated for each member.
In this note, we present Sequential Anchored Ensembles (SAE), a lightweight
alternative to anchored ensembles. Instead of training each member of the
ensemble from scratch, the members are trained sequentially on losses sampled
with high auto-correlation, hence enabling fast convergence of the neural
networks and efficient approximation of the Bayesian posterior. SAE outperform
anchored ensembles, for a given computational budget, on some benchmarks while
showing comparable performance on the others and achieved 2nd and 3rd place in
the light and extended tracks of the NeurIPS 2021 Approximate Inference in
Bayesian Deep Learning competition.
Disciplines :
Computer science Mathematics
Author, co-author :
Delaunoy, Arnaud ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Louppe, Gilles ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Big Data
Language :
English
Title :
SAE: Sequential Anchored Ensembles
Publication date :
14 December 2021
Event name :
Bayesian Deep Learning, NeurIPS 2021 workshop
Event date :
December 14, 2021
Audience :
International
Peer reviewed :
Peer reviewed
Funders :
F.R.S.-FNRS - Fonds de la Recherche Scientifique NRB
Funding text :
The authors would like to thank Tim Pearce for his insightful comments and feedback. Arnaud Delaunoy would like to thank the National Fund for Scientific Research (F.R.S.-FNRS) for his scholarship. Gilles Louppe is recipient of the ULiège - NRB Chair on Big Data and is thankful for the support of the NRB.
Commentary :
4 pages, NeurIPS 2021 Bayesian Deep Learning workshop
