Graphical Normalizing Flows

Wehenkel, Antoine; Louppe, Gilles

Download

Paper published in a book (Scientific congresses and symposiums)

Graphical Normalizing Flows

Wehenkel, Antoine; Louppe, Gilles

2021 • In Proceedings of AISTATS 2021

Peer reviewed

Permalink
https://hdl.handle.net/2268/252739

Files (2)Send to Details Statistics Bibliography Similar publications

Files

Full Text

Graphical_Normalizing_Flows.pdf

Publisher postprint (1.53 MB)

Download

Annexes

NFsandBNs.pdf

Publisher postprint (5.2 MB)

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Normalizing Flows; Bayesian Networks; Density Estimation; Density Estimator; Probability Distribution; Machine Learning; Unsupervised Learning

Abstract :

[en] Normalizing flows model complex probability distributions by combining a base distribution with a series of bijective neural networks. State-of-the-art architectures rely on coupling and autoregressive transformations to lift up invertible functions from scalars to vectors. In this work, we revisit these transformations as probabilistic graphical models, showing they reduce to Bayesian networks with a pre-defined topology and a learnable density at each node. From this new perspective, we propose the graphical normalizing flow, a new invertible transformation with either a prescribed or a learnable graphical structure. This model provides a promising way to inject domain knowledge into normalizing flows while preserving both the interpretability of Bayesian networks and the representation capacity of normalizing flows. We show that graphical conditioners discover relevant graph structure when we cannot hypothesize it. In addition, we analyze the effect of `1-penalization on the recovered structure and on the quality of the resulting density estimation. Finally, we show that graphical conditioners lead to competitive white box density estimators

Disciplines :

Mathematics

Author, co-author :

Wehenkel, Antoine ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Big Data

Louppe, Gilles ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Big Data

Language :

English

Title :

Graphical Normalizing Flows

Publication date :

13 April 2021

Event name :

The 24th International Conference on Artificial Intelligence and Statistics

Event date :

April 2021

Audience :

International

Main work title :

Proceedings of AISTATS 2021

Peer reviewed :

Peer reviewed

Funders :

F.R.S.-FNRS - Fonds de la Recherche Scientifique [BE]

Available on ORBi :

since 18 November 2020

Statistics

Number of views

65 (11 by ULiège)

Number of downloads

33 (3 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

Bibliography

Matúš Bezek. Characterizing dag-depth of directed graphs. arXiv preprint arXiv:1612.04980, 2016.
David Maxwell Chickering, David Heckerman, and Christopher Meek. Large-sample learning of Bayesian networks is NP-hard. Journal of Machine Learning Research, 5(Oct):1287-1330, 2004.
Nicola De Cao, Ivan Titov, and Wilker Aziz. Block neural autoregressive flow. arXiv preprint arXiv:1904.04676, 2019.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Uncertainty in Artificial Intelligence, pages 1263-1273. PMLR, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In International Conference in Learning Representations workshop track, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference in Learning Representations, 2017.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pages 7509-7520, 2019.
F.J. Díez, J. Mira, E. Iturralde, and S. Zubillaga. DIAVAL, a Bayesian expert system for echocardiography. Artif. Intell. Med., 10(1):59-73, May 1997. URL https://doi.org/10.1016/s0933-3657(97) 00384-9.
Dan Geiger, Thomas Verma, and Judea Pearl. d-separation: From theorems to algorithms. In Uncertainty in Artificial Intelligence, volume 10, pages 139-148. Elsevier, 1990. URL https://doi.org/10.1016/b978-0-444-88738-2.50018-x.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881-889, 2015.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Machine Learning, 2018.
David S Greenberg, Marcel Nonnenmacher, and Jakob H Macke. Automatic posterior transformation for likelihood-free inference. arXiv preprint arXiv:1905.07488, 2019.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. arXiv preprint arXiv:1804.00779, 2018.
Priyank Jaini, Kira A Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. arXiv preprint arXiv:1905.02325, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. In Advances in neural information processing systems, pages 2946-2954, 2016.
Charles E. Kahn, Linda M. Roberts, Katherine A. Shaffer, and Peter Haddawy. Construction of a Bayesian network for mammographic diagnosis of breast cancer. Comput. Biol. Med., 27(1):19-29, January 1997. URL https://doi.org/10.1016/s0010-4825(96)00039-x.
Ilyes Khemakhem, Ricardo Pio Monti, Robert Leech, and Aapo Hyvärinen. Causal autoregressive flows. arXiv preprint arXiv:2011.02268, 2020.
Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. arXiv preprint arXiv:1811.02155, 2018.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10236-10245, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pages 4743-4751, 2016.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Daphne Koller and Nir Friedman. Probabilistic graphical models: Principles and techniques. MIT press, 2009.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-based neural dag learning. arXiv preprint arXiv:1906.02226, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016.
Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In International Conference on Machine Learning, pages 3895-3904, 2018.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338-2347, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019a.
George Papamakarios, David C Sterratt, and Iain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), 2019b.
Judea Pearl and Stuart Russell. Bayesian networks. California Digital Library, 2011.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEEE, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538, 2015.
Oren Rippel and Ryan Prescott Adams. High-dimensional probability estimation with deep density models. arXiv preprint arXiv:1302.5125, 2013.
K. Sachs, O. Perez, D. Pe'er, D. A. Lauffenburger, and G. P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308:523-529, 2005.
Robert Sedgewick and Kevin Wayne. Algorithms. Addison-wesley professional, 2011.
Flávio Luiz Seixas, Bianca Zadrozny, Jerson Laks, Aura Conci, and Débora Christina Muchaluat Saade. A Bayesian network decision model for supporting the diagnosis of dementia, alzheimer's disease and mild cognitive impairment. Comput. Biol. Med., 51:140-158, August 2014. URL https://doi.org/10.1016/j.compbiomed.2014.04.010.
Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. The MIT Press, 2001. URL https://doi.org/10.7551/mitpress/1754.001.0001.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66 (2):145-164, 2013.
Esteban G Tabak, Eric Vanden-Eijnden, et al. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1): 217-233, 2010.
Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. An introduction to probabilistic programming. arXiv preprint arXiv:1809.10756, 2018.
Rianne Van Den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05649, 2018.
Aäron Van Den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lock-hart, Luis Cobo, Florian Stimberg, et al. Parallel WaveNet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, pages 3915-3923, 2018.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. In Advances in Neural Information Processing Systems, pages 1543-1553, 2019.
Antoine Wehenkel and Gilles Louppe. You say normalizing flows i see bayesian networks. arXiv preprint arXiv:2006.00866, 2020.
Christian Weilbach, Boyan Beronov, Frank Wood, and William Harvey. Structured conditional continuous normalizing flows for efficient amortized inference in graphical models. In International Conference on Artificial Intelligence and Statistics, pages 4441-4451. PMLR, 2020.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pages 7154-7163, 2019.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pages 9472-9483, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics, pages 3414-3425. PMLR, 2020.