Learning Diffusion Priors from Observations by Expectation Maximization

Rozet, François; Andry, Gérôme; Lanusse, François; Louppe, Gilles

Download

Paper published in a journal (Scientific congresses and symposiums)

Learning Diffusion Priors from Observations by Expectation Maximization

Rozet, François; Andry, Gérôme; Lanusse, François et al.

2024 • In Advances in Neural Information Processing Systems, 37

Peer Reviewed verified by ORBi

Permalink
https://hdl.handle.net/2268/319891

arXiV
2405.13712v1

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

diem.pdf

Author postprint (7.73 MB)

Creative Commons License - Public Domain Dedication

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Computer Science - Learning; Statistics - Machine Learning

Abstract :

[en] Diffusion models recently proved to be remarkable priors for Bayesian inverse problems. However, training these models typically requires access to large amounts of clean data, which could prove difficult in some settings. In this work, we present DiEM, a novel method based on the expectation-maximization algorithm for training diffusion models from incomplete and noisy observations only. Unlike previous works, DiEM leads to proper diffusion models, which is crucial for downstream tasks. As part of our methods, we propose and motivate an improved posterior sampling scheme for unconditional diffusion models. We present empirical evidence supporting the effectiveness of our approach.

Disciplines :

Computer science

Author, co-author :

Rozet, François ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Big Data

Andry, Gérôme ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Lanusse, François

Louppe, Gilles ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Big Data

Language :

English

Title :

Learning Diffusion Priors from Observations by Expectation Maximization

Publication date :

22 May 2024

Event name :

The Thirty-Eighth Annual Conference on Neural Information Processing Systems

Event place :

Vancouver, Canada

Event date :

December 10-15, 2024

Audience :

International

Journal title :

Advances in Neural Information Processing Systems

ISSN :

1049-5258

Publisher :

Curran Associates, United States

Volume :

Peer review/Selection committee :

Peer Reviewed verified by ORBi

Tags :

CÉCI : Consortium des Équipements de Calcul Intensif
Tier-1 supercomputer

Additional URL :

https://openreview.net/forum?id=7v88Fh6iSM
https://github.com/francois-rozet/diem

Funders :

F.R.S.-FNRS - Fonds de la Recherche Scientifique

Available on ORBi :

since 20 June 2024

Statistics

Number of views

80 (22 by ULiège)

Number of downloads

46 (6 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

Bibliography

S. J. Warren and S. Dye. “Semilinear Gravitational Lens Inversion”. In The Astrophysical Journal (2003).
Warren R. Morningstar et al. “Data-driven Reconstruction of Gravitationally Lensed Galaxies Using Recurrent Inference Machines”. In The Astrophysical Journal (2019).
Siddharth Mishra-Sharma and Ge Yang. “Strong Lensing Source Reconstruction Using Continuous Neural Fields”. 2022.
Shanshan Wang et al. “Accelerating magnetic resonance imaging via deep learning”. In International Symposium on Biomedical Imaging. 2016.
Kerstin Hammernik et al. “Learning a variational network for reconstruction of accelerated MRI data”. In Magnetic Resonance in Medicine (2018).
Yoseo Han et al. “k-Space Deep Learning for Accelerated MRI”. In Transactions on Medical Imaging (2020).
Jure Zbontar et al. “fastMRI: An Open Dataset and Benchmarks for Accelerated MRI”. 2018.
Florian Knoll et al. “fastMRI: A Publicly Available Raw k-Space and DICOM Dataset of Knee Images for Accelerated MR Image Reconstruction Using Machine Learning”. In Radiology: Artificial Intelligence (2020).
G. Cowan. “A survey of unfolding methods for particle physics”. In Conf. Proc. C (2002).
Volker Blobel. “Unfolding Methods in Particle Physics”. In PHYSTAT. CERN, 2011.
François-Xavier Le Dimet and Olivier Talagrand. “Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects”. In Tellus A: Dynamic Meteorology and Oceanography (1986).
Yannick Trémolet. “Accounting for an imperfect model in 4D-Var”. In Quarterly Journal of the Royal Meteorological Society (2006).
Thomas M. Hamill. “Ensemble-based atmospheric data assimilation”. In Predictability of Weather and Climate. 2006.
Alberto Carrassi et al. “Data assimilation in the geosciences: An overview of methods, issues, and perspectives”. In WIREs Climate Change (2018).
Jascha Sohl-Dickstein et al. “Deep Unsupervised Learning using Nonequilibrium Thermodynamics”. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
Jonathan Ho et al. “Denoising Diffusion Probabilistic Models”. In Advances in Neural Information Processing Systems. 2020.
Yang Song et al. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models”. In International Conference on Learning Representations. 2022.
Bahjat Kawar et al. “SNIPS: Solving Noisy Inverse Problems Stochastically”. In Advances in Neural Information Processing Systems. 2021.
Bahjat Kawar et al. “Denoising Diffusion Restoration Models”. In Advances in Neural Information Processing Systems. 2022.
Alexandre Adam et al. “Posterior samples of source galaxies in strong gravitational lenses with score-based priors”. 2022.
Hyungjin Chung et al. “Diffusion Posterior Sampling for General Noisy Inverse Problems”. In International Conference on Learning Representations. 2023.
Jiaming Song et al. “Pseudoinverse-Guided Diffusion Models for Inverse Problems”. In International Conference on Learning Representations. 2023.
François Rozet and Gilles Louppe. “Score-based Data Assimilation”. In Advances in Neural Information Processing Systems. 2023.
Marc Anton Finzi et al. “User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems”. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Benjamin Boys et al. “Tweedie Moment Projected Diffusions For Inverse Problems”. 2023.
Y. Zhu et al. “Denoising Diffusion Models for Plug-and-Play Image Restoration”. In Conference on Computer Vision and Pattern Recognition Workshops. 2023.
Noe Dia et al. “Bayesian Imaging for Radio Interferometry with Score-based Priors”. 2023.
Herbert E. Robbins. “An Empirical Bayes Approach to Statistics”. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability. 1956.
George Casella. “An Introduction to Empirical Bayes Data Analysis”. In The American Statistician (1985).
Bradley P. Carlin and Thomas A. Louis. “Empirical Bayes: Past, Present and Future”. In Journal of the American Statistical Association (2000).
Bradley Efron. “Two Modeling Strategies for Empirical Bayes Estimation”. In Statistical Science (2014).
G. D'Agostini. “A multidimensional unfolding method based on Bayes' theorem”. In Nuclear Instruments and Methods in Physics Research (1995).
Tim Dockhorn et al. “Density Deconvolution with Normalizing Flows”. 2020.
Anders Andreassen et al. “OmniFold: A Method to Simultaneously Unfold All Observables”. In Physical Review Letters (2020).
Gilles Louppe et al. “Adversarial Variational Optimization of Non-Differentiable Simulators”. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics. 2019.
Maxime Vandegar et al. “Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference”. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics. 2021.
Julius Vetter et al. “Sourcerer: Sample-based Maximum Entropy Source Distribution Estimation”. 2024.
Bradley Efron. “Empirical Bayes deconvolution estimates”. In Biometrika (2016).
Balasubramanian Narasimhan and Bradley Efron. “deconvolveR: A G-Modeling Program for Deconvolution and Empirical Bayes Estimation”. In Journal of Statistical Software (2020).
H. O. Hartley. “Maximum Likelihood Estimation from Incomplete Data”. In Biometrics (1958).
A. P. Dempster et al. “Maximum Likelihood from Incomplete Data Via the EM Algorithm”. In Journal of the Royal Statistical Society (1977).
C. F. Jeff Wu. “On the Convergence Properties of the EM Algorithm”. In The Annals of Statistics (1983).
Geoffrey J McLachlan and Thriyambakam Krishnan. “The EM algorithm and extensions”. John Wiley & Sons, 2007.
Sivaraman Balakrishnan et al. “Statistical guarantees for the EM algorithm: From population to sample-based analysis”. In The Annals of Statistics (2017).
Yang Song and Stefano Ermon. “Generative Modeling by Estimating Gradients of the Data Distribution”. In Advances in Neural Information Processing Systems. 2019.
Yang Song et al. “Score-Based Generative Modeling through Stochastic Differential Equations”. In International Conference on Learning Representations. 2021.
Brian D. O. Anderson. “Reverse-time diffusion equation models”. In Stochastic Processes and their Applications (1982).
Simo Särkkä and Arno Solin. “Applied Stochastic Differential Equations”. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2019.
Qinsheng Zhang and Yongxin Chen. “Fast Sampling of Diffusion Models with Exponential Integrator”. In International Conference on Learning Representations. 2023.
Jiaming Song et al. “Denoising Diffusion Implicit Models”. In International Conference on Learning Representations. 2021.
Tero Karras et al. “Elucidating the Design Space of Diffusion-Based Generative Models”. In Advances in Neural Information Processing Systems. 2022.
Yaron Lipman et al. “Flow Matching for Generative Modeling”. In International Conference on Learning Representations. 2023.
Aapo Hyvärinen. “Estimation of Non-Normalized Statistical Models by Score Matching”. In Journal of Machine Learning Research (2005).
Pascal Vincent. “A Connection Between Score Matching and Denoising Autoencoders”. In Neural Computation (2011).
M. C. K. Tweedie. “Functions of a statistical variate with given means, with special reference to Laplacian distributions”. In Mathematical Proceedings of the Cambridge Philosophical Society (1947).
Bradley Efron. “Tweedie's Formula and Selection Bias”. In Journal of the American Statistical Association (2011).
Kwanyoung Kim and Jong Chul Ye. “Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images”. In Advances in Neural Information Processing Systems. 2021.
Chenlin Meng et al. “Estimating High Order Gradients of the Data Distribution by Denoising”. In Advances in Neural Information Processing Systems. 2021.
Greg C. G. Wei and Martin A. Tanner. “A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms”. In Journal of the American Statistical Association (1990).
Gilles Celeux and Jean Diebolt. “A stochastic approximation type EM algorithm for the mixture problem”. In Stochastics and Stochastic Reports (1992).
Bernard Delyon et al. “Convergence of a stochastic approximation version of the EM algorithm”. In The Annals of Statistics (1999).
James G. Booth and James P. Hobert. “Maximizing Generalized Linear Mixed Model Likelihoods with an Automated Monte Carlo EM Algorithm”. In Journal of the Royal Statistical Society (1999).
Richard A. Levine and George Casella. “Implementations of the Monte Carlo EM Algorithm”. In Journal of Computational and Graphical Statistics (2001).
Brian S. Caffo et al. “Ascent-Based Monte Carlo Expectation-Maximization”. In Journal of the Royal Statistical Society (2005).
Wolfgang Jank. “The EM Algorithm, Its Randomized Implementation and Global Optimization”. In Perspectives in Operations Research. 2006.
William Ruth. “A review of Monte Carlo-based versions of the EM algorithm”. 2024.
Michael E. Tipping and Christopher M. Bishop. “Mixtures of Probabilistic Principal Component Analyzers”. In Neural Computation (1999).
Christopher M. Bishop. “Pattern Recognition and Machine Learning”. Information Science and Statistics. Springer, 2006.
Lénaïc Chizat et al. “Faster Wasserstein Distance Estimation with the Sinkhorn Divergence”. In Advances in Neural Information Processing Systems. 2020.
Rémi Flamary et al. “POT: Python Optimal Transport”. In Journal of Machine Learning Research (2021).
Friedemann Zenke and Tim P. Vogels. “The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks”. In Neural Computation (2021).
Magnus R. Hestenes and Eduard Stiefel. “Methods of Conjugate Gradients for Solving Linear Systems”. In Journal of Research of the National Bureau of Standards (1952).
James Bradbury et al. “JAX: Composable transformations of Python + NumPy programs”. 2018.
Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. In Advances in Neural Information Processing Systems. 2019.
Youcef Saad and Martin Schultz. “GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems”. In Journal on Scientific and Statistical Computing (1986).
H. A. Van der Vorst. “Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems”. In Journal on Scientific and Statistical Computing (1992).
Luhuan Wu et al. “Practical and Asymptotically Exact Conditional Sampling in Diffusion Models”. In Advances in Neural Information Processing Systems. 2023.
Nicolas Bonneel et al. “Sliced and Radon Wasserstein Barycenters of Measures”. In Journal of Mathematical Imaging and Vision (2015).
Kimia Nadjahi et al. “Statistical and Topological Properties of Sliced Probability Divergences”. In Advances in Neural Information Processing Systems. 2020.
Giannis Daras et al. “Ambient Diffusion: Learning Clean Distributions from Corrupted Data”. In Advances in Neural Information Processing Systems. 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. “Learning Multiple Layers of Features from Tiny Images”. 2009.
Olaf Ronneberger et al. “U-Net: Convolutional Networks for Biomedical Image Segmentation”. In Medical Image Computing and Computer-Assisted Intervention. 2015.
Tim Salimans et al. “Improved Techniques for Training GANs”. In Advances in Neural Information Processing Systems. 2016.
Martin Heusel et al. “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”. In Advances in Neural Information Processing Systems. 2017.
Bahjat Kawar et al. “GSURE-Based Diffusion Model Training with Corrupted Data”. In Transactions on Machine Learning Research (2024).
Ajil Jalal et al. “Robust Compressed Sensing MRI with Deep Generative Priors”. In Advances in Neural Information Processing Systems. 2021.
Yochai Blau and Tomer Michaeli. “The Perception-Distortion Tradeoff”. In Conference on Computer Vision and Pattern Recognition. 2018.
Mauricio Delbracio and Peyman Milanfar. “Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration”. In Transactions on Machine Learning Research (2023).
E. G. Tabak and Cristina V. Turner. “A family of nonparametric density estimation algorithms”. In Communications on Pure and Applied Mathematics (2013).
Danilo Rezende and Shakir Mohamed. “Variational Inference with Normalizing Flows”. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
Asad Aali et al. “Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data”. 2024.
Giannis Daras et al. “Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data”. 2024.
Xinyu Peng et al. “Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance”. 2024.
Jiaming Song et al. “Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation”. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Arpit Bansal et al. “Universal Guidance for Diffusion Models”. In International Conference on Learning Representations. 2024.
Yutong He et al. “Manifold Preserving Guided Diffusion”. In International Conference on Learning Representations. 2024.
Prafulla Dhariwal and Alexander Quinn Nichol. “Diffusion Models Beat GANs on Image Synthesis”. In Advances in Neural Information Processing Systems. 2021.
Jonathan Ho and Tim Salimans. “Classifier-Free Diffusion Guidance”. 2022.
Tero Karras et al. “A Style-Based Generator Architecture for Generative Adversarial Networks”. In Conference on Computer Vision and Pattern Recognition. 2019.
Ashish Vaswani et al. “Attention is All you Need”. In Advances in Neural Information Processing Systems. 2017.
Stefan Elfwing et al. “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning”. In Neural Networks (2018).
Jimmy Lei Ba et al. “Layer Normalization”. 2016.
Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In International Conference on Learning Representations. 2015.
Kaiming He et al. “Deep Residual Learning for Image Recognition”. In Conference on Computer Vision and Pattern Recognition. 2016.
William Peebles and Saining Xie. “Scalable Diffusion Models with Transformers”. In International Conference on Computer Vision. 2023.
Anton Obukhov et al. “High-fidelity performance metrics for generative models in PyTorch”. 2020.
Wenzhe Shi et al. “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network”. In Conference on Computer Vision and Pattern Recognition. 2016.
Richard Zhang et al. “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”. In Conference on Computer Vision and Pattern Recognition. 2018.
Zhou Wang et al. “Image quality assessment: from error visibility to structural similarity”. In Transactions on Image Processing (2004).