Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation

Rozet, François; Ohana, Ruben; McCabe, Michael; Louppe, Gilles; Lanusse, François; Ho, Shirley

Download

Paper published in a journal (Scientific congresses and symposiums)

Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation

Rozet, François; Ohana, Ruben; McCabe, Michael et al.

2025 • In Advances in Neural Information Processing Systems, 38

Peer Reviewed verified by ORBi Dataset

Permalink
https://hdl.handle.net/2268/337977

arXiV
2507.02608v4

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

lola.pdf

Author postprint (10 MB)

Creative Commons License - Public Domain Dedication

Download

All documents in ORBi are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Computer Science - Learning; Physics - Fluid Dynamics

Abstract :

[en] The steep computational cost of diffusion models at inference hinders their use as fast physics emulators. In the context of image and video generation, this computational drawback has been addressed by generating in the latent space of an autoencoder instead of the pixel space. In this work, we investigate whether a similar strategy can be effectively applied to the emulation of dynamical systems and at what cost. We find that the accuracy of latent-space emulation is surprisingly robust to a wide range of compression rates (up to 1000x). We also show that diffusion-based emulators are consistently more accurate than non-generative counterparts and compensate for uncertainty in their predictions with greater diversity. Finally, we cover practical design choices, spanning from architectures to optimizers, that we found critical to train latent-space emulators.

Disciplines :

Computer science

Author, co-author :

Rozet, François ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Big Data

Ohana, Ruben

McCabe, Michael

Louppe, Gilles ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Big Data

Lanusse, François

Ho, Shirley

Language :

English

Title :

Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation

Publication date :

03 July 2025

Event name :

The Thirty-Ninth Annual Conference on Neural Information Processing Systems

Event place :

San Diego, United States - California

Event date :

December 2-7, 2025

Audience :

International

Journal title :

Advances in Neural Information Processing Systems

ISSN :

1049-5258

Publisher :

Curran Associates, United States

Volume :

Peer review/Selection committee :

Peer Reviewed verified by ORBi

Additional URL :

https://openreview.net/forum?id=xoNrbfbekM

Funders :

F.R.S.-FNRS - Fonds de la Recherche Scientifique
Simons Foundation

Funding text :

We thank Géraud Krawezik and the Scientific Computing Core at the Flatiron Institute, a division of the Simons Foundation, for the compute facilities and support. We gratefully acknowledge use of the research computing resources of the Empire AI Consortium, Inc., with support from the State of New York, the Simons Foundation, and the Secunda Family Foundation. Polymathic AI acknowledges funding from the Simons Foundation and Schmidt Sciences, LLC. François Rozet is a research fellow of the F.R.S.-FNRS (Belgium) and acknowledges its financial support

Data Set :

The Well

Available on ORBi :

since 27 November 2025

Statistics

Number of views

184 (3 by ULiège)

Number of downloads

200 (4 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

Bibliography

ECMWF. “IFS documentation CY49R1 - part III: Dynamics and numerical procedures”. In IFS Documentation CY49R1. ECMWF, 2024.
Jongil Han and Hua-Lu Pan. “Revision of Convection and Vertical Diffusion Schemes in the NCEP Global Forecast System”. In Weather and Forecasting 26.4 (2011).
A. J. Hundhausen and R. A. Gentry. “Numerical simulation of flare-generated disturbances in the solar wind”. In Journal of Geophysical Research (1896-1977) 74.11 (1969).
John T. Mariska et al. “Numerical Simulations of Impulsively Heated Solar Flares”. In The Astrophysical Journal 341 (1989).
Chi Wang et al. “Magnetohydrodynamics (MHD) numerical simulations on the interaction of the solar wind with the magnetosphere: A review”. In Science China Earth Sciences 56.7 (2013).
Yuri N. Dnestrovskii and Dimitri P. Kostomarov. “Numerical Simulation of Plasmas”. Berlin, Heidelberg: Springer, 1986.
Yildirim Suzen et al. “Numerical Simulations of Plasma Based Flow Control Applications”. In 35th AIAA Fluid Dynamics Conference and Exhibit. Fluid Dynamics and Co-located Conferences. American Institute of Aeronautics and Astronautics, 2005.
Jonathan Tompson et al. “Accelerating Eulerian Fluid Simulation With Convolutional Networks”. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 2017.
Alvaro Sanchez-Gonzalez et al. “Learning to Simulate Complex Physics with Graph Networks”. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Tobias Pfaff et al. “Learning Mesh-Based Simulation with Graph Networks”. In International Conference on Learning Representations. 2021.
Johannes Brandstetter et al. “Message Passing Neural PDE Solvers”. In International Conference on Learning Representations. 2022.
Kim Stachenfeld et al. “Learned Simulators for Turbulence”. In International Conference on Learning Representations. 2022.
Nikola Kovachki et al. “Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs”. In Journal of Machine Learning Research 24.89 (2023).
Remi Lam et al. “Learning skillful medium-range global weather forecasting”. In Science 382.6677 (2023).
Michael McCabe et al. “Multiple Physics Pretraining for Spatiotemporal Surrogate Models”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Maximilian Herde et al. “Poseidon: Efficient Foundation Models for PDEs”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Rudy Morel et al. “DISCO: learning to DISCover an evolution Operator for multi-physicsagnostic prediction”. 2025.
Phillip Lippe et al. “PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers”. In Advances in Neural Information Processing Systems. Vol. 36. 2023.
Georg Kohl et al. “Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulation”. In ICML 2024 AI for Science Workshop. 2024.
Björn List et al. “Learned turbulence modelling with differentiable fluid solvers: physics-based loss functions and optimisation horizons”. In Journal of Fluid Mechanics 949 (2022).
Michael McCabe et al. “Towards Stability of Autoregressive Neural Operators”. In Transactions on Machine Learning Research (2023).
Salva Cachay et al. “DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting”. In Advances in Neural Information Processing Systems. Vol. 36. 2023.
Aliaksandra Shysheya et al. “On conditional diffusion models for PDE simulations”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Jiahe Huang et al. “DiffusionPDE: Generative PDE-Solving under Partial Observation”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Ilan Price et al. “Probabilistic weather forecasting with machine learning”. In Nature 637.8044 (2025).
Robin Rombach et al. “High-Resolution Image Synthesis With Latent Diffusion Models”. In Conference on Computer Vision and Pattern Recognition. 2022.
William Peebles and Saining Xie. “Scalable Diffusion Models with Transformers”. In International Conference on Computer Vision. 2023.
Patrick Esser et al. “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis”. 2024.
Tero Karras et al. “Analyzing and Improving the Training Dynamics of Diffusion Models”. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Enze Xie et al. “SANA: Efficient High-Resolution Text-to-Image Synthesis with Linear Diffusion Transformers”. In International Conference on Learning Representations. 2025.
Junyu Chen et al. “Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models”. In International Conference on Learning Representations. 2025.
Adam Polyak et al. “Movie Gen: A Cast of Media Foundation Models”. 2024.
Zhihan Gao et al. “PreDiff: Precipitation Nowcasting with Latent Diffusion Models”. In Thirty-seventh Conference on Neural Information Processing Systems. 2023.
Pan Du et al. “Conditional neural field latent diffusion model for generating spatiotemporal turbulence”. In Nature Communications 15.1 (2024).
Anthony Zhou et al. “Text2PDE: Latent Diffusion Models for Accessible Physics Simulation”. In International Conference on Learning Representations. 2025.
Zijie Li et al. “Generative Latent Neural PDE Solver using Flow Matching”. 2025.
Gérôme Andry et al. “Appa: Bending Weather Dynamics with Latent Diffusion Models for Global Data Assimilation”. 2025.
Ruben Ohana et al. “The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Jascha Sohl-Dickstein et al. “Deep Unsupervised Learning using Nonequilibrium Thermodynamics”. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
Jonathan Ho et al. “Denoising Diffusion Probabilistic Models”. In Advances in Neural Information Processing Systems. 2020.
Yang Song and Stefano Ermon. “Generative Modeling by Estimating Gradients of the Data Distribution”. In Advances in Neural Information Processing Systems. 2019.
Yang Song et al. “Score-Based Generative Modeling through Stochastic Differential Equations”. In International Conference on Learning Representations. 2021.
Brian D. O. Anderson. “Reverse-time diffusion equation models”. In Stochastic Processes and their Applications 12.3 (1982).
Simo Särkkä and Arno Solin. “Applied Stochastic Differential Equations”. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2019.
Shanchuan Lin et al. “Common Diffusion Noise Schedules and Sample Steps are Flawed”. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2024.
Xingchao Liu et al. “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow”. In International Conference on Learning Representations. 2023.
Yaron Lipman et al. “Flow Matching for Generative Modeling”. In 2023.
Aapo Hyvärinen. “Estimation of Non-Normalized Statistical Models by Score Matching”. In Journal of Machine Learning Research (2005).
Pascal Vincent. “A Connection Between Score Matching and Denoising Autoencoders”. In Neural Computation (2011).
Jiaming Song et al. “Denoising Diffusion Implicit Models”. In International Conference on Learning Representations. 2021.
Tero Karras et al. “Elucidating the Design Space of Diffusion-Based Generative Models”. In Advances in Neural Information Processing Systems. 2022.
M. C. K. Tweedie. “Functions of a statistical variate with given means, with special reference to Laplacian distributions”. In Mathematical Proceedings of the Cambridge Philosophical Society (1947).
Bradley Efron. “Tweedie's Formula and Selection Bias”. In Journal of the American Statistical Association (2011).
Kwanyoung Kim and Jong Chul Ye. “Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images”. In Advances in Neural Information Processing Systems. 2021.
Chenlin Meng et al. “Estimating High Order Gradients of the Data Distribution by Denoising”. In Advances in Neural Information Processing Systems. 2021.
Sander Dieleman. “Generative modelling in latent space”. 2025.
Richard Zhang et al. “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”. In Conference on Computer Vision and Pattern Recognition. 2018.
Ian J. Goodfellow et al. “Generative Adversarial Networks”. 2014.
Patrick Esser et al. “Taming Transformers for High-Resolution Image Synthesis”. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
Xi-Lin Li. “Preconditioned Stochastic Gradient Descent”. In IEEE Transactions on Neural Networks and Learning Systems 29.5 (2018).
Vineet Gupta et al. “Shampoo: Preconditioned Stochastic Tensor Optimization”. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018.
Nikhil Vyas et al. “SOAP: Improving and Stabilizing Shampoo using Adam for Language Modeling”. In International Conference on Learning Representations. 2025.
Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In International Conference on Learning Representations. 2015.
Lucas Nestler and François Rozet. “HeavyBall: Efficient optimizers”. 2022.
Vikram Voleti et al. “MCVD - Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation”. In Advances in Neural Information Processing Systems. Vol. 35. 2022.
Alex Henry et al. “Query-Key Normalization for Transformers”. In Findings of the Association for Computational Linguistics. Ed. by Trevor Cohn et al. Online: Association for Computational Linguistics, 2020.
Jianlin Su et al. “RoFormer: Enhanced transformer with Rotary Position Embedding”. In Neurocomputing 568 (2024).
Byeongho Heo et al. “Rotary Position Embedding for Vision Transformer”. In European Conference on Computer Vision. Ed. by Aleš Leonardis et al. Cham: Springer Nature Switzerland, 2025.
Zhanchao Zhou et al. “Value Residual Learning”. 2024.
Zilong Huang et al. “CCNet: Criss-Cross Attention for Semantic Segmentation”. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
Jonathan Ho et al. “Axial Attention in Multidimensional Transformers”. 2019.
Ali Hassani et al. “Neighborhood Attention Transformer”. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Qinsheng Zhang and Yongxin Chen. “Fast Sampling of Diffusion Models with Exponential Integrator”. In International Conference on Learning Representations. 2023.
Alexey Dosovitskiy et al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”. In International Conference on Learning Representations. 2021.
V. Fortin et al. “Why Should Ensemble Spread Match the RMSE of the Ensemble Mean?” In Journal of Hydrometeorology 15.4 (2014).
Hugh L. Dryden. “A Review of the Statistical Theory of Turbulence”. In Quarterly of Applied Mathematics 1.1 (1943).
Kyle T. Mandli et al. “Clawpack: building an open source ecosystem for solving hyperbolic PDEs”. In PeerJ Computer Science 2 (2016).
François Rozet et al. “Learning Diffusion Priors from Observations by Expectation Maximization”. In Advances in Neural Information Processing Systems. Vol. 37. 2024.
Jonathan Ho et al. “Video Diffusion Models”. In ICLR Workshop on Deep Generative Models for Highly Structured Data. 2022.
Hyungjin Chung et al. “Diffusion Posterior Sampling for General Noisy Inverse Problems”. In International Conference on Learning Representations. 2023.
François Rozet and Gilles Louppe. “Score-based Data Assimilation”. In Advances in Neural Information Processing Systems. Vol. 36. 2023.
Zongyi Li et al. “Fourier Neural Operator for Parametric Partial Differential Equations”. In International Conference on Learning Representations. 2021.
Bogdan Raonic et al. “Convolutional Neural Operators for robust and accurate learning of PDEs”. In Advances in Neural Information Processing Systems. Vol. 36. 2023.
Zhongkai Hao et al. “GNOT: A General Neural Operator Transformer for Operator Learning”. In Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023.
Peter Benner et al. “A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems”. In SIAM Review 57.4 (2015).
Bethany Lusch et al. “Deep learning for universal linear embeddings of nonlinear dynamics”. In Nature Communications 9.1 (2018).
Hugo F. S. Lui and William R. Wolf. “Construction of reduced-order models for fluid flows using deep feedforward neural networks”. In Journal of Fluid Mechanics 872 (2019).
S. Wiewel et al. “Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow”. In Computer Graphics Forum 38.2 (2019).
Romit Maulik et al. “Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders”. In Physics of Fluids 33.3 (2021).
Xu Han et al. “Predicting Physics in Mesh-reduced Space with Temporal Attention”. In International Conference on Learning Representations. 2022.
Nicholas Geneva and Nicholas Zabaras. “Transformers for modeling physical systems”. In Neural Networks 146 (2022).
Peter Yichen Chen et al. “CROM: Continuous Reduced-Order Modeling of PDEs Using Implicit Neural Representations”. In International Conference on Learning Representations. 2023.
AmirPouya Hemmasian and Amir Barati Farimani. “Reduced-order modeling of fluid flows with transformers”. In Physics of Fluids 35.5 (2023).
Zijie Li et al. “Latent neural PDE solver: A reduced-order modeling framework for partial differential equations”. In Journal of Computational Physics 524 (2025).
B. O. Koopman. “Hamiltonian Systems and Transformation in Hilbert Space”. In Proceedings of the National Academy of Sciences 17.5 (1931).
Enoch Yeung et al. “Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems”. In American Control Conference (ACC). 2019.
Xiaoyu Xie et al. “Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization”. 2024.
Francesco Regazzoni et al. “Learning the intrinsic dynamics of spatio-temporal processes through Latent Dynamics Networks”. In Nature Communications 15.1 (2024).
Adrien Bardes et al. “Revisiting Feature Prediction for Learning Visual Representations from Video”. In Transactions on Machine Learning Research (2024).
Nicholas Geneva and Nicholas Zabaras. “Modeling the dynamics of PDE systems with physics-constrained deep auto-regressive networks”. In Journal of Computational Physics 403 (2020).
Daniel E. Worrall et al. “Spectral Shaping for Neural PDE Surrogates”. 2024.
Jingfeng Yao et al. “Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models”. 2025.
Yilun Xu et al. “A Theory of Usable Information under Computational Constraints”. In International Conference on Learning Representations. 2019.
Hao Chen et al. “Masked Autoencoders Are Effective Tokenizers for Diffusion Models”. 2025.
Lijun Yu et al. “MAGVIT: Masked Generative Video Transformer”. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Kaiming He et al. “Masked Autoencoders Are Scalable Vision Learners”. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022.
Alekh Karkada Ashok and Nagaraju Palani. “Autoencoders with Variable Sized Latent Vector for Image Compression”. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018.
Chi-Hieu Pham et al. “PCA-AE: Principal Component Analysis Autoencoder for Organising the Latent Space of Generative Networks”. In Journal of Mathematical Imaging and Vision 64.5 (2022).
Roman Bachmann et al. “FlexTok: Resampling Images into 1D Token Sequences of Flexible Length”. In Forty-second International Conference on Machine Learning. 2025.
Keaton J. Burns et al. “Dedalus: A flexible framework for numerical simulations with spectral methods”. In Physical Review Research 2.2 (2020).
Keiya Hirashima et al. “3D-Spatiotemporal forecasting the expansion of supernova shells using deep learning towards high-resolution galaxy simulations”. In Monthly Notices of the Royal Astronomical Society 526.3 (2023).
Kaiming He et al. “Deep Residual Learning for Image Recognition”. In Conference on Computer Vision and Pattern Recognition. 2016.
Stefan Elfwing et al. “Sigmoid-weighted linear units for neural network function approximation in reinforcement learning”. In Neural Networks. Special issue on deep reinforcement learning 107 (2018).
Jimmy Lei Ba et al. “Layer Normalization”. 2016.
Nicolas Bonneel et al. “Sliced and Radon Wasserstein Barycenters of Measures”. In Journal of Mathematical Imaging and Vision (2015).
Soheil Kolouri et al. “Generalized Sliced Wasserstein Distances”. In Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., 2019.
Tung Nguyen et al. “PhysiX: A Foundation Model for Physics Simulations”. 2025.
Zhikai Wu et al. “TANTE: Time-Adaptive Operator Learning via Neural Taylor Expansion”. 2025.
Payel Mukhopadhyay et al. “Controllable Patching for Compute-Adaptive Surrogate Modeling of Partial Differential Equations”. 2025.
Laurens van der Maaten and Geoffrey Hinton. “Visualizing Data using t-SNE”. In Journal of Machine Learning Research 9.86 (2008).
Gabriel Peyré and Marco Cuturi. “Computational Optimal Transport: With Applications to Data Science”. In Foundations and Trends in Machine Learning 11.5-6 (2019).