Inductive Bias In Deep Probabilistic Modelling

One of the most notable distinctions between humans and most other animals is our ability to grow collective intelligence along generations. This development appears exponential; we are witnessing an explosion of knowledge and technical capabilities. Since the invention of computers, artificial intelligence (AI) has enabled machines to push further the boundaries of our collective intelligence.
The rapid progress of information technology and the profusion of data has made machine learning (ML), a sub-field of AI, crucial for the collective intelligence growth.

The probabilistic modelling framework unifies the ML-based and human-based knowledge discovery processes, i.e. the creation of mathematical descriptions of real-world phenomena. Thus it is at the root of this disruption of innovation. In this context, this thesis collects five scientific papers that have contributed to developing modern approaches to probabilistic modelling between 2018 to 2022.

This thesis provides a thorough introduction to modern probabilistic modelling. We discuss the why and the how of probabilistic modelling, and we introduce two important classes of models: probabilistic graphical models and deep probabilistic models. Then, we contrast our work into contributions to uninformed and informed models. The former models are prefered when data contains enough information to retrieve the targetted model instance. In contrast, informed models embed stronger prior knowledge of the phenomenon of interest. Data is only there to complement this knowledge. The quality of informed model instances depends on the data and validity of the prior knowledge.

The second part of the thesis focuses on three distinct contributions to uninformed probabilistic models. First, we are interested in bringing together distinct model classes; the combination of diffusion models and variational auto-encoders unlocks new modelling features. Second, we draw explicit connections between Bayesian networks and normalizing flows. We exploit this connection to study some representational aspects of normalizing flows. Finally, we present a new neural network architecture that enforces a monotonic response. We demonstrate the effectiveness of this representation in modelling continuous probability distributions.

In the third part of the manuscript, we consider informed probabilistic models. In the fourth contribution, we introduce the graphical normalizing flows, a new normalizing flow architecture that embeds independence assumptions. Finally, our last contribution shows that informing deep probabilistic models with a partial physical understanding of the studied phenomenon unlocks generalisation capabilities inaccessible to non-informed models. We conclude this work with a summary and a brief prospect of future developments in deep probabilistic modelling.