Theoretical perspectives on modern machine learning paradigms: generative, scientific and out-of-distribution

Over the past decade, machine learning models have grown in scale and complexity. Generative models, for instance, have gone through many iterations of model classes (e.g., GANs, diffusion models, autoregressive models), architectures (e.g., Transformers), as well as an increasingly involved training and inference-time pipeline. Machine-learned systems have also started to permeate new domains beyond images and language, such as scientific computing and discovery. Finally, trained models have started to be widely deployed - often in situations that are very different from what they were trained on. These developments pose both challenges and opportunities for developing theoretical foundations. In particular, devising the right mathematical abstractions can reveal new algorithmic possibilities, diagnose failure modes, and guide engineering efforts.

In this talk, I will showcase a few vignettes illustrating theoretical perspectives on generative models, machine learning for scientific computing, and out-of-distribution (OOD) generalization. I will spend most of the talk on a framework to understand the design space of score-based losses (a common family of losses used to train energy-based models, as well as discrete and continuous diffusion models), and a surprising connection between designing computationally efficient inference-time algorithms and statistically well-behaved losses. Additionally, I will discuss theoretical approaches to neural network architecture design for partial differential equation (PDE) solvers, and introduce mathematical sandboxes to diagnose failure modes of recent invariance-based approaches to OOD generalization. Throughout, I will highlight emerging directions in these areas for which a theoretical lens can be particularly useful.

Speaker: Andrej Risteski is an Assistant Professor in the Machine Learning Department at Carnegie Mellon University.