Some Building Blocks for Foundation Model Systems

I fell in love with foundation models because they radically improved data systems that I had been trying to build for a decade. Motivated by this experience, the bulk of the talk focuses on efficient building blocks for foundation models. The first line of work describes fundamental trends in hardware accelerators for AI that we can leverage, e.g., optimizing memory accesses in Flash Attention on GPUs. The second line of work describes new architectures that are asymptotically more efficient than transformers for long sequences. One family of these architectures is inspired by signal processing and classical architectures, like RNNs and CNNs, in a mathematically precise way. These new architectures have achieved state-of-the-art quality on long-sequence tasks, are promising general purpose architectures, and are being applied to new areas. Of course, as researchers, we want to understand their limits and how to improve them, which the talk will focus on. Two underlying themes in the talk are understanding the role of inductive bias in AI models and understanding how robust our recipe is to get amazing AI.

Speaker: Chris Re, Stanford University

Attend in person or online (See weblink for Zoom information)