Linalg is All Your Need

Abstract

This is a whirlwind overview of tensor programming in machine learning compilers. We will discuss the ubiquity of linear algebra in the hardware acceleration domain and why tiling is such an important optimization for it. We then cover the various programming workflows that have emerged in the machine learning space and how tensor programs are optimized by compilers. Lastly, we will present our work on optimizing and tiling the attention mechanism in transformers, the key mechanism in large language models.