Modular: CUDA is the incumbent, but is it any good? (Democratizing AI Compute, Part 4)
Answering the question of whether CUDA is “good” is much trickier than it sounds.| www.modular.com
In our earlier post, diffusion-fast, we showed how the Stable Diffusion XL (SDXL) pipeline can be optimized up to 3x using native PyTorch code. Back then, SDXL was an open SoTA pipeline for image generation. Quite unsurprisingly, a lot has changed since then, and it’s safe to say that Flux is now one of the most capable open-weight models in the space.| pytorch.org
Attention, as a core layer of the ubiquitous Transformer architecture, is a bottleneck for large language models and long-context applications. FlashAttention (and FlashAttention-2) pioneered an approach to speed up attention on GPUs by minimizing memory reads/writes, and is now used by most libraries to accelerate Transformer training and inference. This has contributed to a massive increase in LLM context length in the last two years, from 2-4K (GPT-3, OPT) to 128K (GPT-4), or even 1M (Ll...| pytorch.org
Answering the question of whether CUDA is “good” is much trickier than it sounds.| www.modular.com
If we as an ecosystem hope to make progress, we need to understand how the CUDA software empire became so dominant.| www.modular.com
