Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration | Franz Louis Cesista
A small step towards hardware-architecture-optimizer codesign in deep learning.| leloykun.github.io
Muon is an optimizer for the hidden layers in neural networks. It is used in the current training speed records for both NanoGPT and CIFAR-10 speedrunning. Many empirical results using Muon have already been posted, so this writeup will focus mainly on Muon’s design. First we will define Muon and provide an overview of the empirical results it has achieved so far. Then we will discuss its design in full detail, including connections to prior research and our best understanding of why it works.| kellerjordan.github.io
We propose a factorization-free method for orthogonal projection onto the positive semidefinite (PSD) cone, leveraging composite polynomial filtering. Inspired by recent advances in homomorphic encryption, our approach approximates the PSD cone projection operator using a carefully optimized composite polynomial evaluated exclusively via matrix-matrix multiplications. This approach enables efficient GPU implementations with low-precision arithmetic, significantly outperforming the classical P...| arXiv.org
A small step towards hardware-architecture-optimizer codesign in deep learning.| leloykun.github.io
