Rethinking Maximal Update Parametrization: Steepest Descent on the Spectral Ball | Franz Louis Cesista
Novel optimizers for maximally updating both the weights and activations of neural networks while keeping weight norms under control. To get there, we needed to invent an efficient, GPU/TPU-friendly method for eigenvalue clipping and solve the Steepest Descent problem on the Positive Semidefinite Cone, Convex Spectrahedron, and finally on the Spectral Ball.| leloykun.github.io
