Muon quietly "starves" 25% of neurons: Aurora's repair boosts data efficiency by a hundredfold

According to Beating Monitoring, Tilde Research discovered that the optimizer Muon used in leading models like DeepSeek V4, Kimi K2.5, and GLM-5 has a hidden flaw: it causes more than a quarter of the neurons in the MLP layers to die permanently early in training. Based on this, the team designed an alternative optimizer called Aurora and open-sourced it. A 1.1B parameter model trained with only about 100B tokens was able to match the performance of Qwen3-1.7B, which was trained on 36T tokens, on language understanding benchmarks like HellaSwag and Winogrande.
The problem lies in a mathematical property of Muon when handling the MLP weight matrices. Early in training, some neurons happen to receive weaker gradient signals. Traditional optimizers like AdamW normalize parameters step-by-step, naturally smoothing out these differences; but Muon’s orthogonalization step passes the weak signals unchanged. Weak neurons continue to receive weak updates, becoming increasingly silent, creating a “rich get richer” dead cycle. By the 500th training step, over a quarter of the neurons have effectively died, wasting parameter capacity.
Previous improved version NorMuon mitigated this by forcing the update magnitudes of each row to be uniform, but at the cost of destroying the orthogonality of the update matrix (which makes each update as efficient as possible and is a core advantage of Muon), resulting in reduced optimization accuracy. Aurora sets “uniform updates” and “orthogonality” as joint constraints, iteratively satisfying both: ensuring each neuron has a fair learning opportunity without sacrificing update precision.
Unparameterized Aurora only adds about 6% more computational overhead compared to Muon and can be used as a direct replacement. In modded-nanoGPT benchmark runs, Aurora achieved the current best record in 3,175 steps. Aurora’s advantage will also grow with increasing MLP width; the higher the expansion factor, the more significant the improvement.
The code and the 1.1B pre-trained model have been open-sourced.