"Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models" The sudden loss spikes during LLM training might be more predictable than we thought. This paper uses simple quadratic models to show how heavy-tailed data and noisy gradients can trigger these spikes, without requiring complex neural network behavior. On a 1.3B Transformer, it identifies spike-prone training configurations using simulations that take just 5 seconds on a CPU per…