Press Space to continue
Chunked Cross-Entropy Reduces Model Memory Usage by 23x
Press Space to continue
SkyRL Tinker implementation is so cool, but still early days so I went ahead and implemented a basic chunked cross-entropy for 23x less memory usage! Now most models fit much more seamlessly in memory when combined with Megatron.
open on xViews and clicks count once per visitor per day. Clicks are opens of this post on X.