"Self-Play Pretraining with Zero Data" Most pretraining still depends on humans deciding what data a model should learn from, so can a model learn entirely through self-play and no real data? To test that, they trains two models from scratch. One writes small programs that generate byte sequences, while the other learns to predict those sequences with standard next-token training. The generator is then updated with RL to pick the sequences that line up with the learner’s recent progress. So…