Press Space to continue
Language models plan ahead while writing one token at a time
Press Space to continue
My biggest takeaways from @AnthropicAI interpretability researcher @mlpowered: 1. Predicting the next token requires understanding the universe. Language models got better than humans at predicting the next token around 2019. Once a model masters the easy tokens (sentence structure, then basic facts), the only ones left are hard, like the final paragraph of a PhD thesis. To finish “The capital of the state that contains Dallas is . . .,” a model has to know that Dallas is in Texas and that…
open on x · 3 viewsViews and clicks count once per visitor per day. Clicks are opens of this post on X.