Press Space to continue
Models can't recreate real programs from scratch yet
Press Space to continue
The creators of SWE-Bench just dropped a really simple new benchmark every LLM gets 0% on. ProgramBench asks: can models recreate real executable programs (ffmpeg, SQLite, ripgrep) from scratch with no internet? We are far from saturated on model quality.
open on x · 3 viewsViews and clicks count once per visitor per day. Clicks are opens of this post on X.