Human values are a kind of misgeneralization. If humans learn our motivations through neural reinforcement, pleasure and pain, in theory we might expect all humans to care about nothing else. But we do. We learn other values, because they are pretty good proxies for reward, and in practice it's hard to learn how to effectively pursue reward directly. We misgeneralize. More generally, in the limit, any system trained on reward will learn to pursue reward directly - including through reward…