This is great review of the highlights and breakthroughs in AI over the last decade. Two things I found especially interesting:
- predicting the next word of text requires the model include some representation of the world
- models are effectively compressing the data they are trained on, ie primarily remembering the interesting and surprising parts of the training data
It's remarkable that world models are somehow a side effect, an unintentional artifact, of learning to predict the next word. How much of being a human is just a side effect of needing to do something relatively mundane?
Ilya is mostly famous right now for being Altman's rival and is being labelled a deaccel but he's one of the primary figures behind recent advancements in AI.
Also Jensen looks like a dad that loves you but is never satisfied with your accomplishments.