This is great review of the highlights and breakthroughs in AI over the last decade. Two things I found especially interesting:
- predicting the next word of text requires the model include some representation of the world
- models are effectively compressing the data they are trained on, ie primarily remembering the interesting and surprising parts of the training data
It's remarkable that world models are somehow a side effect, an unintentional artifact, of learning to predict the next word. How much of being a human is just a side effect of needing to do something relatively mundane?