This is great review of the highlights and breakthroughs in AI over the last decade. Two things I found especially interesting:
  1. predicting the next word of text requires the model include some representation of the world
  2. models are effectively compressing the data they are trained on, ie primarily remembering the interesting and surprising parts of the training data
It's remarkable that world models are somehow a side effect, an unintentional artifact, of learning to predict the next word. How much of being a human is just a side effect of needing to do something relatively mundane?
Ilya is mostly famous right now for being Altman's rival and is being labelled a deaccel but he's one of the primary figures behind recent advancements in AI.
Also Jensen looks like a dad that loves you but is never satisfied with your accomplishments.
reply