Intro
I'm going to slam some rocks together. It doesn't really culminate in anything, so if that bugs you, abort now!
Rock 1
I'm not one generally to talk about elites all the time, but being elite is real [0]. The social class is largely but not completely closed to new entrants, globally pervasive, and resilient over time [1]. Members of this class are rich [2] and they have a lot of influence over stuff, particularly governance structures and institutions of all flavors.
Rock 2
You've heard of ChatGPT and probably are aware that there's a host of other large language models of comparable intelligence. You may not be aware that the general recipe for creating one of these things is to feed it as much data as you can get your hands on. A problem is that at this point, the state-of-the-art LLMs need all the digital data that exists in the world, but that's not the main problem. The main problem is that each leap forward in intelligence generally requires an order of magnitude more data. Since the current SOTA LLMs already have consumed all the reasonable data, that puts us in a bit of a pickle [3].
A common move to get around this is, in addition to using all the data in existence, to also use data that's generated somehow, so you can gin up an infinite amount of it. And what's the best way to generate certain kinds of data? LLMs! Which means we have a pointing-two-mirrors-at-each-other situation, which intuitively seems ... disturbing, for reasons you might not be able to name.
Fortunately, you don't have to name it because Shumailov et al [4] have named it for you, and the name is the curse of recursion. I would summarize their findings this way:
-
the world is really complicated and full of weird nuances that are hard to characterize
-
LLMs are really good, like, they are excellent at boiling down that complexity and giving you very intelligent responses to whatever you ask them about
-
but they're not perfect!
-
so the data they create is, in various hard-to-describe ways, less complex and rich than data from the actual world
-
which means that, the more you train LLMs on LLM-trained data, the more they get weird ideas about the world due to the reduced complexity of the data that they're seeing
-
and if you use these LLM-trained LLMs to generate data to train new LLMs you get, after a few generations, really strange and inbred LLMs
which makes deep intuitive sense when you think about it.
Rock 3
There's a an idea, introduced by Dan Klein, called the half-life of policy (intro here) which basically says that any policy you can think of was created in a certain socio-technological context [5], and (if we're lucky) it made sense in that context, but the nature of technological change is such that it probably doesn't make sense any more.
The obvious policies that you would imagine are laws passed by political bodies of some sort, but the logic works everywhere -- rules get outdated when the underlying circumstances change, and nothing changes underlying circumstances as quickly or as profoundly as technology does [6].
Discussion
If you bang these rocks together, you get some interesting implications.
First, you have a class of people who, as time passes, become increasingly abstracted from the physical and social universe that the rest of us occupy. As a birthright, they're been plopped down next to the machinery that runs reality, but they have little conception of reality because they've never touched it, because they don't need to touch it. All they have to do to have wealth continue to rain down on them like gently falling snow [7] is not run a child sex ring, which, admittedly, is a bridge too far for some. But mostly, they're good, they get on the conveyor belt and they have a nice time.
But something weird happens when the world start changing faster and faster: the handbook for world domination gets increasingly outdated. It's like picking up an old book on software development from Goodwill for $1 - the process described to build a program doesn't work, you get all kinds of unparseable errors, the menu options don't correspond to what they show you in the figures. It's confusing and hard to take effective action.
Despite this, most of the same people continue to run the show, even though they suck at it. And the longer they do, and the more things change, the worse the effects are. And yet these effects give rise to the next wave -- the system consumes the fruits of its own mindless grinding, and those fruits are malformed, and eating them warps the people who do the eating, making them even less fit for the challenges to come.
The whole thing is a flywheel: the faster it turns, the quicker the machine lurches forward into some impossible Cthulu geography. It's not clown world so much as a Dali painting.
References
[0] Mills, C. W. (1981). The power elite [1956]. New York.
[1] Cousin, B., & Chauvin, S. (2021). Is there a global super‐bourgeoisie? Sociology Compass, 15(6), e12883. https://doi.org/10.1111/soc4.12883
[2] Piketty, T. (2014). Capital in the twenty-first century. Harvard University Press.
[3] Don't nitpick me -- how long do you want this post to be?
[4] Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget (arXiv:2305.17493). arXiv. http://arxiv.org/abs/2305.17493
[5] Perez, C. (2003). Technological revolutions and financial capital. Edward Elgar Publishing.
[6] Lyn Alden talks about a related aspect of this: the technological determinism at the heart of both money and fiat monetary dysfunction. We've discussed this a few times on SN.
[7] Brooks, D. (2012). The social animal: The hidden sources of love, character, and achievement. Random House Trade Paperbacks.