There's a pretty easy answer to this, actually.
Even in your hypothetical world of hyper-accurate AI/ML (which I have to say is an exaggeration of our capabilities today), the AI/ML algorithms rely on data generated by consumer behavior in a market system.
In a society absolutely devoid of markets and interpersonal trade, there wouldn't be even the data for the algorithms to determine how people value things, because what people get is what the government determines. The AI would simply replicate the government's decision function.
This actually happened in the Soviet Union, where in order to determine how much of things to produce, they'd rely on market prices from western economies to get an estimate of relative value. See Joseph Salerno's YouTube lecture on Calculation.
That's the cool technical answer I guess. Of course, the simplest answer is that you wouldn't be able to trust central planners to be benevolent, versus privileging themselves over everyone else.
That's not what we're talking here. I'm saying assume the capability to surveil enough of a population to get data. The much stronger version is assume "we" can track and compute information about individual purchases/transactions. "We".
reply
In a mixed economy it is as you say. We can use market data to learn about peoples' preferences and then we can make policy based on that. This is actually what the government does. It's what businesses do too. As ML gets smarter, and surveillance more widespread, planners can get better at implementing their objectives... in a way that people might not even realized they're being manipulated.
Which, in my opinion, is a scary thought.
reply