Maybe, but the further down you re-org, the more mining rewards you reverse too. So let's say you have a 100 BTC utxo you got from me at block `t`. I selfish-mine, without publishing, a doublespend to myself in block `t'`. At `t+100` I publish my superior work chain of `t'+105`, but now, not only do I economically harm you, I also harm 100 blocks worth of block rewards that I've just reversed.

The question is now: are 100 miners + your 100BTC enough to make y'all say `naw brah`, and `preciousblock t` (and with that invalidate `t'`)? I think yes. Now I have to start all over, lost ALL my fiat and you still have your 100BTC.

This is why the real enemy is the one with a fiat printer: the state, the bank, the saylor.

It's not binary but gradual. If double spend attacks or just casual forks happened all the time, you would have to rise your thresholds. That is annoying and bad for the network.

bitcoin

Bitcoin Security Budget: Creating Asymmetries for 51% Attack Resistance

You already have to go through this. If you don't, you're still a slave.