That's a weak argument. There are a lot of possible attacks. A double spend that only affects someone and benefits the miner wouldn't harm the network much. A government concentrating huge amounts of hashrate and only mining OFAC compliant transactions, and ignoring blocks from other miners, is another example. And those attacks can be performed without having any Bitcoin at all. So, no, “trust” is not a solid defense strategy.