You've just picked a very narrow slice -- transacting btwn A and B -- but have skimmed over like ten thousand pieces of trust on the way to realizing that event. Like literally, some electrons get sent out on a line when you're at your house sending your btc tx. Those are traveling over a substrate. They are mediated by a computer, running software on hardware. The btc software itself is absurdly complex. The algorithms that underlie its core properties rely on number theory that occasionally is obsoleted and compromised.
I don't know you, but my guess is that you have audited zero of these elements. Unless you are the world's foremost genius, and learned finite field arithmetic while being bored during high school English, you are incapable of auditing 99% of them. Even if you were capable, you don't have access to the 3Com hardware, or whatever, to make sure there's not some exploit in there.
You (or someone) may accuse me of nitpicking here. But nitpicking is exactly the point. Is it likely that someone has burgled your house, installed a keylogging camera above your faraday cage holding your btc rig? Is it likely an evil made replaced the BIOS at its Chinese manufacturer? These things are not likely. They are absurdly unlikely. And yet entire fathoms of trust are needed to exist in society on the way to making your 'trustless' transaction. Unless you live in the woods, eating grubs and fermented bark, you're trusting SO MANY PEOPLE and institutions and companies and entities of all stripes. You're just not thinking about them.
So yeah, minimize your attack surface. But get over the idea that anything about this is trustless.