Agreed. If you only have 1-2 safe locations, either seed + passphrase or a 2-of-3 collaborative custody model might make more sense.
For the record, I actually use a seed + passphrase. I have metal backups of each in two different locations. The passphrase I also have memorized, and the seed also in a hardware wallet (so two backups each). I figured this was robust enough for my circumstances, and I like the ease of recovery and plausible deniability that passphrases offer.
If my ownership of bitcoin were to become widely known (say an exchange hack), I'd upgrade to multi-sig.