how are they not massive honey pots?

In the case of Ledger... they are. This was a whole scandal a few years back and it made me regret using that (and telling others to use it).

It all comes down to the secure element being sufficiently hard to extract data from (and arguably since any ledger app can extract a key, that isn't "hard", it's trustmebro). Hardware security buys you time to move funds; that's really all it does.