I do not do that and I would doubt many do that. (also interesting to see the comments).

I cannot speak to all the HW wallets out there, however, if there is a business whose entire premise is HW wallets and not your keys/not your coins slogan, they better be sure to control the manufacturing process of their chips if they want to thrive (that is probably even more important than the software since that can be easier to control). It is in their interest to do so. Trezor for example now started to manufacture their own chip if I understand it correctly.

Of course, you never know, but if for example one of the major HW wallet sellers would have a security issue as in scamming users, then this would be the largest scam in history :).