I personally just keep my hardware wallet handy in my desk. It has its own PIN that prevents access, and even with the pin, I use a passphrase/hidden wallet in addition to get even more security: https://wiki.trezor.io/Multi-passphrase_encryption_(hidden_wallets)

If someone stole my hardware wallet, I would restore my seed to another wallet temporarily and move those coins to a new hardware wallet with a newly generated seed just in case, but the urgency and pressure to do it quickly isn't there. The seed words are what you have to protect as much as possible.