but aren't I still trusting the hardware wallet not to screw up turning my dice roll entries into a key?

No you're not because you don't trust, you verify. Thursday night when someone here asked me about that I opened the code and ran through it (as in besides code glaring, stub it and execute) end-to-end, to be absolutely sure I wouldn't give the wrong answer. That was for someone else. They should have done that themselves imho, but I try to be nice and do the effort, even for a stacker that I extremely dislike, which was the case.

Go through your setup and ask: if this part is pwnd what happens?

That is the way. Also: what do I do when unlikely event xyz happens anyway. Like... some crazy RNG integration error.