This is certainly secure, but I would never use it.

I don't want a hardware signing device to REQUIRE communicating with an app, which then must connect to a server (even if I were running my own), just to unlock.

Yes you can't entirely trust a closed source secure element, but if the device is completely offline, there isn't really an attack surface. Use QR or sd card PSBT.