I agree that there is no bitcoin on the phone, its just a signing device.

As for just not turning on the phone... I think that's a pretty naive take. They can confiscate the device and hold you in jail because its the law.