(deleted and reposted, because I messed up the explanation!)

edit: and fixed it again... explaining LN is hard. :D

You are incorrect here. The only way your channel balance can be swept is by either party publishing an invalid, revoked, commitment transaction. But if that happens, the other party, or a watch tower used by the other party, can use the revoked private keys to sweep the balance. This is a strong disincentive from either party intentionally committing fraud. Provided your wallet doesn't have bugs in it, this isn't very likely to happen to you.

Note that pending lightning payments are, roughly speaking, unconfirmed transactions that would actually make the payment if mined. The trick that makes Lightning work is that those unconfirmed transactions are revoked by revealing private keys unique to that specific transaction. But both sides of a channel need to be online for the revoking process to work. So if there is ever an unconfirmed transaction that isn't revoked in time due to an offline party, the other party that is still online it into an on-chain payment by getting that unconfirmed transaction mined.