I got a very interesting question from @Bitcoiner1 here #658729 that for years, I myself I tried to answer it. But is quite hard to have a clear, straight unique answer. I couldn't find myself a general cause and specific tasks to avoid 100% these force closed channels.
But I will try to enumerate here in this post, some of my findings and give some advice, based on my long experience with running various nodes, public and private ones. I will not enter into super technical details, code, protocol procedures etc because I have limited skills into this part and I do not like to talk about something I really do not have knowledge.
I also invite all other LN devs, gurus, to come here and give more technical explanations for this issue and explain to all noobs what should do to avoid these force closures.
@kevin_lightning please can you give more explanations?
First of all, for all noobs, I will try to recap a bit, how a LN channel is created and maintained so you could have a better understanding about what we are talking about.
- a LN channel is a multisig tx between 2 peers (nodes), from which they both start spending sats, little by little, back and forth, moving the balance of the channel on one side or another. These little spendings are so called HTLC that practically are bitcoin txs, but not broadcasted yet.
- a LN channel have some particularities in regards of reserves (I never understand why do we have so many reserves but we still have force closures):
- channel reserve to avoid cheating, that is around 1% but can vary.
- anchor output reserve, that is 330 sats per channel and with a maximum of 100k sats per UTXO, used to bump the fee for a stuck channel closing.
- commitment fees: he balance fluctuates because commit fees for channels get renegotiated as the cost to publish to the bitcoin blockchain changes. When the commit fees rise (btc fees become more expensive) some of your local balance is reserved to meet those fees, resulting in your local balance dropping. When fees drop the reverse happens. To avoid this issue what you want to do is sum all your channel local balances with the commit fees for only the channels you have opened. There is currently no easy way to tell if you opened a channel or not but there was just a PR in lnd for example to make this easier on developers. Right now the solution I am using is to infer whether or not I opened a channel by its sent, received, and local balance. (See setting a default policy (LND)
- LN nodes are also using specific mempool fee rates to update these commitment fees. Each wallet app or LN implementation is using a specific one, is not a general one. But users can change it as they like. Here you have a discussion example about how these negotiating commitment fees could stuck a channel in closing procedure due to non-negotiable fees.
So, from my understanding what could cause a force closed channel?
- wrong negotiation of commitment fees, especially between LN implementations. For example if you have a LDK node and open channels with a LND node (that is using a different fee table), your node cannot negotiate clearly a huge fluctuation in mempool fees and automatically could trigger a force closure of the channel.
- your node or your peer node is unstable during long period (bad connection over Tor or weak VPN, hardware failures that cannot write in time the changes in the databases, weak memory etc)
- other forwarding peers that are routing your HTLCs and they have weak / bad systems. The HTLCs could got stuck, not routed and your end up with pending HTLCs. Your peers could trigger force closures if these pending HTLCs are passing a certain time.
I will try to explain a bit now how this situation of force closed channels because of huge fluctuation of mempool fees, based on my understand. I may be wrong in some parts.
Example of fee rates over time:
- you start with t=0, right now: 10 sat/vB mempool
- +1 hr: 15 sat/vB mempool <-- commitment tx fee updated here, peer goes offline now so you cannot cooperative close anymore
- +2 hr: 30 sat/vB mempool
- +3 hr: 15 sat/vB mempool
- +4 hr: 10 sat/vB mempool
- +any hr: @ any fee in the mempool: you will broadcast 15 sat/vB feerate tx
Because it is a force close, you are just publishing the last state of the commit transaction as-is, no fee rate is specified at the time of close, nor can it be specified at time of close because it is a pre-signed transaction.
As a result, the fees in the mempool doesn't really matter when you force close the channel. if you close it now, when fees are 10 sat/vB, it'll still broadcast with 15 sat/vB feerate since that was the last updated fee rate even if fees were 1 sat/vB you would still broadcast the 15 sat/vB tx.
this is an essential part of risk to manage in lightning: offline peers, it is not just an inconvenience, it can cost you sats as you see here if the peer was online, you could have instead negotiated a closing tx fee instead of using the most recent commitment tx fee update.
the commitment tx fee only updates periodically, so this usually happens when fees are moving up and down a lot. if fees are very stable over a time period, it doesn't happen as much because even if you have not updated the commit tx fee in a while, it is similar to what the mempool is later.
This risk is also controllable: by setting max-commit-fee you can limit the feerate you are willing to pay on the close. So for eg. if you specified that the max commit feerate should have been 60 sat/vB, and that you would handle fee bumping it if you needed to using CPFP, you wouldn't have overpaid as much on the fee for the channel close. But that savings would be balanced by the possibility of needing to manually fee bump it with CPFP later.
I hope with this I could clear out some issues about force closed channels.
It sucks, I know from myself, but for the moment it is what is and until the LN code will not have a big improvement in this matter, we have to deal with it.