We've all been been there, a channel gets closed, no-matter yet if it was a cooperative or force-closed, but somehow the close transaction hangs around and doesn't get confirmed. You want to clear the house, or need the pending onchain sats, but wonder how to go about it? Follow this guide and you'll hopefully find remedy.
But before, read this article from LND. Then read it again. Once you think you understand it, read it again. It's not super cohesive, since a number of things from the top need to be understood to grasp the section Channel closing, so ensure you spend some time in there.
Situation
This is written for LND, but similar methodologies are applicable for CLN. Eclair folks know their way around anyway.
You identified a channel-close, either in one of the GUIs you use, or from a Bot notification. But it doesn't confirm. First thing to do is SSH into your node via
ssh user@node-ip and get an overview of all pending (open, and close) with
lncli pendingchannels1.{ "channel": { "remote_node_pub": "123456789889786857463636288291929458767688493999392324", "channel_point": "dhksfkeh8458734kldjfksd39393932033fkk3003022244:1", "capacity": "10000000", "local_balance": "9999711", "remote_balance": "70137", "local_chan_reserve_sat": "10053", "remote_chan_reserve_sat": "10053", "initiator": "INITIATOR_REMOTE", "commitment_type": "ANCHORS", "num_forwarding_packages": "2", "chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator" }, "limbo_balance": "9999711", "commitments": { "local_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3", "remote_txid": "ta48849fkkek39947uuc80371ec96a88297465821ee79ff0b85f8624f83a7455f1cc", "remote_pending_txid": "", "local_commit_fee_sat": "2510", "remote_commit_fee_sat": "2510", "remote_pending_commit_fee_sat": "0" }, "closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"
What we see here is that our LND sees that channel as borked, we have a an opening_tx (
channel_point:vout), a local_tx and a remote_tx. What's usually the next step, we wonder why it's not closing, so we copy the"closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"
and copy it into our favorite mempool tool. Just for guiding sake, we check with mempool.space, and now it's the time to split into two scenarios
Scenario 1: mempool.space shows the closing-tx, but it's unconfirmed and the fee is too low
What many runners don't know, mempool.space runs with a mempool-size of 2GB, while most standard node-systems run with just 300MB. This is usually enough, but not in high fee environments.
So let's check if your local mempool has it:
bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3 error code: -5 error message: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
In this case your mempool probably purged the transaction, or LND never published it due to it's low commitment fee. Let's jump to the next section to see what we can do about it.
Scenario 2: mempool.space shows Transaction not found
Okay, we can assume that your closing-tx probably never hit the public-mempool, we can cut things short here and move to the next section, but just to be safe, let's check if our local bitcoin-mempool has it
bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3 error code: -5 error message: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
Complication
Why are there transactions of channel-closes with too low fee? Can't LND figure out what fee is necessary and get things just out there? I want my funds!
Well... and this is already way better than earlier last year
Let's assume best intend, no-one built that protocol to make things intentionally complicated. But since you have two parties involved in the process, and we should assume no-trust between those, we need to install processes to keep things fair, balanced and transparent. Read up on the HTLC-detail guide from Elle Mouton to make yourself more familiar with the details.
In short, we have 2 (out of 3) possible ways for a channel-close:
- cooperative-close, where both parties are online, one of the two sends a channel-close signal, and then both nodes get to court and negotiate a closing price. Since only the opener pays for the closing-cost, it's imminent it's not sky high. There are settings in
lnd.conflikecoop-close-target-confs=100which help to keep things rational from your side if the other side triggered the close. - force- or uncooperative-close, where either one of the two parties is unresponsive / offline, or other protocol / node-OS disagreements happen (like expiring HTLCs). In either case one of the two parties force-close, there is no court-negotiation, but based on the last channel_commit agreement the two nodes handshaken on, will be used for the closing-fee. Those agreements are also based on a few
lnd.confsettings, likemax-commit-fee-rate-anchorsandbitcoind.estimatemode2.
Either have their intricacies, but when the channel-close is not in your local mempool, or too low fee in the public mempool, you can't get the channel closed and the onchain-sats stay locked away.
Proposed Solutions
We won't outline every possible scenario, but a couple of most happening ones. Some will work, others not. To avoid this article becoming a master-class, let's go through a couple of those which are doing no-harm, but helping you worse case to get some more information what's happening.
For each of those attempts, prepare a second terminal window to your node, and filter (
grep) the channel-point and pubkey of the channel, so you'll be more informed after of what's happening. This terminal will monitor your lnd.log and show only entries where either the Pubkey of your to be closed channel-peer, or your mutual channel-point is triggered.sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -e 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT'
While Terminal 2 stays open, let's move to Terminal Window 1 for the next couple of commands. We'll try one after the other, and this guide will be verbose if there's risk or sat-spending involved. It's sorted from lowest risk / spend to highest, top down.
1. Reconnect
Many coop-close attempts fail, because when both nodes go to court, there is a possibility to not find consensus on the closing fee. Let's see if this is the reason, you'll see entries like commit-fee proposal staggering up, and whether you'll find consensus
lncli disconnect PUBKEY && lncli connect PUBKEY@ADDRESS:PORT
If the closing attempts fail, you'll see that each time, the delta between your node and your channel-peer will get closer. Eventually it'll work, if not, check your lnd.conf settings and eg change bitcoind.estimatemode=ECONOMICAL to bitcoind.estimatemode=CONSERVATIVE.2. Restart LND
Sometimes closing attempts miss a broadcast due to a missing connection to bitcoind. Restarting LND rebroadcasts every pending channel-close and sweeps, so give this a shot and check your terminal 2 while LND is taking it's time to come back up. Really, you need some patience here.
3. Reinitiate coop-close (and force-close)
Try to close the channel again, in rare cases this would solve something which 1. didn't:
lncli closechannel --chan_point CHANPOINT:VOUT --sat_per_vbyte XX. This may offer some more insight in your Terminal 2 window. You can do the same for the force-close, just need to adjust the command slightly. For the record, I never had this situation, but it's not doing any harm in case you know it's a force-close anyway: lncli closechannel --chan_point CHANPOINT:VOUT --force. We can omit the target sat/vbyte setting since force-closes use the last agreed commit-fee of both you and your peer.4. Increase local Mempool size
This solves for a couple of things, both Scenario 1 and Scenario 2 above, at least as a preparation. Open your bitcoind-configuration with
sudo nano .bitcoin/bitcoin.conf to lookup and edit the following entry to 2000, which increases it to 2GB:# Increase Mempool size maxmempool=2000
Restart bitcoind (typically
sudo systemctl restart bitcoind to take this setting into effect. And either LND already restarted by itself, or trigger a manual restart with sudo systemctl restart lnd. Watch Terminal 2, to see whether your 300MB previous mempool was too limited to accept your LND-closing. With 2GB, that limitation is lifted (a little) and potentially allows your LND to broadcast the closing-tx.5. Manually broadcast the closing-transaction
Eventually 4. allows your local mempool to pick up the closing transaction. Sometimes it doesn't. Or your mempool has it, but the public mempool didn't. Without either your bitcoind node, or the public mempools knowing about this tx, your channel can't get closed. In this event, let's look at how to broadcast it manually and see what response we get in both Terminals.
- Local Mempool check: Follow Scenario 1.
- If it's a No, does mempool.space have it? Then open the tx there, and grab the RAW-tx by appending the txid here:
https://mempool.space/api/tx/[TXID]/hex. Copy the whole long string, and execute the local broadcast in your Terminal 1 withlncli wallet publishtx RAWTX. - If that's a No too, check your LND, it should have it (only in rare circumstances, like SCBs, LND doesn't have the raw-tx:
lncli listchaintxns --start_height 818181 --end_height -1. Substitute818181with the current mempool height to filter the transaction output showing only not confirmed pending transactions (utxos). You'll get the raw-tx there, and can use it withlncli wallet publishtx RAWTXas above. - If your local Mempool has it, but mempool.space doesn't, there is no harm broadcasting it there as well. Follow Scenario 1 to retrieve the raw-txid from your local mempool, and copy it in here.
6. Increase the closing-fee
So mempools know about your closing-tx, but it's way too low. Let's assume it hangs at 10sat/vbyte (like many pending closing-tx these days), but let's also assume we won't see <30sat/vbyte anytime soon. Because you'd always have the option to wait for 10sats, your tx will eventually confirm. But you don't want to wait, then you'll need to pay up.
Disclaimer: This one is costing you. No matter who the opener of the channel was (who always pays for both un- and cooperative-closings), the next step costs You. So be concious about it.
The LND Guide linked above has a succinct closing channels section. And since you inhaled that guide, you'll know the next steps:
-
For Coop-Closes, you can
lncli wallet bumpfee -hto make yourself familiar with the options. It's pretty self-explanatory,lncli wallet bumpfee --conf_target 6 CHANPOINT:VOUTwill ask your local mempool for a sat/vbyte target and set the new closing fee. Withlncli wallet bumpfee --sat_per_vbyte 36 CHANPOINT:VOUT, you can set a confident 36 sats/vbyte yourself. Watch Terminal 2 and see what lnd.log tells you. -
For Force-Closes, you need a so called CPFP (Child Pays For Parent) transaction, which will become a child of your existing closing fee, and at the same time expensive enough, that both parent + child and their fees will be attractive enough for a miner to pick up and mine it. You can either do some pre-calc with a helper script, or try current mempool medium fee x 2. Not a problem if you don't start high enough, you can follow step 1. above here with this new child-transaction and bump it's fee with RBF any time later:
lncli wallet bumpclosefee -hfor the options:
lncli wallet bumpclosefee --conf_target 6 CHANPOINT:VOUTwill ask your local mempool for a sat/vbyte target and set the fee for the child-tx.lncli wallet bumpclosefee --sat_per_vbyte 100 CHANPOINT:VOUTwill set the child-tx at 100, presumably ending up with 50 sats/vbyte effective rate for both your parent and your child to be mined.
Note that none of this works if your local mempool doesn't have that txid. So follow steps 1-5 before trying this one.
Now but if it worked, you'll be prompted with a new txid for your child, which you can later check in your local mempool, and ideally it'll land in the public mempools to be mined as well. If it ended up too low, bump it with RBF (replace by fee) described in point 1, but targeting your child.
7. Go out of band
Just for completion, there are various miners who offer their mining service for your tx for a hefty cost. This sometimes might be necessary, eg for your channels where the other side force-closed on you, but you only have a static channel-commitment. If your peer is unresponsive or not cooperative to do a CPFP on their side, you'll wait until forever. Check your favorite search engine, ViaBTC or mempool.space for their acceleration services.
Hope you enjoyed this article. Please do share feedback and suggestions for improvement.
If this guide was of any help, I'd appreciate if you share the article with others, give me a follow on X or nostr, perhaps even donating some sats to hakuna@getalby.com
I'm also always grateful for incoming channels to my node: HODLmeTight
See the gist of it on my github.
Footnotes
-
For
umbrel, you need to jump through some loops for the commandlncliandbitcoin-cli. You need to add this whole string before:~/umbrel/scripts/app compose lightning exec lnd lncliand~/umbrel/scripts/app compose bitcoin exec bitcoind bitcoin-clirespectively. To simplify things, create a temporary alias withalias lncli="/home/umbrel/umbrel/scripts/app compose lightning exec lnd lncli"andalias bitcoin-cli="~/umbrel/scripts/app compose bitcoin exec bitcoind bitcoin-cli", then all of the above commands should work. ↩ -
Wonder what your channel-peers current commit-fee is? And whether they use anchors or static channel-fee commitments? Check out this bash-script, among other Node-Tools available as open-source. ↩
lncli pendingchannels? What might also be is that there are anchors of those channels still pending, or more likely they got stolen and your LND didn't notice it. You could identify missing anchors as the 330sat sweeps still pending atlncli wallet pendingsweeps. If it's anchors, you may see those 330 sat not atlimbo, butstolen.--forcethe bump (like in Solution 6 above)Appendix
Change-Log
-E:sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -E 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT'2000might be a bit of an overkill. If your hanging tx is not older than 2 weeks, 600MB will suffice for nowlncli wallet bumpclosefee --sat_per_vbyte 50 CHANPOINT:VOUTand check the effective-fee rate. LND might have improved their method, so it auto-calculates everything for youlncliorbitcoin-cli, adjust the call with-itlike this:~/umbrel/scripts/app compose lightning exec -it lnd lnclilncli pendingchannels. But as mentioned in 7), if commit=static, then you don't have anchors, and then yes, unfortunately, only two options remain: