The Guide where your Lightning close-transaction can't get the channel closed \ stacker news ~lightning

We've all been been there, a channel gets closed, no-matter yet if it was a cooperative or force-closed, but somehow the close transaction hangs around and doesn't get confirmed. You want to clear the house, or need the pending onchain sats, but wonder how to go about it? Follow this guide and you'll hopefully find remedy.

But before, read this article from LND. Then read it again. Once you think you understand it, read it again. It's not super cohesive, since a number of things from the top need to be understood to grasp the section Channel closing, so ensure you spend some time in there.

Situation

This is written for LND, but similar methodologies are applicable for CLN. Eclair folks know their way around anyway. You identified a channel-close, either in one of the GUIs you use, or from a Bot notification. But it doesn't confirm. First thing to do is SSH into your node via ssh user@node-ip and get an overview of all pending (open, and close) with lncli pendingchannels¹.

{  
  "channel": {  
	"remote_node_pub": "123456789889786857463636288291929458767688493999392324",  
	"channel_point": "dhksfkeh8458734kldjfksd39393932033fkk3003022244:1",  
	"capacity": "10000000",  
	"local_balance": "9999711",  
	"remote_balance": "70137",  
	"local_chan_reserve_sat": "10053",  
	"remote_chan_reserve_sat": "10053",  
	"initiator": "INITIATOR_REMOTE",  
	"commitment_type": "ANCHORS",  
	"num_forwarding_packages": "2",  
	"chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator"  
  },  
  "limbo_balance": "9999711",  
  "commitments": {  
	"local_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3",  
	"remote_txid": "ta48849fkkek39947uuc80371ec96a88297465821ee79ff0b85f8624f83a7455f1cc",  
	"remote_pending_txid": "",  
	"local_commit_fee_sat": "2510",  
	"remote_commit_fee_sat": "2510",  
	"remote_pending_commit_fee_sat": "0"  
  },  
"closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"

What we see here is that our LND sees that channel as borked, we have a an opening_tx (channel_point:vout), a local_tx and a remote_tx. What's usually the next step, we wonder why it's not closing, so we copy the

"closing_txid": "eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3"

and copy it into our favorite mempool tool. Just for guiding sake, we check with mempool.space, and now it's the time to split into two scenarios

Scenario 1: mempool.space shows the closing-tx, but it's unconfirmed and the fee is too low

What many runners don't know, mempool.space runs with a mempool-size of 2GB, while most standard node-systems run with just 300MB. This is usually enough, but not in high fee environments. So let's check if your local mempool has it:

bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

In this case your mempool probably purged the transaction, or LND never published it due to it's low commitment fee. Let's jump to the next section to see what we can do about it.

Scenario 2: mempool.space shows `Transaction not found`

Okay, we can assume that your closing-tx probably never hit the public-mempool, we can cut things short here and move to the next section, but just to be safe, let's check if our local bitcoin-mempool has it

bitcoin-cli getrawtransaction eeof599490930002dcc0db3bfa789eb4afbd89a37794a4c49a5f529ddb1b23e85b3
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

Complication

Why are there transactions of channel-closes with too low fee? Can't LND figure out what fee is necessary and get things just out there? I want my funds! Well... and this is already way better than earlier last year

Let's assume best intend, no-one built that protocol to make things intentionally complicated. But since you have two parties involved in the process, and we should assume no-trust between those, we need to install processes to keep things fair, balanced and transparent. Read up on the HTLC-detail guide from Elle Mouton to make yourself more familiar with the details.

In short, we have 2 (out of 3) possible ways for a channel-close:

cooperative-close, where both parties are online, one of the two sends a channel-close signal, and then both nodes get to court and negotiate a closing price. Since only the opener pays for the closing-cost, it's imminent it's not sky high. There are settings in lnd.conf like coop-close-target-confs=100 which help to keep things rational from your side if the other side triggered the close.
force- or uncooperative-close, where either one of the two parties is unresponsive / offline, or other protocol / node-OS disagreements happen (like expiring HTLCs). In either case one of the two parties force-close, there is no court-negotiation, but based on the last channel_commit agreement the two nodes handshaken on, will be used for the closing-fee. Those agreements are also based on a few lnd.conf settings, like max-commit-fee-rate-anchors and bitcoind.estimatemode².

Either have their intricacies, but when the channel-close is not in your local mempool, or too low fee in the public mempool, you can't get the channel closed and the onchain-sats stay locked away.

Proposed Solutions

We won't outline every possible scenario, but a couple of most happening ones. Some will work, others not. To avoid this article becoming a master-class, let's go through a couple of those which are doing no-harm, but helping you worse case to get some more information what's happening.

For each of those attempts, prepare a second terminal window to your node, and filter (grep) the channel-point and pubkey of the channel, so you'll be more informed after of what's happening. This terminal will monitor your lnd.log and show only entries where either the Pubkey of your to be closed channel-peer, or your mutual channel-point is triggered.

sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -e 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT'

While Terminal 2 stays open, let's move to Terminal Window 1 for the next couple of commands. We'll try one after the other, and this guide will be verbose if there's risk or sat-spending involved. It's sorted from lowest risk / spend to highest, top down.

1. Reconnect

Many coop-close attempts fail, because when both nodes go to court, there is a possibility to not find consensus on the closing fee. Let's see if this is the reason, you'll see entries like commit-fee proposal staggering up, and whether you'll find consensus lncli disconnect PUBKEY && lncli connect PUBKEY@ADDRESS:PORT If the closing attempts fail, you'll see that each time, the delta between your node and your channel-peer will get closer. Eventually it'll work, if not, check your lnd.conf settings and eg change bitcoind.estimatemode=ECONOMICAL to bitcoind.estimatemode=CONSERVATIVE.

2. Restart LND

Sometimes closing attempts miss a broadcast due to a missing connection to bitcoind. Restarting LND rebroadcasts every pending channel-close and sweeps, so give this a shot and check your terminal 2 while LND is taking it's time to come back up. Really, you need some patience here.

3. Reinitiate coop-close (and force-close)

Try to close the channel again, in rare cases this would solve something which 1. didn't: lncli closechannel --chan_point CHANPOINT:VOUT --sat_per_vbyte XX. This may offer some more insight in your Terminal 2 window. You can do the same for the force-close, just need to adjust the command slightly. For the record, I never had this situation, but it's not doing any harm in case you know it's a force-close anyway: lncli closechannel --chan_point CHANPOINT:VOUT --force. We can omit the target sat/vbyte setting since force-closes use the last agreed commit-fee of both you and your peer.

4. Increase local Mempool size

This solves for a couple of things, both Scenario 1 and Scenario 2 above, at least as a preparation. Open your bitcoind-configuration with sudo nano .bitcoin/bitcoin.conf to lookup and edit the following entry to 2000, which increases it to 2GB:

# Increase Mempool size
maxmempool=2000

Restart bitcoind (typically sudo systemctl restart bitcoind to take this setting into effect. And either LND already restarted by itself, or trigger a manual restart with sudo systemctl restart lnd. Watch Terminal 2, to see whether your 300MB previous mempool was too limited to accept your LND-closing. With 2GB, that limitation is lifted (a little) and potentially allows your LND to broadcast the closing-tx.

5. Manually broadcast the closing-transaction

Eventually 4. allows your local mempool to pick up the closing transaction. Sometimes it doesn't. Or your mempool has it, but the public mempool didn't. Without either your bitcoind node, or the public mempools knowing about this tx, your channel can't get closed. In this event, let's look at how to broadcast it manually and see what response we get in both Terminals.

Local Mempool check: Follow Scenario 1.
If it's a No, does mempool.space have it? Then open the tx there, and grab the RAW-tx by appending the txid here: https://mempool.space/api/tx/[TXID]/hex. Copy the whole long string, and execute the local broadcast in your Terminal 1 with lncli wallet publishtx RAWTX.
If that's a No too, check your LND, it should have it (only in rare circumstances, like SCBs, LND doesn't have the raw-tx: lncli listchaintxns --start_height 818181 --end_height -1. Substitute 818181 with the current mempool height to filter the transaction output showing only not confirmed pending transactions (utxos). You'll get the raw-tx there, and can use it with lncli wallet publishtx RAWTX as above.
If your local Mempool has it, but mempool.space doesn't, there is no harm broadcasting it there as well. Follow Scenario 1 to retrieve the raw-txid from your local mempool, and copy it in here.

6. Increase the closing-fee

So mempools know about your closing-tx, but it's way too low. Let's assume it hangs at 10sat/vbyte (like many pending closing-tx these days), but let's also assume we won't see <30sat/vbyte anytime soon. Because you'd always have the option to wait for 10sats, your tx will eventually confirm. But you don't want to wait, then you'll need to pay up. Disclaimer: This one is costing you. No matter who the opener of the channel was (who always pays for both un- and cooperative-closings), the next step costs You. So be concious about it.

The LND Guide linked above has a succinct closing channels section. And since you inhaled that guide, you'll know the next steps:

For Coop-Closes, you can lncli wallet bumpfee -h to make yourself familiar with the options. It's pretty self-explanatory, lncli wallet bumpfee --conf_target 6 CHANPOINT:VOUT will ask your local mempool for a sat/vbyte target and set the new closing fee. With lncli wallet bumpfee --sat_per_vbyte 36 CHANPOINT:VOUT, you can set a confident 36 sats/vbyte yourself. Watch Terminal 2 and see what lnd.log tells you.
For Force-Closes, you need a so called CPFP (Child Pays For Parent) transaction, which will become a child of your existing closing fee, and at the same time expensive enough, that both parent + child and their fees will be attractive enough for a miner to pick up and mine it. You can either do some pre-calc with a helper script, or try current mempool medium fee x 2. Not a problem if you don't start high enough, you can follow step 1. above here with this new child-transaction and bump it's fee with RBF any time later: lncli wallet bumpclosefee -h for the options:

lncli wallet bumpclosefee --conf_target 6 CHANPOINT:VOUT will ask your local mempool for a sat/vbyte target and set the fee for the child-tx.
lncli wallet bumpclosefee --sat_per_vbyte 100 CHANPOINT:VOUT will set the child-tx at 100, presumably ending up with 50 sats/vbyte effective rate for both your parent and your child to be mined.

Note that none of this works if your local mempool doesn't have that txid. So follow steps 1-5 before trying this one. Now but if it worked, you'll be prompted with a new txid for your child, which you can later check in your local mempool, and ideally it'll land in the public mempools to be mined as well. If it ended up too low, bump it with RBF (replace by fee) described in point 1, but targeting your child.

7. Go out of band

Just for completion, there are various miners who offer their mining service for your tx for a hefty cost. This sometimes might be necessary, eg for your channels where the other side force-closed on you, but you only have a static channel-commitment. If your peer is unresponsive or not cooperative to do a CPFP on their side, you'll wait until forever. Check your favorite search engine, ViaBTC or mempool.space for their acceleration services.

Hope you enjoyed this article. Please do share feedback and suggestions for improvement. If this guide was of any help, I'd appreciate if you share the article with others, give me a follow on X

or nostr, perhaps even donating some sats to hakuna@getalby.com

I'm also always grateful for incoming channels to my node: HODLmeTight

See the gist of it on my github.

For umbrel, you need to jump through some loops for the command lncli and bitcoin-cli. You need to add this whole string before: ~/umbrel/scripts/app compose lightning exec lnd lncli and ~/umbrel/scripts/app compose bitcoin exec bitcoind bitcoin-cli respectively. To simplify things, create a temporary alias with alias lncli="/home/umbrel/umbrel/scripts/app compose lightning exec lnd lncli" and alias bitcoin-cli="~/umbrel/scripts/app compose bitcoin exec bitcoind bitcoin-cli", then all of the above commands should work. ↩
Wonder what your channel-peers current commit-fee is? And whether they use anchors or static channel-fee commitments? Check out this bash-script, among other Node-Tools available as open-source. ↩

972 sats \ 1 reply \ @freetx 23 Jan

Nice writeup. Going thru this now.

I will add a few tips I picked up. Get chantools from lightninglabs. They have lots of handy utilities to help out in these situations (triggerforceclose, etc).

Secondly Lightning Terminal is a handy tool if you are on Umbrel (or anything where your lnd is in a container). It basically gives you a bash session with some other handy tools so you can run things from inside your lnd container.

10 sats \ 0 replies \ @Hakuna OP 23 Jan

Thank you for the addition. Chantools is indeed a wonderful helper, but this, SCB management and anchor bumping need another whole article, since it's quite advanced vs what I've outlined here. Will put this into my todo to follow up if there's interest.

599 sats \ 0 replies \ @0xbitcoiner 22 Jan

Fucking Awesome ! this should be found nowhere else on the net. thanks

20 sats \ 0 replies \ @nikotsla 23 Jan

Added to my personal wiki, thanks!

20 sats \ 0 replies \ @aoeu 23 Jan

It is for posts like this that I frequent stacker news. Well done!

I've run into these issues myself and this will be a helpful guide.

10 sats \ 0 replies \ @kurszusz 23 Jan

Hm... I try to understand this, but seems like is beyond my mental limits :)

I have a long way to learn...very long way. And as you know, the most important is the way, not the destination! ;)

10 sats \ 0 replies \ @south_korea_ln 23 Jan

We're still far from out-of-the-box solutions for such issues. I wonder if we'll ever get there. Still, seems like a lot of progress in a few years. Thanks for the write-up!

0 sats \ 1 reply \ @shyfire 10 Feb

deleted by author

0 sats \ 0 replies \ @Hakuna OP 10 Feb

I have no clue about CLN sorry. But here's a suggestion, log the context details on CLN's Github as an issue, there are plenty of people in the know.

0 sats \ 2 replies \ @BTC_LN 27 Jan

Great writeup! You helped me resolve two channels closures that were pending for a long time.

What if I have force closed channels that are still showing as pending in Thunderhub, even though they are confirmed and many weeks or even months old? How can I force LND to recognize them as closed channels? I did try to restart LND. Do you have any recommendation?

200 sats \ 1 reply \ @Hakuna OP 27 Jan

Glad that you got your 2 channel closures resolved!

The two in Thunderhub, are they showing up in lncli pendingchannels? What might also be is that there are anchors of those channels still pending, or more likely they got stolen and your LND didn't notice it. You could identify missing anchors as the 330sat sweeps still pending at lncli wallet pendingsweeps. If it's anchors, you may see those 330 sat not at limbo, but stolen.

To mitigate, you have 3 choices:

just ignore them
if they are not stolen: force-collect those anchor sats. In the current fee-environment, this is going to be unprofitable. Hence LND doesn't autosweep them, since it's not making any sense economically. But you could RBF the anchor transaction up to like 25sats/vbyte now and --force the bump (like in Solution 6 above)
if they are stolen: try broadcasting the raw-tx of the anchor-tx (like in Solution 5 above). Best case, your LND will notice that those anchors are stolen and they'll disappear from your pendingsweeps list

10 sats \ 0 replies \ @BTC_LN 28 Jan

Thank you!

0 sats \ 0 replies \ @Hakuna OP 23 Jan

Appendix

Note that there might be things I want to add on to, or fix. If you want to get the most recent version of this article, please check the github-gist which has all the changes. The Guide where you got a Lightning close-transaction and can't get the channel closed

Change-Log

Typo: Terminal 2 monitoring of two strings via boolean OR only works with an uppercase -E: sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -E 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT'
Tuning: Increasing your bitcoin-mempool size to 2000 might be a bit of an overkill. If your hanging tx is not older than 2 weeks, 600MB will suffice for now
Spending: CPFP child does not necessarily need 2x the sat-fee, try with your effective sat/vbyte target first, and bump if it's too low: lncli wallet bumpclosefee --sat_per_vbyte 50 CHANPOINT:VOUT and check the effective-fee rate. LND might have improved their method, so it auto-calculates everything for you
☂️ footnotes: If you enounter a tty-error for lncli or bitcoin-cli, adjust the call with -it like this: ~/umbrel/scripts/app compose lightning exec -it lnd lncli

0 sats \ 2 replies \ @BlokchainB 23 Jan

Bookmarked! I got a pending channel force close that hasn’t hit the chain for about a month now. The sats on my side are very low so I don’t need or want to hurry the close but will keep this handy just incase I get the urge to do it as a learning experience

50 sats \ 1 reply \ @Hakuna OP 23 Jan

Cheers. Yes, it's always a consideration based on who has more sats on the line. If you have contact to the peer, who in this case has a higher risk due to more sats staled, you could just forward this guide

Or, if it's a big node / LSP and they don't care, or it's LDK and they can't CPFP (yet), you may just run it as a learning investment.

Please consider reading the appendix if and before doing your CPFP, might save you some sats.

0 sats \ 0 replies \ @BlokchainB 23 Jan

Thanks yeah it’s the blockstream store node. Channel was offline for like 2 months and I have no reason why so I force closed it

0 sats \ 1 reply \ @bartoli 23 Jan

What about when the force close transaction has no anchor outputs (don't ask my why it doesn't, i have not idea), so bumpclosefee cannot CPFP it? Is accelerating the transaction the only way?

You can check which commit method your channel has with lncli pendingchannels. But as mentioned in 7), if commit=static, then you don't have anchors, and then yes, unfortunately, only two options remain:

wait for your peer, who has a receiving output of the tx they could CPFP
go out of band and accelerate the tx