Looking for some expert lightning help.Looking for some expert lightning help.
BackgroundBackground
There are two nodes in question:
Node A is running Core Lighting. Node B is running LND.
I have control over both of these Nodes.
Node B open channel to Node A. Channel was open for several months without issue and severed my needs well.
In recent historyIn recent history
I was using Node B and noticed I had one fewer channel. Node B is reporting a Remote Channel Close to Node A.
Question and Goal:Question and Goal:
- I would love to know why Node A closed the channel. Both Nodes had very high up time and very rarely showed any issue.
- I am likely to reopen the channel between the Nodes but I do want to prevent an closure from happening in the future.
Thanks in advance for any help.
You'll need access to the logs from the time it happened to tell for sure. Was this a cooperative close or unilateral?
For a unilateral close I might look for a stuck htlc, or a feerate disagreement. Are they using the same bitcoind backend? If not, I might expect occasional disagreements in channel feerates. If that persists for too long, they won't be able to keep the channel open. Was there a short cltv expiry delta on either side of the channel? I usually increase this from the default to reduce the odds of an unwanted force-close.
If it was a cooperative close, maybe check if there's any plugin or channel management running on either node that might be able to close channels.
It's important to note, that while these issues can create pain points from time to time, it's also what keeps nodes on the lightning network honest. If you allowed your peer to lowball fees on every update, they could attempt a pinning attack and all your channel funds would be at risk of being stolen. Similarly with an HTLC, if it's about the expire and your peer hasn't removed it, the only recourse is to drop to chain so that your node doesn't lose money.
I hope you're able to find some clue as to the root cause. It can either help inform sensible defaults for the implementations, or at least maybe identify some best practice for node runners.
I personally didn't close it, It would seem Node A appeared to close it for some reason.
Where would see the logs on a CLN node?
No, two separate systems
Node B has it marked as "Remote Force Close" I don't have any automated node management software running on either node.
Interesting in Node A's ride the lighting there is 1 htlc in a status of "Sent Remove Commit " even know there are zero channels showing.
Cooperative doesn't mean anything about you personally, it just means both nodes agree to close the channel, so they can both sign the transaction without the unilateral timeout script. Node B saying remote force close, means the CLN node unilaterally closed the channel because there was a disagreement or timeout with the other node.
In this case, it looks like there was an HTLC that CLN was trying to remove, but it didn't get a revocation from LND before it had to drop it to chain to resolve. That still doesn't quite tell us the root issue. It could be connectivity, or the feerates are too far apart to update the commitment transaction, or maybe another issue that we'd need the log for.
On my node, I set the log destination in my config with
log-file=cl.logand it runs from my ~/.lightning/bitcoin directory so the log file ends up there. You can also setlog-level=debugif you want them more verbose, or even specify a subsystem likelog-level=debug:channeldIt might be useful to check the LND log prior to the force close and see if it was ever offline or if it complained about any channel traffic. The root cause could have started up to 34 blocks prior to the force close hit the chain.
If I was going to create a new channel between two nodes, which I have control over, would it make sense to increase the CLTV Delta to let the two nodes be more forgiving to each other?
I am understanding CLTV Delta correctly?
Looking at my other Node B channels, CLTV Delta is set to 40 which I assume is default as I have not adjusted them.
Yes. If you increased it to 80, you would have twice the time to resolve the connectivity, feerate or other issue before resorting to force closing the channel. So your odds of hitting that would drop by at least half I would think. It would benefit your other channels as well because all nodes can have these disagreements. A CLTV delta of 40 seems reasonable, but it doesn't hurt to give your channel a bit more leeway if you're having issues with force closes.
Still, this kind of papers over the problem, so it would be good to get as much data as possible if this happens again.
Edit: In this case it was the CLN timeout that hit. To update that one, adding
cltv-delta=80to your config should do the trick. I should note that you can also manually override the feerate acceptance rules for this particular channel usinglightning-cli setchannel -k id=<channel id> ignorefeelimits=true. I wouldn't do this for any channel where you don't control the other side. If you can't punch the channel owner in the face, don't set it! It puts your channel's funds at risk.I am not oppose to papering over problems. :)
Is CLTV-Delta measure in numbers of Block?
For Core Lightning does ride the lightning (or another tool) let this be set from the GUI or only Terminal?
Yes, CLTV delta is the number of blocks, so for example, you could add 12 to the default to give your node about 2 more hours to resolve any issues with an active htlc.
RTL should allow you to edit your node's config file, but it doesn't give access to all of the setchannel parameters (even though I think it uses it under the hood to set channel fees.)
@DarthCoin
#658770
@Wumbo Here’s your answer man!
Thanks.
Both Nodes are running on Tor. So that may of also been a contributing factor.
I think I will increase the cltv-delta limit next time.
Thanks for asking this. I have a similar situation, but no closes yet. Generally speaking, I am often baffled by channel closures. They seem random at times.
First I got mad thinking I need to go spend a few hundred more stats to reopen but then realizing I control both sides.
So hopefully we can all learn something.