21 sats \ 1 reply \ @Hakuna 17 May \ on: Recover force closed channels lightning
In case your channel close are still pending, you may want to check out my guide at The Guide where your Lightning close-transaction can't get the channel closed
Let me know where you get stuck and I can help out further
You can check the integrity of your SCB file for your node with the following command below*
And yes, it triggers a signal to your peers (who are currently online) to FC your channel, which in the standard setup transfers the channel sats from your side to your LND onchain wallet. Which you will be able to access with your 24 words.
Always note that this should be something like a last resort, cause it's incurring costs for everyone. Better to mitigate risk by having good hardware, a UPS, and not mess with your production system if avoidable.
https://node-recovery.com/ is a bookmark worthy ressource, too.
*Test and verify chan-backup file
lncli verifychanbackup -h NAME: lncli verifychanbackup - Verify an existing channel backup. USAGE: lncli verifychanbackup [command options] [--single_backup] [--multi_backup] [--multi_file] CATEGORY: Channels DESCRIPTION: This command allows a user to verify an existing Single or Multi channel backup for integrity. This is useful when a user has a backup, but is unsure as to if it's valid or for the target node. The command will accept backups in one of four forms: * A single channel packed SCB, which can be obtained from exportchanbackup. This should be passed in hex encoded format. * A packed multi-channel SCB, which couples several individual static channel backups in single blob. * A file path which points to a packed single-channel backup within a file, using the same format that lnd does in its channel.backup file. * A file path which points to a packed multi-channel backup within a file, using the same format that lnd does in its channel.backup file. OPTIONS: --single_backup value a hex encoded single channel backup obtained from exportchanbackup --multi_backup value a hex encoded multi-channel backup obtained from exportchanbackup --single_file value the path to a single-channel backup file --multi_file value the path to a multi-channel back up file
Worth noting that the backup'ed channel.db shouldn't be used to come back online, but just to use chantools to do the SCB with your channel.db instead of your SCB file.
Coming back online with an outdated channel.db imposes great risk of losing funds via penalty transactions.
Binance's node doesn't look dead to me mate. It's a feature, they just go unannounced and hence have a sort of a white-list approach to only tell you their details by opening a peer-connection to you (or by getting in touch).
This way you can gatekeep 100% who opens to you, avoiding the usual way of just upping your min-capacity
I have no clue about CLN sorry. But here's a suggestion, log the context details on CLN's Github as an issue, there are plenty of people in the know.
Mate, you've been one of my earliest channels, always routing and cooperative. Some small zaps here from me, and I'm certain that all the penalty transactions will be returned. That's what kept me so excited about lightning all the time, we're all head2head in competition (at least the routing nodes), but never have I encountered a routing peer who didn't care, share their work and experience just like you did, so others can benefit.
In which industry do you find something like that? I'll wait!
So first off thanks for sharing the post-mortem. Keep us posted. And keep your hunger, we're still so early, and starting from scratch still gives you an early start.
All the best, stay strong!
Got you.
Yeah, that works obviously as well. If you swap out / in via Boltz, or with your other channel-peers using peerswap, the principle outcome is the same. The difference is, peerswap doesn't charge what Boltz does. So it's ~300sats onchain fee, at least temporarily, since peerswap doesn't have a fee-feature build in yet.
Glad that you got your 2 channel closures resolved!
The two in Thunderhub, are they showing up in
lncli pendingchannels? What might also be is that there are anchors of those channels still pending, or more likely they got stolen and your LND didn't notice it. You could identify missing anchors as the 330sat sweeps still pending at lncli wallet pendingsweeps. If it's anchors, you may see those 330 sat not at limbo, but stolen.To mitigate, you have 3 choices:
- just ignore them
- if they are not stolen: force-collect those anchor sats. In the current fee-environment, this is going to be unprofitable. Hence LND doesn't autosweep them, since it's not making any sense economically. But you could RBF the anchor transaction up to like 25sats/vbyte now and
--forcethe bump (like in Solution 6 above) - if they are stolen: try broadcasting the raw-tx of the anchor-tx (like in Solution 5 above). Best case, your LND will notice that those anchors are stolen and they'll disappear from your pendingsweeps list
Cheers.
Yes, it's always a consideration based on who has more sats on the line. If you have contact to the peer, who in this case has a higher risk due to more sats staled, you could just forward this guide
Or, if it's a big node / LSP and they don't care, or it's LDK and they can't CPFP (yet), you may just run it as a learning investment.
Please consider reading the appendix if and before doing your CPFP, might save you some sats.
Appendix
Note that there might be things I want to add on to, or fix. If you want to get the most recent version of this article, please check the github-gist which has all the changes.
The Guide where you got a Lightning close-transaction and can't get the channel closed
Change-Log
- Typo: Terminal 2 monitoring of two strings via boolean OR only works with an uppercase
-E:sudo tail -f .lnd/logs/bitcoin/mainnet/lnd.log | grep -E 'PUBKEYOFCHANNELPARTNER|CHANNELPOINT:VOUT' - Tuning: Increasing your bitcoin-mempool size to
2000might be a bit of an overkill. If your hanging tx is not older than 2 weeks, 600MB will suffice for now - Spending: CPFP child does not necessarily need 2x the sat-fee, try with your effective sat/vbyte target first, and bump if it's too low:
lncli wallet bumpclosefee --sat_per_vbyte 50 CHANPOINT:VOUTand check the effective-fee rate. LND might have improved their method, so it auto-calculates everything for you - ☂️ footnotes: If you enounter a tty-error for
lncliorbitcoin-cli, adjust the call with-itlike this:~/umbrel/scripts/app compose lightning exec -it lnd lncli
You can check which commit method your channel has with
lncli pendingchannels. But as mentioned in 7), if commit=static, then you don't have anchors, and then yes, unfortunately, only two options remain:- wait for your peer, who has a receiving output of the tx they could CPFP
- go out of band and accelerate the tx
Thank you for the addition. Chantools is indeed a wonderful helper, but this, SCB management and anchor bumping need another whole article, since it's quite advanced vs what I've outlined here. Will put this into my todo to follow up if there's interest.
There is no complexity of dealing with non-standard ports. If you're somewhat decently connected, gossip needs 2-7 minutes transferring changes across 95% of the network.
And once you have the custom-port, it doesn't change with your subscription. So port 1377 is as good as 9735, doesn't matter at all.
So you already figured that syncing the whole file on each state change isn't going to work. Here are two alternatives
- CLN / eclair already has it, LND is going to put more work into it this year and next: the move from boltd db to postgres (and SQL). It's on the roadmap, and pg allows synchronous replication across clusters, which is ideal for disaster failover.
- if you want to try something on your own, just for tinkering with the idea: remote raid solutions like drbd 👉
apt install drbd8-utilsare offering just that. You mount a remote server, ideally through a tunnel, and setting up drbd. As a result, the channel.db states get written to both devices at the same time, just like a local raid
Of course the latter has high demand on your network availability and latency. But it's better than nothing in case of catastophic failures.
It's just as long as the last outgoing HTLC expiry, and since I'm lazy, I keep it at 10,000hrs.
This is specifically helpful if you have FCs with pending HTLCs, since each of them needs to go onchain. If those are mostly outgoing, I stay at 10,000hrs and switch to 10 minutes, restart LND when the fees are low are all pending HTLCs are mature.
In a result, those pending HTLCs get batched into one tx, hence saving you onchain costs.
Then back to 10,000hrs.
Sharing only the ones I saw a number of runners not leveraging
[Application Options] # Number of logfiles that the log rotation should keep. Setting it to 0 disables deletion of old log files. maxlogfiles=20 # Max log file size in MB before it is rotated. maxlogfilesize=300 maxbackoff=5m max-commit-fee-rate-anchors=200 payments-expiration-grace-period=10000h gc-canceled-invoices-on-startup=true gc-canceled-invoices-on-the-fly=true sync-freelist=false [bolt] db.bolt.auto-compact=true db.bolt.auto-compact-min-age=730h [Bitcoind] bitcoind.estimatemode=ECONOMICAL [healthcheck] healthcheck.torconnection.attempts=3 healthcheck.torconnection.timeout=10s healthcheck.torconnection.backoff=30s healthcheck.torconnection.interval=1m healthcheck.chainbackend.attempts=3 healthcheck.chainbackend.timeout=2m0s healthcheck.chainbackend.interval=1m30s [sweeper] # Duration of the sweep batch window. The sweep is held back during the batch # window to allow more inputs to be added and thereby lower the fee per input. # Careful for the HTLCs of an FC with incoming=true, you can only batch those with # incoming=false (which are yours) sweeper.batchwindowduration=1000h # sweeper.batchwindowduration=10m
If you're in the Lightning game for non-custodial payment systems, Tor will suffice.
But in case you'll see more routing going through, and that usually starts at ~15-25 channels+, you may see Tor as a risk factor for availability and speed.
To avoid going hybrid and hence doxing your home-IP - as further explained here, I'd recommend checking out Tunnel⚡️Sats or delve into the FAQ there to evaluate options how to setup your own VPS (which often comes with KYC again, unfortuntely).