I was using mynode premium for about 1 month now and I used to have 13 channels up... Yesterday I noted it stopped responding and after a restart, I was unable to run LND cause of an error. I proceeded to telegram premium mynode support and got assisted by a member, he went through my logs and said my channel.db got corrupted so I could delete the file so mynode could create a fresh on!
Ok, I did as he said and I could log in to RTL, made a restore and my funds were back (BUT I lost all my channels on the process).
Today I used BOS OPEN function of Balance of Satoshis to reopen the channels, I opened 5 channels with 5 million satoshis each using BOS OPEN.
And thats where the vulnerability jumps in:
A few moments after bos open was executing, I could see in"Pending Open" section inside ride the lightning my 5 channels waiting the confirmations on the blockchain. AND THEN my machine crashed, it stopped responding and I had to reboot.
And there goes my funds:
  1. RTL reported I had no channels open, dont know if channels.db got corrupted again
  2. The pending open now shows 0 channels
  3. I was short 25 million satoshis on my onchain balance
  4. I can still see the funding transaction in the onchain transactions showing 0.25 btcs spent
  5. LND didnt provide me a backup (SCB) because the crash was on PENDING OPEN process, with no backup file.
  6. At that moment I couldnt see my channel on my peer's node through Amboss.space, even inside the closed channels it doesnt show up!
For all those running a node be carefull... In that windows before opening the channel that shows pending open, if you have a crash and have to restart the system you might lose your funds for good!
Here how it developed and how i managed to get my funds back:
I contacted Alex Bosworth through yalls, he then informed me that LND automatically keeps a channel.backup UPON pending open transactions at the following directory:
The process is automatic and you dont have to press a button to this backup work!
I perfomed the lncli restorechanbackup and voila, pending close 5 channels and got my funds back!
If that automatic backup wouldnt be created id probably lose those funds forever!
Thank you for all efforts related to this posts!!
I am sorry to hear that this happened to you, but glad you took the time to warn us and explain how it happened, I see a lot of LN peeps talking about how its so much safer than DEFI and yes it is from an economic stand point but the tech is still experimental
Hmmm. Sounds to me like you haven't lost anything except maybe the onchain fee for opening the channel that will now have to be closed because of your lack of backup (SCB). If the channel open transaction made it onto the blockchain then that channel will open and the funds are still yours albeit locked up in that channel. If the channel open transaction failed and never made it onto the blockchain then the funds allocated to that channel should still be in your on chain wallet.
Either way, I don't see how you could have lost the funds. They're either still in your on chain wallet or they're locked up in the channel that you opened in which case you can ask the node that you opened with to force close that channel. Only potential issue I can see is if your channel peer decides to be nefarious and publish a false channel state and take your funds. Normally a watchtower protects against this. I'm sure others more knowledgeable than me can chime in with details on what you can do to recover your channel funds.
They're not on my onchain balance, neither i see any channels open (cause of the corrupt channels.db), I cant even see my node on the peer's channel list on amboss.space!
I posted this on the private BoS telegram, btw, hope that helps.
Thanks for share And I have similar story when i use a app to open a channel with 250000sat. but unfortunately the transaction sent and confirmed and no channel opened. My little advice for this kind is open channel use the commmandline if possible. that would be safe than a third party app
Try to ask the peers with the pending channels to force-close them on you. The funds should be returned to your node's on-chain wallet.
Should these pending open appear on the peer node? Because on my end it doesnt show due to teh channels.db being corrupted!
Should the transaction be in pending force close state on their end? In that case, upon maturity it should bounce back on my onchain wallet right?? Thats what im hoping...
what what what?
running a lightning node on a rpi is a nightmare
indeed. I switched to a dedicated hardware for running it. This idea that you can just use a PI was ok at the beginning but now apps are getting more and more demanding and a single PI can't handle all that
Do you want to post your amboss profile here? I think maybe someone can help you...
If you look at a node on amboss like Bcash_is_Trash and then look at an edge of their connection to another node like this, and that will show you the onchain funding transaction. Along with that UTXO, you need to recover your LND's wallet seed phrase you could construct a force close transaction and broadcast the transaction to send all your funds minus fees to a good old bitcoin address.
As long as your channels haven't had any state changes (sats moved across them) you should have access to the total information need to generate the force close transaction. (If you had sats move across the channels you were in the process of opening, then you'd be in trouble because your counterparty could and probably would justice transaction you.)
(I'd add that I think your title would be better as "MyNode crash" or "corruption" since "vulnerability" usually implies illicit access by a 3rd party.)
The unfortune happened on june 19th
I have the seed, but I only have an old backup file (the one I used to recover the first time) so if I restore the wallet i believe ill restore only onchain funds
As I told... No channels seem to be opened on all peers (i looked on their pages on amboss), it seems that my channels werent ever opened
Once you have pending open transaction, If your machine crashes in that time, I dont know If the peers reject your channel (once the node is off)... Once the system had backed up online 3 confirmations were there, but no channels and no pending open channels... So could it be that my channels.db had been corrupted again!
(My onchain transaction stating those 25 mill sats) the 11 mil sats is me withdrawing the remaining to another wallet
(The more info screen... Those 5 onchain wallets my bos open sent funds to open the channels)
That's not going anywhere ;)
I'd say hang onto your seed/recovery phrase, note down the version of LND / BoS / Umbrel that your running and maybe the nodeId's of the channel you opened (though that last part isn't nec I think).
I'm not sure if the channels you open will attempt a mutual close or a force close on their end, but you'll want to be able to recover funds in both cases.
The ideal is of course that you can initiate a force close by building a transaction and broadcasting it on the bitcoin network. I'm (hopefully) a couple months away from being able to do that.
What you could do:
  • ask for help from github repos or bitcoin stackexchange, they may point you in the right direction
  • post this idea to bitcoin hackathons
  • post a bounty for help with recovery?
  • what I would do is setup a way to replicate the problem: create a new second node "Goofus" that opens a channel to your primary node and then you pretend it "loses state". You can study how it constructs a force close tx when it's working that try to reverse engineer that process. Looks like BoS code is relatively approachable being in js and well written: https://github.com/alexbosworth/balanceofsatoshis/blob/c9b6c894713edb43e9f2cffaf85552d50fbaba5d/network/remove_peer.js
Good luck, and I think you will get your coins back someday :)
Ill wait some days and hope they force close the channels themselves... But to be honest, im just an end user with simple knowledge about mynode... Those steps you mentioned is too technical for me! Id rather ask for assistance so I dont make mistakes by my own!
Thanks for sharing and sorry for your loss.
Are the funding transactions not somewhere on the blockchain? (Not sure if this is a dumb question)
I guess what I'm asking is what has made the funds unrecoverable?
The funding transaction shows on the blockchain yes!! Balance of satoshis funded 5 wallets to open channel with the peers, I can see the transactions on blockchain but probably as the crash corrupted channels.db, lnd cant find the channel point to identify that those funds are mine!
I can't recover because it didnt give me time to have the backup static channel (BSC), because the crash happened right after BOS open had the channels pending opened. You cant have the funds back once you have corrupted the channels.db and dont have proper backup file!
Is it possible that: if you still have the seed for the created lnd wallet, then it's OK that your channel db data is lost, because you could still cooperatively close the channel with the counterparty (because all that's required for that is your key for the 2 of 2 multisig that was funded into)? Well not 'OK' but I mean technically might the funds be recoverable, in that case?
As per the comments, this is both technical and tricky and certainly not a button press (and I can't even guarantee it's possible).
Very very unlikely that you actually lost the corn. Either the corn was committed to a channel or it wasn't. If it was, closing the channel (by you or your peer) would return the funds. If it wasn't, restoring your on chain wallet via your seed phrase will show the funds there.