This is an official statement about my node. TLDR: if you have channel with me, I recommend force-closing it. If your node did penalty/breach tx, please return my funds, see below. Contact see on amboss - https://amboss.space/node/033878501f9a4ce97dba9a6bba4e540eca46cb129a322eb98ea1749ed18ab67735?section=General - preferrably Telegram or email, haven't checked nostr in ages, but will do.

Penalty tx

See instructions here: https://gist.github.com/xmrk-btc/f21fd8649dcd573679f043836e3ef05f , note that my lightning address is published on amboss - pls use address from amboss, you don't want to pay any impostor. Ideally I would sign it with my node's private key, doing lncli signmessage but cannot do that while node is down, and cannot safely bring it up. I am linking a separate document because I may need to edit it.
I'd like to publicly praise all peers who do return me sats, and not sure how to do it so that it has maximum visibility. Suggestions welcome. Thinking about using Node Bilboard on Amboss -> Socials, but not satisfied, it may get lost in the spam. And it is valuable information - other things being equal, you strongly prefer a honest peer which doesn't try to rob you even when he can do so with relative impunity.
Although I may sue peers which don't return money - well, at least I can serve them using OP_RETURN. /s
Already 3 peers out of 13 reached out:
Still, I am a bit worried about creating bad precedent, penalty txs exist for a reason. And yes, this surely doesn't look like a succesful attack by me, but how about unsuccesful one? Even unsuccesful attacker should be punished to deter attacks. In my favor speaks the fact of having run a node for > 2 years, an attacker probably wouldn't bother.

Amboss Magma channel buyers

Headache... I want to return funds I received. But there is magma fee, channel buyer pays to seller (me) and to amboss, guess I should refund both, as this is not fault of amboss. Hoping to beg amboss to forgive something. I am planning doing keysends, but there may be peers who neither support keysend nor published contact info. May publish some info here: https://gist.github.com/xmrk-btc/456666f1681c65dd0469df786de5d632 .

Other peers - active

As stated, I recommend force-closing. I still hope for a miracle, so you may wait a bit, but ziggie has already force-closed, and I have discussed details with him. Remember there are watchtowers, don't try to steal or you will be punished just as I was.
Another possibility is doing zombie recovery with SCBs even if your node is runnning fine. The advantage is that you don't need to wait for your fund, and less on-chain footprint & fees. But it is unsafe for me, it is essentialy a cooperative closure, I lose watchtower protection. And is more work for both of us. May do it with some trusted peers. (Correct me if I am wrong...)

Other peers - crashed/zombie

My SCB / channel.backup seems up to date, so zombie recovery should be possible. I have one such case, haven't attempted it yet though.

What happened

I stopped lnd, installed updates for 30-60 minutes, wanted to reboot because systemd did not respond, but even reboot did nothing. Decided to do sync, wait a few seconds and do reboot -f. As if I haven't heard about umount or mount -o remount,ro ... sigh. lnd data are stored on ZFS pool as RAID1. Working theory is that important data stayed only in cache, lnd was running on 32 GB RAM machine while channel.db has 14GB so enough to cache everything. Still surprises me, usually lnd is writing at least 1 MB/s, as seen in dstat, perhaps some data is written and some is not?

Community

The reaction was amazing, special thanks to Rocket Node who insisted on me stopping lnd, and thus possibly saved me several BTC. Zap-O-Matic tried to calm me down. Ziggie and Alex Bosworth investigated. They all gave me strength to continue.

Future

I probably want to start another lightning node, but not sure if I have all the skills. I can be reckless and undisciplined, perhaps less so after this event. I have little idea how to choose good HW. And the node was distracting me from work. OTOH, it would be a pity to abandon such community. And this gives me some opportunities to improve, I can finally migrate to SQLite or Postgres which could have better replication.
Mate, you've been one of my earliest channels, always routing and cooperative. Some small zaps here from me, and I'm certain that all the penalty transactions will be returned. That's what kept me so excited about lightning all the time, we're all head2head in competition (at least the routing nodes), but never have I encountered a routing peer who didn't care, share their work and experience just like you did, so others can benefit.
In which industry do you find something like that? I'll wait!
So first off thanks for sharing the post-mortem. Keep us posted. And keep your hunger, we're still so early, and starting from scratch still gives you an early start.
All the best, stay strong!
reply
FYI, if you SCB, it's just telling the other side to broadcast their latest commitment tx. So your balance will always be timelocked 1 block and the their balance will be timelocked up to 2016 blocks depending on channel size.
reply
ok, thanks, cannot edit the original, so zapping your post to give it visibility and hopefully decreasing confusion
reply
64 sats \ 1 reply \ @Wumbo 5 Feb
Tip of my cowboy hat to the 3 nodes that have reached out.
Their karma stock went up with me.
reply
And added to future channels in case I don't have one already.
reply
Thanks for the write-up. I hope you stick around in the community. We are still treading the waters at the edge of the world, recklessly throwing our coins into beta environments and tools. But over time, it all gets easier and more stable, so maybe in 6-12 months there will be a reason to start up again :)
reply
Lol. In that range im pretty sure fees will be psycho.. im hoping that will finally be pay dirt for lightning… maybe in 24-48 months during a dip and with some tech fees will relax
reply
Still, I am a bit worried about creating bad precedent, penalty txs exist for a reason.
But what if, in spite of the concerns expressed by Peter Todd, we had LNSymmetry aka Eltoo where your channel partner could have just replaced your tx with the newest one instead of hitting you with a Justice TX
reply
0 sats \ 0 replies \ @anon 9 May
You're awesome xmrk. I (gotoshi.ca / galago) have had on and off channels with you over the past 18 months. Thanks for the summary on what went down. I barely know what I'm doing technically and interesting to read your story.
Best of luck - seeya on the flip side!
reply
0 sats \ 1 reply \ @anon 16 Feb
Was wondering why my channel to you (which I opened s few months ago) was force-closed from your side. Now I read this. Looks like this is an automated process?
reply
well, I finally started SCB recovery 15/Feb, so yes, automatic if your node is online.
reply
0 sats \ 1 reply \ @anon 7 Feb
zfs cache is for read, all writes go to disk.... at least every 5 sec. reboot -f could give you some trouble. just a head up. Good luck!
reply
yes, zfs cache doesn't seem sufficient to explain my problem. I was running with sync=standard, so ZFS should have honored fsyncs.
But there are write caches of the disks themselves. Just today I tried https://brad.livejournal.com/2116715.html. Created a new 5 GB zpool using the same two SSD disks, turned off compression and atime, did sync; reboot -f to simulate my case. And diskchecker showed no errors, even though both disks report write cache enabled (hdparm -I).
reply
Lightning can do wonders. One of those is for you.
reply
One of my oldest channel :( ... just sent the force close. I use nostr to communicate with node operators, so if you need something, check that medium.
reply
Force close txn in the blockchain, if you need something just ask.
reply
0 sats \ 0 replies \ @lada 6 Feb
Did your node automatically force close the channels that led to the justice transaction?
I was always under the impression that if you start lnd with a stale state, you are at least safe for a while because the stale channel will just stay offline when the peer returned a higher number. Unless you manually force close them with a stale state, hmm.
reply
I had several channels with you. Will sort it out later. Will return any funds that were inappropriately claimed.
You were a fantastic routing partner, and thank you for your efforts.
reply
Channels were forced closed and settled on chain. They had mostly drained towards me. Thank you, and good luck!
reply
Damn, I remember we had a channel for years, sad to hear. Hope everything works out
reply
Thanks for sharing.
It probably doesn't make a huge difference in what happen but what OS (linux, distro, start9, etc..) where you running LND on?
reply
Debian Linux 12 lnd 0.17.3 - no start9/umbrel or similar, just compiled from sources
reply
Did you use zfs from Debian contrib?
reply
no, zfs from bookworm-backports. Version zfs-2.1.12-0-g86783d7d9-dist according to zdb -h , updated to zfs-2.2.2-0-g494aaaed8-dist just before the fatal reboot.
reply
well, I wasn't asking for a donation , although zapping is a bit a point of stacker.news... Don't want to keep those sats (>9000) , will donate for useful tips, like how to sign message without lnd running, or how to improve my crappy instructions.
reply
If you have your seed words, you should be able to use Blixt Wallet without going online.
reply
Blixt is a good tool in this scenarios.
reply
Was wondering why my channel to you (which I opened s few months ago) was force-closed from your side. Now I read this. Looks like this is an automated process?
Sorry for this unfortunate turn of events. Hope you are able to recover.
35 sats \ 0 replies \ @9 5 Feb
No one robbed you.
reply