This is an official statement about my node. TLDR: if you have channel with me, I recommend force-closing it. If your node did penalty/breach tx, please return my funds, see below. Contact see on amboss - https://amboss.space/node/033878501f9a4ce97dba9a6bba4e540eca46cb129a322eb98ea1749ed18ab67735?section=General - preferrably Telegram or email, haven't checked nostr in ages, but will do.
Penalty tx
See instructions here: https://gist.github.com/xmrk-btc/f21fd8649dcd573679f043836e3ef05f , note that my lightning address is published on amboss - pls use address from amboss, you don't want to pay any impostor. Ideally I would sign it with my node's private key, doing lncli signmessage but cannot do that while node is down, and cannot safely bring it up. I am linking a separate document because I may need to edit it.
I'd like to publicly praise all peers who do return me sats, and not sure how to do it so that it has maximum visibility. Suggestions welcome. Thinking about using Node Bilboard on Amboss -> Socials, but not satisfied, it may get lost in the spam. And it is valuable information - other things being equal, you strongly prefer a honest peer which doesn't try to rob you even when he can do so with relative impunity.
Although I may sue peers which don't return money - well, at least I can serve them using OP_RETURN. /s
Already 3 peers out of 13 reached out:
- Garlic - https://amboss.space/node/033b63e4a9931dc151037acbce12f4f8968c86f5655cf102bbfa85a26bd4adc6d9
- Cypher - https://amboss.space/node/02d5246fa6bccbb80732146e198d243c49348a29d847ca8486742f5bbec158744f
- Satway - https://amboss.space/node/02916bb52b33836e28a3649baea2e4a29c16fd8ad97901b2c97d408f428edef108
Still, I am a bit worried about creating bad precedent, penalty txs exist for a reason. And yes, this surely doesn't look like a succesful attack by me, but how about unsuccesful one? Even unsuccesful attacker should be punished to deter attacks. In my favor speaks the fact of having run a node for > 2 years, an attacker probably wouldn't bother.
Amboss Magma channel buyers
Headache... I want to return funds I received. But there is magma fee, channel buyer pays to seller (me) and to amboss, guess I should refund both, as this is not fault of amboss. Hoping to beg amboss to forgive something. I am planning doing keysends, but there may be peers who neither support keysend nor published contact info. May publish some info here: https://gist.github.com/xmrk-btc/456666f1681c65dd0469df786de5d632 .
Other peers - active
As stated, I recommend force-closing. I still hope for a miracle, so you may wait a bit, but ziggie has already force-closed, and I have discussed details with him. Remember there are watchtowers, don't try to steal or you will be punished just as I was.
Another possibility is doing zombie recovery with SCBs even if your node is runnning fine. The advantage is that you don't need to wait for your fund, and less on-chain footprint & fees. But it is unsafe for me, it is essentialy a cooperative closure, I lose watchtower protection. And is more work for both of us. May do it with some trusted peers. (Correct me if I am wrong...)
Other peers - crashed/zombie
My SCB / channel.backup seems up to date, so zombie recovery should be possible. I have one such case, haven't attempted it yet though.
What happened
I stopped lnd, installed updates for 30-60 minutes, wanted to reboot because systemd did not respond, but even reboot did nothing. Decided to do sync, wait a few seconds and do
reboot -f
. As if I haven't heard about umount
or mount -o remount,ro
... sigh. lnd data are stored on ZFS pool as RAID1. Working theory is that important data stayed only in cache, lnd was running on 32 GB RAM machine while channel.db has 14GB so enough to cache everything. Still surprises me, usually lnd is writing at least 1 MB/s, as seen in dstat, perhaps some data is written and some is not?Community
The reaction was amazing, special thanks to Rocket Node who insisted on me stopping lnd, and thus possibly saved me several BTC. Zap-O-Matic tried to calm me down. Ziggie and Alex Bosworth investigated. They all gave me strength to continue.
Future
I probably want to start another lightning node, but not sure if I have all the skills. I can be reckless and undisciplined, perhaps less so after this event. I have little idea how to choose good HW. And the node was distracting me from work. OTOH, it would be a pity to abandon such community. And this gives me some opportunities to improve, I can finally migrate to SQLite or Postgres which could have better replication.
sync=standard
, so ZFS should have honored fsyncs.sync; reboot -f
to simulate my case. And diskchecker showed no errors, even though both disks report write cache enabled (hdparm -I
).