I had a forced close 4 days ago and my funds have not come back. Browsing the git hub issues there are several users reporting the same thing and comments from the devs suggesting the LSP swept the funds due to a bug. I.e. here: https://github.com/MutinyWallet/mutiny-web/issues/668 and here: https://github.com/MutinyWallet/mutiny-web/issues/634
I was aware about the forced close issue but did not appreciate that LSPs could "sweep" funds due to bugs.
I feel like this trade off was not fully disclosed, or at least I missed it. It seems like users could potentially be rugged by the LSP if that's the case.
Appreciate the hard work of the Mutiny guys and what the project is trying to achieve but a bit surprised by this. Was this issue well know to everybody else and it's just me who thought self sovereign meant that funds could not be taken by the LSP in this way?
We're really sorry about this and working to fix the issues.
Currently there are two main issues that cause force closes in mutiny.
Paying to a zeus lightning address almost always causes a force closure because it uses hodl invoices so the user need to come online between the time the zeus user claims the payment and before the htlcs time out, this can be a few hour window and often no user will hit it. Ldk is doing some work to help prevent this to give us more leeway in when the user can come back online but this will still be a problem, frankly zeus lightning addresses are just a nonsensical way to use lightning. https://github.com/lightningdevkit/rust-lightning/issues/2698
The second being multiple instances of mutiny running for the same seed. We have a bunch of checks to try to prevent this but it is difficult. We've exhausted so many options we're beginning to suspect some browsers (mainly safari) just clears some of our state which can cause the error. In our latest release we've started using a fork of ldk so instead of force closing it'll fail to start up instead, just to add more protection.
After all this, if you do everything right but still get a force close, you needed to come back online within 24 hours of the force close to sweep your money. This is just the nature of lightning with its high availability requirements. Our latest release extended this to two days but the best solution would be to add watch tower support so it can be swept without the user. Ldk does have experimental support for this but not something we've built out yet.
We do work closely with the LSP and if the user's funds are swept we are always able to get the money back.
These always really pain me because everyone on the team uses the wallet everyday and haven't ran into any loss of funds issues, so it's really hard to try and reproduce what exactly is causing all the problems. Fixing these is our top priority but a lot of it is just trying something and putting it out into the wild and seeing if our number of channel closures go up or down. Posting any logs when you get an issue is always extremely helpful.
reply
frankly zeus lightning addresses are just a nonsensical way to use lightning
I disagree. One of the reasons they are not nonsensical is that they expose liveness assumptions about LN that many users (including wallet devs) did not realize it has or did not prepare for. Users often only learn when they get burned. More users are informed now, thanks to a perfectly innocent feature of zeus wallet, and that is a good thing. Also, LN wallets will have to handle hodl invoices better now, and that is good too. Hopefully, wallet devs will design better wallets as a result, including by disabling payments to destinations that they can't yet handle in a way that protects user funds.
reply
Also, LN wallets will have to handle hodl invoices better now, and that is good too.
Sorry for my ignorance, but I thought hodl invoices are indistinguishable to wallets? To them it just looks like the payment is still on route? Which is kind of a feature and not a bug?
So how could a wallet handle this case better if this case is not even reliably detectable?
On the other hand, iirc, Phoenix was able to tell that my payment is on hold when I used it to pay a bond on RoboSats 🤔
reply
Only way to detect it is having a list of nodes that create them. Ideally there should be a flag in the invoice saying that so wallets can act differently
reply
So how could a wallet handle this case better if this case is not even reliably detectable?
The main way has been by manually banning payments to LN nodes that are known for hodl invoices.
In the future, there will probably be more defenses implemented, like better ways to cancel a pending payment, charges for holding up HTLCs, etc.
reply
We've exhausted so many options we're beginning to suspect some browsers (mainly safari) just clears some of our state which can cause the error.
It's always Safari...
reply
Hey there, Graham from Voltage (the LSP Mutiny uses). I'll let the Mutiny team comment on the actual issues, LDK, sweeps, etc because we don't directly work on that. I just wanted to comment that any time this happens we work with Mutiny to send all funds back to users. So I understand the concern and annoyance, but you always get your money back. I know there's a lot of work ongoing to prevent this and make it better so it's one of those subpar solutions until we get the right solutions in place.
reply
FYI for anyone reading this. The user funds are not missing. This dude has a force closed transaction and then immediately opens an issue then blasts it on multiple social media platforms. He's got a transaction in the mempool but no LSP funds were swept.
Maybe you should chill for a second for us to help you before jumping to conclusions.
reply
Thanks for all the replies to this thread. The summary for me is:
1: my particular issue seems NOT to be an LSP sweep and the transaction seems stuck in Mempool due to the high fee situation. The Mutiny UI doesn't help showing 0 balance with no indication of unconfirmed funds, which goes against standard wallet UI, but thanks to @TonyGiorgio kindly looking into the GitHub ticket and demonstrating the transaction is in Mempool the funds appear to be safe assuming they are confirmed eventually.
2: the users who did suffer loss of funds with Mutiny are iOS users and due to a bug in iOS Mutiny PWA broadcasting incorrect state they lost their funds as is by design in the LN. However, funds are apparently being returned by the LSP (Voltage, as posted by @gkrizek and confirmed by @benthecarman ) so this is nothing malicious and is ultimately an inconvenience but not leading to any permenant loss of funds
3: The FC issue appears related to hodl invoices an innovation from Zeus which allows users to receive LN payments even when offline. The issue is that the HTLCs are often sort of in a pending state and that can lead to FCs in the case the sender is offline for an extended period of time. Another cause appears to be Safari browsers clearing state leading to errors (As per @benthecarman). Mitigation steps include not being offline for too long, or not sending to users of hodl invoices. Further mitigations mentioned are an LDK fork from Mutiny which will caused a failure to start node rather than a FC of channel. (As per @benthecarman ) Also there is talk of blacklist to prevent payments to hodl invoices or at least in the future a way to flag them for special treatment to mitigate the issue, such as the ability to cancel pending payments before they expire .(mentioned by @petertodd ) LDK is also working on some means to extend the time interval within which a sender must come online in order to avoid the FC. (As per @benthecarman) there is also talk of adding watchtower support by LDK to make sure FC--of they do occur--are swept in a timely fashion without user needing to be online (as per @benthecarman)
4: As @TonyGiorgio mentions, Mutiny is very clear that Mutiny is a beta project and that issues could happen, and to fund channels with this risk in mind. So people should not expect things to work perfectly at all times in such a fast paced development frontier as LN dev.
5: For me, I had been guilty of being naive not so much with Mutiny, as the team were always upfront on the risks and I never had much in there, but with other wallets like Phoenix I've been too cavalier. What this incident has revealed to me is that the nature of LN means that if you are not running a 24/7 connected LN backend or at least a 24/7 connected watchtower then funds will always be at serious risk in a LN mobile wallet. Not just because of malicious rug/ exit scam risk but because of bugs and things that can and do happen and can lead to loss of funds. Yes, I should have known this. But they polish and exceptional user experience of a wallet like Phoenix sort of hides the risk from users. Going forward I will be trying to keep this lesson in mind and be a lot more wary when using mobile LN wallets without hosting my own backend.
reply
Isn't Mutiny Wallet a self custodial Lightning Wallet? How is this possible?
reply
Because on lightning it is dangerous to go offline for a long time. On lightning, your funds are in a multisig address where you have one key and your channel partner has another.
Suppose you want to pay someone $5 in a channel where you have $100. To make that LN payment, you first make a contract with your channel partner. The contract says you will quickly let your channel partner have $5 as long as he shows you the preimage to the payment hash of a certain invoice. To enforce this contract, you use your entire channel balance as collateral -- all $100. So if you act slowly, your channel partner gets $100 from you instead of $5 as a penalty for breaking your promise that you would act quickly.
But LN payments can be delayed by anyone along your route, including your channel partner. So if you are using Mutiny, and your payment gets delayed, and you close the app, then -- when your payment finally goes through -- your channel partner will (try to) show you the preimage (but your phone app can't see it because it is closed), force close your channel when you don't respond (this triggers a kind of countdown during which you must let your channel partner have the $5 before the timer runs out), and -- if you still don't respond -- sweep your entire channel balance as a penalty for breaking your promise.
Your channel partner can do this even though LN is non-custodial because, when you clicked Send, you signed a transaction allowing your channel partner to do this. Once you sign a transaction, don't be surprised if someone uses it to take your money.
reply
(deleted and reposted, because I messed up the explanation!)
edit: and fixed it again... explaining LN is hard. :D
sweep your entire channel balance as a penalty for breaking your promise
You are incorrect here. The only way your channel balance can be swept is by either party publishing an invalid, revoked, commitment transaction. But if that happens, the other party, or a watch tower used by the other party, can use the revoked private keys to sweep the balance. This is a strong disincentive from either party intentionally committing fraud. Provided your wallet doesn't have bugs in it, this isn't very likely to happen to you.
Note that pending lightning payments are, roughly speaking, unconfirmed transactions that would actually make the payment if mined. The trick that makes Lightning work is that those unconfirmed transactions are revoked by revealing private keys unique to that specific transaction. But both sides of a channel need to be online for the revoking process to work. So if there is ever an unconfirmed transaction that isn't revoked in time due to an offline party, the other party that is still online it into an on-chain payment by getting that unconfirmed transaction mined.
reply
deleted by author
reply
deleted by author
reply
thank you for the correction
reply
Heh, my correction needed a correction!
reply
Because lightning.
reply
We literally have a big fat warning box that pops up when you use the wallet for the first time. Let us know what we can do to have this even more apparent so you don't miss it.
Also, this is a property of LN. Feel free to browse the github issues of LND, CLN, etc. and you'll see the same.
reply
Similar cases with LDK mobile nodes here (that are still not resolved until today, nobody knows where the funds go after the FC):
reply
LDK have issues.
reply
I know. That's why I am very reticent using LDK wallets as daily apps. I just test them to know how are working.
reply
I accept FCs being "issues", but a bug that causes loss of funds to me is a more serious matter as Mutiny calls itself self custodial. But if the LSP is able to sweep your funds then it clearly is not actually self custodial. So unless I'm missing something Mutiny seems to be misrepresenting itself. However, if this issue is not just Mutiny but other wallets like Phoenix too that would be a major concern as I have more sats in Phoenix and am also under the assumption it's "self custodial". But if this is an illusion I will need to rethink my threat model.
reply
But if the LSP is able to sweep your funds then it clearly is not actually self custodial.
The reason the LSP was able to sweep funds was because what is essentially a bug in Mutiny lead to an invalid, revoked, transaction being signed and published by the Mutiny wallet. This can also be an attempt at fraud, intended to steal money from the LSP. So the Lightning protocol deals with this by allowing the LSP to sweep all the funds, discouraging attempts at fraud.
Harsh. But still self-custodial.
The solution here is to use better wallets that don't have bugs like this. IIUC Mutiny is particularly vulnerable due to bugs in the underlying web browser storage implementation, which can lead to state information not being recorded/modified correctly.
reply
But if the LSP is able to sweep your funds then it clearly is not actually self custodial.
This is how all lightning works, and is not specific to Mutiny, LDK, or any other lightning node software. If you run two copies of your lightning node, one is behind, and that one that is behind force-closes, you will lose your funds. Sadly iOS-PWA has a bug which has been causing that for Mutiny, but they've been working with their LSP to send users their funds back in that case.
reply
Not sure if this is an issue specifically for Mutiny. I think is more about LDK. Let's not forget that Mutiny is an "interface" for the LDK node under the hood.
If Phoenix app will have the same issue, it doesn't mean that is Phoenix app fault because the Acinq node behind is having issues.
Let's separate these things and treat them separately.
reply
That's point.
reply
Maybe you should pay attention to the beta warning. That should be in your threat model.
reply
Perfect. I'm building an app use LDK, I'm thinking remove and add other lightning implementations.
reply
other options are harder to uses by a mile...
reply
reply
Sadly there are only really three (and a half) options for integrating lightning into an existing application - (full) LDK, ldk-node, greenlight/breez, and (kinda) LND. Integrating LND into an existing application is a ton of work, and lots to maintain, so generally hasn't been something people do, or have stuck with long-term.
The full LDK interface is a lot of work, but very flexible in terms of how it gets integrated into an existing application, with customization for how/where you store your data, how you generate all key material, how/where you get blockchain data from, how/where you do signing operations, several different models of potential watchtower integration, etc, etc. This is great if you need some of that customization, but if not its a lot of work just to build the same thing everyone else has.
ldk-node is a standard node that does all the usual things to build a lightning node, packaged up with a super simple API (start node, open channel, pay money, list transactions) that handles storage and keys and blockchain sync for you. It works great on mobile, and has some flexibility, but is generally targeted at devs who just want something that just works.
greenlight/breez SDK are fairly similar to ldk-node in their API design, but unlike ldk-node they rely on hosting from greenlight/blockstream and integration with the breez LSP, rather than being able to choose your own.
reply
You can use your own LSP with the Breez SDK
Good points Matt. I'll rethink again. I still no removed LDK.
Does Phoenix use LDK?
reply
No, Phoenix is using Eclair implementation. Eclair in French means lightning. Acinq - the dev team behind Eclair and Phoenix is a french team
reply
Ok. Thanks Darth that is reassuring I guess.
reply
Exactly
reply
This is a property of LN. I can go through LND and CLN GitHub issues and pull many instances of funds being revoked and swept by the other counterparty due to software bugs. This has nothing to do with LDK explicitly.
reply
Make sense
reply
This is a strange comment - both of the wallets linked are unreleased beta wallets, with bluewallet LDK support being quite a long ways from production, and has gone unmaintained for several years now. The issue you linked is also now a year old, and last I checked Bluewallet ldk was running an LDK version from several years ago (because its been unmaintained - they never shipped it and haven't removed the code). Bitkit is in beta, and both of the issues you link to the Bitkit folks have commented that they've fixed (without talking to us, I guess it wasn't an LDK bug).
Trying to suggest these issues are representative of production LDK use is more than a little insulting, not to mention the insinuation of theft. Cash App has used LDK in production for several years, powering a large volume of payments. While it wasn't always easy, its been very stable for a year or more (modulo lightning network hiccups, which come and go...).
Of course you shouldn't select a "beta" wallet as your daily driver, that's the point of "beta" :)
Indeed, its the case that some dev teams have taken some time to go from zero to production with LDK. This is an issue we've recognized and months ago we built ldk-node to address it - providing a super easy to use API which bundles everything for developers so they don't have to think about the super flexible, but somewhat complicated full LDK API. Sadly, none of the wallets mentioned here use it, in large part because it didn't exist when they started, but in some cases because they have different requirements.
The original issue in this thread with Mutiny appears to actually be an iOS-PWA issue, not an LDK one (though LDK did update to be a bit safer in the face of this bug recently), and not really a Mutiny one either (though one they'll have to fix, or maybe have by now). For some reason iOS allows PWAs to have two running copies at once, which of course causes lightning state machines to go haywire...not a lot we can do about that on the LDK end.
reply
None of those issues are fixed... not even after 1 year, the funds were never recovered. Beta or non-beta, the issue is there, don't try to pass the dead cat into their garden. That's why we test these beta version, to inform you about these issues.
You guys should talk more to each others and deliver better apps if you really want users to trust your software.
reply
The bitkit issue the devs believe has been addressed, at least per the final comment on the issue, I assume they're waiting on confirmation from the user before closing the issue. As I said above BlueWallet's LDK integration is fully unmaintained at this point. They haven't touched it in several years and never got it through a lot of testing. I'd believe it has bugs, it may even have bugs with the several year old version of LDK its using. But in either case its definitely not an example of how LDK performs :)
reply
All software have bugs, is understandable, especially in the early phase. The only thing that I expect is that these kind of serious issues (losing funds) will get more PRIORITY.
As you saw, that issue is old from 1+ year ago and is still not fixed! After that last comment on github, where more back and forth emails with the support guy, that went nowhere. Funds are still lost or nobody knows where they go.
If LDK have some devs building some new apps, you should stay in touch all the time with them and see all these serious issues, trying to help them identify the problems. Blaming them or the users using beta (for testing) is not the way...
reply
Software that is pre-release has very different priorities from software that is released. Indeed, released software with end-users putting lots of funds in these kinds of issues become high priority, but before its released, they may not always be, depending on what other issues developers are working on.
If LDK have some devs building some new apps, you should stay in touch all the time with them and see all these serious issues, trying to help them identify the problems.
We tend to! LDK in general provides very hands-on support for developers building LDK-based applications. Some developers take more or less advantage of that, but we generally don't have time to actively monitor github issues for all dependent projects. Instead, we let developers using LDK bring issues to us when they feel like it may be a bug in LDK or that LDK is doing something they don't understand. We are generally pretty responsive and get back to such requests within a day or so.
If LDK have some devs building some new apps, you should stay in touch all the time with them and see all these serious issues, trying to help them identify the problems.
I don't believe I ever blamed Mutiny, BlueWallet, or bitkit here. I only pointed out that looking at unmaintained multi-year-stale code (BlueWallet) and pre-release software (bitkit) is not a good indication of the maturity of LDK. None of that is blame, BlueWallet decided to go a different direction and never shipped their LDK logic, bitkit is working towards release but isn't there yet. There's nothing wrong with either of those.
I also noted that LDK can be hard to use, and that can cause issues by itself, that's on us, we don't ever blame any downstream developer for misunderstanding how LDK works, we try to improve the interface and documentation instead! But, LDK's interface is very powerful and low-level, which certainly requires a lot of work. This is why, mentioned above, we built ldk-node, to get developers from 0 to a running, safe, reliable lightning node in a week or less.
reply
There's nothing wrong with either of those.
Only that users testing them are losing funds and when are reporting these issues, the devs are trying to suppress their voice, ignore the github issues for months and not responding to support calls.
John Carvlaho called me crazy because I was trying to report the issue on their telegram group and on github: #155848 When in fact I was just trying to help identify the problem.
reply
Okay, I'm done arguing this, you can take it up with bitkit, but it sounds like you have and they told you to chill out and wait until they finish building before you complain about specific bugs. Maybe worth listening to them.
Random idea but have you tried restoring the wallet from seed phrase in Sparrow to check there?
reply
Yes, I tried. Nothing shows up.
reply
The LSP has the funds.
reply
I lost sats on munity too. Like the project but it’s not ready yet
reply
Have the same issue, crated in GitHub as well, my force close is from 26.10, still no funds back
reply
Mutiny is still a fully trusted solution unless you host it yourself, in which case you're better off running a proper remote node on a Pi / VPS or with Voltage etc.
Force closes happen more often because of hacky workarounds like hold invoice flows for async payments, and spotty connectivity that comes with a roaming device.
LDK also isn't a serious project, to date it's just a hole for shady grant money to attack Lightning with shit like Bolt12 and tricking people into thinking mobile nodes are trustless.
reply
The common denominator I've seen across Bitcoin is that almost everything Blockstream is involved with (either directly, or through employee involvement) turns out to be shit, yet God forbid you point out Blockstream's corporatization of and repeated attacks on Bitcoin with shit like Liquid.
>"oNlY bCaShErS mAkE tHaT aRgUmEnT"
Lol.
reply
Well they've raised like 200M in funding with little to show, so like half of the ecosystem is on their payroll even if in an indirect capacity.
Bolt12 is a logical thing for them to push as it would make users oblivious to whether or not they're using lightning vs. mints / liquid, and their greenlight cloud would have a feature benefit as sovereign nodes become crippled by the added latency.
The question is why is Spiral so aggressive in pushing it? Is it as simple as Matt and Steve ego's make them incapable of honest conversations? or is it something else?
reply
LDK is not related to Blockstream in any way.
It is indeed a serious project, albeit a WIP.
All LDK wallets are beta stage at best, and I'm pretty sure they all warn their users as much.
reply
It’s time to reconsider what we want from layer 2s
reply
deleted by author
reply
Too new client, generally. I once tried Eclair and had instant 5 FC in a row. After them i figured out a proper setup but it was just a setup in my case.
reply
Not sure on the FC. I heard it was something to do with HTLCs getting stuck and so you need to make sure you open up the wallet once every 24 hours to avoid the FC. But life got in the way for me and I had left it a couple of days. With that said there was no stuck HTLCs I was aware of so no idea why it FCd.
That was one thing, and I decided to give up on Mutiny for that reason. Was only using it for zaps anyway. For spending LN I'm using Phoenix and there are none of these kinds of issues. For zaps I am switching to WoS because it works and because the amounts are so small the rug pull wouldn't hurt so much.
But anyway this funds disappearing thing with Mutiny is a real concern because FCs are one thing and annoying, cost fees etc., , but outright losing funds I thought were self custodial is a different matter. I wonder if this kind of thing could also happen on Phoenix? If as Darth points out it's an LDK issue maybe any of these "non custodial" wallets that use LDK and LSPs could be at risk of similar bugs? Although like I said I have never had these kinds of problems on any other wallet than Mutiny.
reply
you need to make sure you open up the wallet once every 24 hours to avoid the FC
If this is true, it's stupid af.
I'm still scared af to open channels, regardless of app.
reply