The site went down after midnight last night following our recent changes. The root cause was a bug in rewards introduced by upgrading our ORM (more on that later).
The bug led rewards to payout 86743171000 sats rather than the intended 8774 sats ... 867 bitcoin in rewards.
This made the site crash in various contexts when trying to express the rewards, rewarded stacker balances, or rewarded stacker stacked tallies (nearly everywhere) because our API layer expects 32bit signed ints.
The mistake has been fixed and the reward reversed where possible. However, one stacker managed to withdraw 20610000 sats that they didn't - in reality - earn before we could correct the mistake.
If we can manage to get the sats back, I'll drip all those sats back into the rewards pool over the next couple months.
If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew. It's ultimately my fault and I should pay for the mistake, but I'd prefer to give those sats back to the community.
the bugthe bug
The latest version of our ORM started returning type String for the result of postgresql SUM aggregates. Rewards add (+) the sum of all revenue earned to the sum of all donated sats. The sum of all revenue yesterday was 8674317 msats and the donation sum was 1000000 msats.
Because they were strings and not numbers, they got concatenated rather than added like numbers. What should have been 8774317 msats (8774 sats) became 86743171000000 msats (86743171000 sats) ... the result of visually joining the numbers together.
A couple days before we deployed the changes, when talking about the upgrade, I wrote to @ekzyis:
... if [a value that comes our of our ORM] ever ends up getting stored back in the DB, be paranoid. Currently they never go back to the DB after coming out ... except with earning [ie rewards] which only uses BigInts in js.
Apparently they ceased being returned as BigInts after the upgrade yesterday and I wasn't as paranoid as I should've been. Anyway, rewards should be functional and accurate again. I'll continue working on solving some of the other wonkiness that occurred as a result of the upgrade before deploying some of the features we have in the pipe.
... +1 for the pro-type-safety devs out there
Been wondering what happened. I figured the superconductor was used to finally create Skynet, which in turn destroyed Bitcoin and everything adjacent to it, starting with Stacker News. Glad I was wrong.
All joking aside, sorry about the pilfered coin. Hopefully the culprit decides to do the right thing.
You should absolutely consider putting limits and tigher security around your LND node if it's capable of withdrawing that much in one go. Users are one step function away from admin access to your funds. As a custodian, those need to be protected better.
The stakes are so much higher with Bitcoin it will force us to write better code.
Maybe it'll finally wean us off JavaScript.
hurrah.
sending dev and op hugs!
People have been criticizing my joke that we won't even let Paul do any math in the frontend, it has to be converted to rust web assembly and added there.
This is why. JavaScript no math.
You can mangle numbers pretty easy in Rust too. Rust and C++ may be called "static typed languages" but they are a lot less static than Go. You really have to go out of your way to muddle strings, 64 bit values and big integers in Go. In rust you can easily do it with a macro invocation you forgot you had in there.
Which one is it? a) You have personal experience with muddling types in Rust b) Heard war stories from other Rust programmers c) Assumptions based on the existence of a macro system
Yeah idk what this guy is going on about, never heard of someone fucking up an equation in rust.
We ordinarily do math in postgres stored procedures for this reason. It throws on under/overflow ... this is the one exception and I fucked up ... not sleeping enough and trying to get too much done too fast.
🫂🫂🫂
This is why prioritise native apps over web apps. Javascript's loose typing will make you lazy.
You don’t know who the stacker is? Or you do and you are keeping it private hoping they will do the right thing?
I know who they are. They've even done a user interview with us before.
I don't want to shame them into doing what I consider the right thing if they don't want to.
It was @DarthCoin wasn't it
Wow, now I am interested in what @DarthCoin would say to this
You know me well enough:
Glad to see you slowly coming back :) I've seen you mentioned in another comment you'll be back 100% in October.
And I feel honored that you replied to me first. But probably my mention was just as at the top in your notifications lol
And I agree, I don't get why some people thought about you regarding this. Not sure if they were serious or it was just some kind of weird joke
Speaking for myself it was 100% a joke, it was just amusing to me to picture someone like darth secretly making off with the money.
Also, welcome back @DarthCoin. Eager to hear your thoughts on WorldCoin lol
Indeed, are weird jokes. They should not joke about stealing funds by well known people. It's a really bad joke and the shame is on them. Yes, I will be sporadically posting when I am back from mountains. I have a lot of hard work to do on my citadel. Maybe I will make an offtopic post about that later.
deleted by author
As for me I was implying @DarthCoin would deserve a sats reward of that magnitude, therefore my proposed bounty wouldn’t apply to him - apologies if that tone did not come through clearly
No, I do not deserve the sats from a bug. O always want to work hard for earning sats, like I've done for the last 10 years. PoW is what makes Bitcoin unique. People slowly will realize that earning BTC is ONLY through PoW, not from cheating, not from stealing, not from scamming. Bitcoin is bringing out what is the most valuable and important from human nature.
They would never!
shame them. Stealing is wrong!
Public shaming is a thing. But hopefully that user does what’s right.
What's a user interview? 🤔
We occasionally do video interviews with stackers to get to know what they want better.
I havent had one of these yet lol
FYI tried to do some sleuthing to see top stackers listings but that page seems to be broken...
Should be fixed
Crap. Thanks!
Who dunnit?
I also created a ticket in the Prisma repo since I am not sure if this is expected behavior and if they are aware of this.
That's a hell of a (fraudulent) payday for the guy they pulled out.
Hope the best for you and the site as a whole.
Sorry that this happened. I hope that user will do the right thing and return the sats. If you are reading this, think about the karma. Very serious thing.
Well now that we're on this topic...who likes the idea of a sats pot that gets paid out to one user every now and then? could be a fun feature on SN.
The Stacker Lottery.
nice username btw, i watched the movie for the first time the other night and really enjoyed it
Thanks. I am a big fan of Hong Kong cinema.
my power went out for a few days, and i was trying to conserve battery by cutting all nonessential device usage. i was already pissed about losing my cowboy hat, and now i come back to find out k00b was throwing btc around like a first round draft pick at the strip club, while i was reading books by lamplight like a goddamn amish person?
Thank you for fixing it!
Oooof that's a bad bug. Hope the stacker returns every single sat. Thanks for the writeup @k00b
Oh shit... This sucks. I'll send the balance I've got on SN to the @K00b handle. Must've been a frustrating night behind the scenes. Do you know who the stacker is by chance?
No need for charity. I know who they are ... They are an early stacker.
JavaScript moment
See this is why I told k00b to build on Solana.
Had he done that, the network would have been down and he never would have lost the coins.
On a serious note, great postmortem. Curious for the Typescript folks, does TS actually protect against this? If so, how?
You are handling this mistake like a gentleman. I hope the person that benefited from this will do the right thing.
Stacker News Dev just became the 3rd most stressful job, right behind military personnel and brain surgeon.
TypeScript FTW :)
hate that this happened - cost of doing business at this stage
Sorry for your lost @k00t I don’t feel good when nice projects get hurt by bugs.
Sounds like a feature and not a bug to me
Ooof. Reverse Cantillon effect.
I hope the stacker goes honest and pays back.
This is why people need to be a lot more careful about what languages and systems they use when money is at stake. This is the kind of problem you just wouldn't slip up on with a hard static typing system like Go (even rust is more wibbly wobbly, sorry rustaceans). or have a concatenation of strings somehow slip past without flagging a compile time error.
I know all you cool kids think that Rust fixes this, but actually, it doesn't.
But Go would have.
By the way, I was just about to make a post about the recent spate of 503 errors. I guess that's fixed now :D
This is why Go. Just shut up about performance, who cares if it goes fast if all it does is lose money.
it does (but go ahead, garbage collect all you like)
Javascript is a shitcoin
Then Typescript is bitcoin?
c is bitcoin. Litteraly.
Then Windows only is Bitcoin
Typescript is bitcoin cash maybe
JavaScript is more like fiat, assembly is Bitcoin, and everything else is a shitcoin.
Rust is solana lets be real.
rust is Solana but it wasn't premined and it doesn't have downtimes
(what are we on about at this point)
C++ -> WebAssembly FTW
No typescript is still a shitcoin
based
Nice write-up, thanks.
I wonder if some static analysis tools like ESLint or similar would have picked up that.
TypeScript is the answer to this kind of problem.
We lint already. ESLint can only help with syntax.
A strict type system might've caught this.
Fuck me- Oh fuck it, fuck me twice!
My God
“If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew.” … and the rest of us will post a 20,610,000 sats bounty … unless it was @darthcoin
#217698
Wow, that is just unbelieveable it is just scary how one bug can reverse it all but hopefully the site has get into work and got fix wsih that similair bugs woukdn't show up here for moments not unkess people enjoy their time being in this community