The site went down after midnight last night following our recent changes. The root cause was a bug in rewards introduced by upgrading our ORM (more on that later).
The bug led rewards to payout 86743171000 sats rather than the intended 8774 sats ... 867 bitcoin in rewards.
This made the site crash in various contexts when trying to express the rewards, rewarded stacker balances, or rewarded stacker stacked tallies (nearly everywhere) because our API layer expects 32bit signed ints.
The mistake has been fixed and the reward reversed where possible. However, one stacker managed to withdraw 20610000 sats that they didn't - in reality - earn before we could correct the mistake.
If we can manage to get the sats back, I'll drip all those sats back into the rewards pool over the next couple months.
If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew. It's ultimately my fault and I should pay for the mistake, but I'd prefer to give those sats back to the community.

the bug

The latest version of our ORM started returning type String for the result of postgresql SUM aggregates. Rewards add (+) the sum of all revenue earned to the sum of all donated sats. The sum of all revenue yesterday was 8674317 msats and the donation sum was 1000000 msats.
Because they were strings and not numbers, they got concatenated rather than added like numbers. What should have been 8774317 msats (8774 sats) became 86743171000000 msats (86743171000 sats) ... the result of visually joining the numbers together.
A couple days before we deployed the changes, when talking about the upgrade, I wrote to @ekzyis:
... if [a value that comes our of our ORM] ever ends up getting stored back in the DB, be paranoid. Currently they never go back to the DB after coming out ... except with earning [ie rewards] which only uses BigInts in js.
Apparently they ceased being returned as BigInts after the upgrade yesterday and I wasn't as paranoid as I should've been. Anyway, rewards should be functional and accurate again. I'll continue working on solving some of the other wonkiness that occurred as a result of the upgrade before deploying some of the features we have in the pipe.
... +1 for the pro-type-safety devs out there
Been wondering what happened. I figured the superconductor was used to finally create Skynet, which in turn destroyed Bitcoin and everything adjacent to it, starting with Stacker News. Glad I was wrong.
All joking aside, sorry about the pilfered coin. Hopefully the culprit decides to do the right thing.
reply
You should absolutely consider putting limits and tigher security around your LND node if it's capable of withdrawing that much in one go. Users are one step function away from admin access to your funds. As a custodian, those need to be protected better.
reply
The stakes are so much higher with Bitcoin it will force us to write better code.
reply
Maybe it'll finally wean us off JavaScript.
reply
sending dev and op hugs!
reply
People have been criticizing my joke that we won't even let Paul do any math in the frontend, it has to be converted to rust web assembly and added there.
This is why. JavaScript no math.
reply
You can mangle numbers pretty easy in Rust too. Rust and C++ may be called "static typed languages" but they are a lot less static than Go. You really have to go out of your way to muddle strings, 64 bit values and big integers in Go. In rust you can easily do it with a macro invocation you forgot you had in there.
reply
Which one is it? a) You have personal experience with muddling types in Rust b) Heard war stories from other Rust programmers c) Assumptions based on the existence of a macro system
reply
Yeah idk what this guy is going on about, never heard of someone fucking up an equation in rust.
reply
We ordinarily do math in postgres stored procedures for this reason. It throws on under/overflow ... this is the one exception and I fucked up ... not sleeping enough and trying to get too much done too fast.
reply
🫂🫂🫂
reply
This is why prioritise native apps over web apps. Javascript's loose typing will make you lazy.
reply
You don’t know who the stacker is? Or you do and you are keeping it private hoping they will do the right thing?
reply
I know who they are. They've even done a user interview with us before.
I don't want to shame them into doing what I consider the right thing if they don't want to.
reply
It was @DarthCoin wasn't it
reply
Wow, now I am interested in what @DarthCoin would say to this
reply
You know me well enough:
  1. I respect and follow the Natural Law that says very clearly: do not steal.
  2. My personal values are way more up than some few thousands sats.
  3. I have enough BTC to live my life without taking what IS NOT MINE!
  4. If I would see in my user balance such a high change, I will contact immediately contact @k00b (as I did several times in the past).
  5. Just because I shit on shitcoiners it doesn't mean I will steal from such a nice bitcoin project!
  6. FUCK YOU ALL THOSE THAT THINK IT WAS ME! Shame on you that you didn't read ALL my posting history on SN to know me better.
reply
Glad to see you slowly coming back :) I've seen you mentioned in another comment you'll be back 100% in October.
And I feel honored that you replied to me first. But probably my mention was just as at the top in your notifications lol
And I agree, I don't get why some people thought about you regarding this. Not sure if they were serious or it was just some kind of weird joke
reply
Speaking for myself it was 100% a joke, it was just amusing to me to picture someone like darth secretly making off with the money.
Also, welcome back @DarthCoin. Eager to hear your thoughts on WorldCoin lol
reply
Indeed, are weird jokes. They should not joke about stealing funds by well known people. It's a really bad joke and the shame is on them. Yes, I will be sporadically posting when I am back from mountains. I have a lot of hard work to do on my citadel. Maybe I will make an offtopic post about that later.
reply
deleted by author
As for me I was implying @DarthCoin would deserve a sats reward of that magnitude, therefore my proposed bounty wouldn’t apply to him - apologies if that tone did not come through clearly
reply
No, I do not deserve the sats from a bug. O always want to work hard for earning sats, like I've done for the last 10 years. PoW is what makes Bitcoin unique. People slowly will realize that earning BTC is ONLY through PoW, not from cheating, not from stealing, not from scamming. Bitcoin is bringing out what is the most valuable and important from human nature.
They would never!
reply
shame them. Stealing is wrong!
reply
Public shaming is a thing. But hopefully that user does what’s right.
reply
What's a user interview? 🤔
reply
We occasionally do video interviews with stackers to get to know what they want better.
reply
FYI tried to do some sleuthing to see top stackers listings but that page seems to be broken...
reply
Should be fixed
reply
Crap. Thanks!
reply
reply
I also created a ticket in the Prisma repo since I am not sure if this is expected behavior and if they are aware of this.
reply
That's a hell of a (fraudulent) payday for the guy they pulled out.
Hope the best for you and the site as a whole.
reply
Sorry that this happened. I hope that user will do the right thing and return the sats. If you are reading this, think about the karma. Very serious thing.
reply
Well now that we're on this topic...who likes the idea of a sats pot that gets paid out to one user every now and then? could be a fun feature on SN.
reply
The Stacker Lottery.
reply
nice username btw, i watched the movie for the first time the other night and really enjoyed it
reply
Thanks. I am a big fan of Hong Kong cinema.
reply
my power went out for a few days, and i was trying to conserve battery by cutting all nonessential device usage. i was already pissed about losing my cowboy hat, and now i come back to find out k00b was throwing btc around like a first round draft pick at the strip club, while i was reading books by lamplight like a goddamn amish person?
reply
Thank you for fixing it!
reply
Oooof that's a bad bug. Hope the stacker returns every single sat. Thanks for the writeup @k00b
reply
Oh shit... This sucks. I'll send the balance I've got on SN to the @K00b handle. Must've been a frustrating night behind the scenes. Do you know who the stacker is by chance?
reply
No need for charity. I know who they are ... They are an early stacker.
reply
You are handling this mistake like a gentleman. I hope the person that benefited from this will do the right thing.
reply
Stacker News Dev just became the 3rd most stressful job, right behind military personnel and brain surgeon.
reply
See this is why I told k00b to build on Solana.
Had he done that, the network would have been down and he never would have lost the coins.
On a serious note, great postmortem. Curious for the Typescript folks, does TS actually protect against this? If so, how?
reply
TypeScript FTW :)
reply
hate that this happened - cost of doing business at this stage
reply
Sorry for your lost @k00t I don’t feel good when nice projects get hurt by bugs.
reply
Sounds like a feature and not a bug to me
reply
Ooof. Reverse Cantillon effect.
I hope the stacker goes honest and pays back.
reply
This is why people need to be a lot more careful about what languages and systems they use when money is at stake. This is the kind of problem you just wouldn't slip up on with a hard static typing system like Go (even rust is more wibbly wobbly, sorry rustaceans). or have a concatenation of strings somehow slip past without flagging a compile time error.
I know all you cool kids think that Rust fixes this, but actually, it doesn't.
But Go would have.
By the way, I was just about to make a post about the recent spate of 503 errors. I guess that's fixed now :D
This is why Go. Just shut up about performance, who cares if it goes fast if all it does is lose money.
reply
Javascript is a shitcoin
reply
Then Typescript is bitcoin?
reply
c is bitcoin. Litteraly.
reply
Then Windows only is Bitcoin
reply
Typescript is bitcoin cash maybe
reply
JavaScript is more like fiat, assembly is Bitcoin, and everything else is a shitcoin.
reply
Rust is solana lets be real.
reply
C++ -> WebAssembly FTW
reply
No typescript is still a shitcoin
reply
Nice write-up, thanks.
I wonder if some static analysis tools like ESLint or similar would have picked up that.
reply
TypeScript is the answer to this kind of problem.
reply
We lint already. ESLint can only help with syntax.
A strict type system might've caught this.
reply
Fuck me- Oh fuck it, fuck me twice!
reply
reply
“If the stacker that withdrew them is reading this and values sats earned more than sats taken, I'll voluntarily give you a nice chuck of what you withdrew.” … and the rest of us will post a 20,610,000 sats bounty … unless it was @darthcoin
Wow, that is just unbelieveable it is just scary how one bug can reverse it all but hopefully the site has get into work and got fix wsih that similair bugs woukdn't show up here for moments not unkess people enjoy their time being in this community
reply