During a database backup last night, the filesystem where we store temporary (non-critical) database files got borked somehow. For some reason, the database's cope, likely recreating the temporary files over and over again without anywhere to store and reuse them, caused it to get really slow. It took us awhile to figure out that was the cause and it was fixed by literally turning it on and off again.
Thank you to everyone that reported it, and sorry for the downtime and the verbose error messages. (It's not wholly intentional to show you the sausage making but it helps us diagnose issues easier until we create some kind of error digest system.)
Show me Devs who respond? Devs that come back to you with an explanation of the problem and what they did to fix it?
Anyone? No? Yeah didn't think so.
This is why these guys are diamonds.
Y'all can have a brofist from Pub 👊
To be fair, I would say it looks like most devs don't do that since there's no easy way for them to reach most of their users and vice versa. We're in a unique position where what we're building is a very natural way to interact with our users and for them to interact with us.
You can hear more about this in this panel at minute 17:
For full context, watch from 15:28.
the wild west ate @TonyGiorgio
Tech shit could happen all the time. IT guys knows that.
Understandable and we appreciate your effort to put it back in time.
No need to apologize, man. This show us that SN is getting serious.
Happy that is not more serious.
This is a reminder that SN runs on only 1 server. In before SN is yuge
It's so awesome that we get to hear about this directly from the guy who maintains it. You guys rock. No apology necessary. Thank you for all your work. Keep it up!
Yes, agreed. I saw in the saloon thread there were others with the same slowness issue and I could wait patiently knowing k00b would come to the rescue and give a prompt update. Pretty decent UX.
Glad to hear you used my favorite technique for solving puzzling technical issues.
No problem glad you got it fixed
Hey, I see you less active these days on SN?
So often the rebooting or recycling seems to fix vexing problems. I have also found that sometimes the simplest correction to be the one that works. You must be using PostGre. Sometimes a beast. i have had good luck with MySQL before.
You could try duplicating your database files, setting up another area and run MySQL on the data to see which runs it better for your usage. Then make the new data fork into both databases and run them in parallel to see which is better.
All goody sir, appreciate the transparency and dedication to us compulsory stackers!
I was wondering if it was only me. Thank you for the debrief!
Thank you. It feels the speed and performance has improved from what we had before we had the issue.
There's no need of apology..it can happen.
Haha
Simple solutions.
Always nice.
I was worried!
@k00b
always grateful for what you tech wizards have built
Thankfully I didn't spend all my fiat on Black Friday Sales while SN was down. I still had enough left to do my daily bitcoin DCA. Phew.
Appreciate your effort on a holiday weekend. I know it's part of the deal when you run the show but still worth our appreciation.
I work in IT. It happens. Forward!
AWS asserted its dominance and tested our incident response
Unfortunately, the A bitcoin experiment post has been loading slower and slower. Haha.
Hat off, sir! First time in 6 months I have noticed a small issue and got it sorted fast. You do a great job! Thanks for all you do.
It's cool, these things happen. A simple reboot fixed it, just like I do to my computer all the time. I was only out of SN for 20 minutes.
🫡
Thank you for keeping us posted.
So in the end was just this?
Gotta love the power cycle
Have you tried turning it off and on again?
No worries! We are still early!
Glad it’s fixed now. Thanks for the update! sounds like the database just needed a little nap
Thank you for your courage.