pull down to refresh
21 sats \ 5 replies \ @k00b 21 Mar 2022 \ on: Daily discussion thread
I'm going to begin working on making notifications faster. For OG's who are as active as I am, notifications are likely getting really slow. At least my notifications are quite slow and it really creates a lot of friction to conversations.
I'm not sure why they're so slow but I think it's the thing that I'm doing that I haven't really done before: a big union of like 6 queries. I attempt to make each of these queries return as few records as necessary, but something about it is still making it slow. I'm hoping it's something simple because if this approach is flawed, I'll have to denormalize notifications which is going to suck.
After that I might do a review other low hanging perf enhancements, then probably get to work on media uploads.
It appears the issue is the table where a store all the "Item Acts" as I call them. That's all the votes, tips, boost, etc. In order to display your notifications about your recent post/comment stacking, I have to determine if one of these "Item Acts" were recent.
I think what I'm going to have to do is create a materialized view, i.e. a cache, with the relevant bits in it so it doesn't have to be computed every time you visit your notifications.
reply
This was actually a red herring. While the above is slow, the biggest problem is notifying you about descendant comments, but not notifying you more than once if you have more than one ancestor to a reply.
That is YOU1 > someone1 > YOU2 > ... > someone2 ... When someone2 comments, you only want one notification even though it's the descendant of two of your comments, YOU1 and YOU2. In order to make only one notification, I have to take the list of ALL replies to you, then make them distinct which requires hashing every reply to you ever.
As you become a mature user the issue gets worse and worse which is why my notifications are so slow. I need to figure out a hack that prevents me from having to make these items distinct.
reply
Found a hack that reduces the cost from 548089.02 -> 5193.26, 100x improvement ... still need to test, but hopefully that will let us grow another couple months without overhauling notifications.
reply
fascinating to read... there are probably papers out in obscure journals about how to optimize this... i'm amazed how responsive twitter notifications are as such a large scale...
reply
They denormalize them most certainly (lots of data duplication in the data models) which is fast but relatively complex to maintain.
reply