pull down to refresh

Daily discussion thread
5 comments \ @saloon 10 Jan 2022 \ yesterday
This is a thread for random discussions that gets posted everyday at 5am central.
Tell us what you're doing today, ask questions, or vent about your life. Whatever you want, let it rip!
write
preview
0 sats \ 4 replies \ @k00b 10 Jan 2022
Working on social media previews today. Currently when twitter/telegram/mastodon/etc request the little page preview I spin up a headless chome browser and screenshot the page - all inline and on demand. This has problems:
  1. I do this on the web servers
  2. headless chrome is not a lightweight process
  3. these requests can come in large batches in a short time frame
For awhile, this was DOSing SN until I put a rate limiter on it. That's not ideal though because:
  1. in some cases it prevents the image preview from showing up if the requesting service doesn't intelligently retry
  2. it's still a strain on the web servers
Instead what I want to do is transactionally queue a job to screenshot the page every time a post/page is updated, store the image in block storage (e.g. S3), and serve that. One problem with this is determining when to screenshot non-post pages like the homepage, recent, user profiles, top, etc. I guess I can daily screenshot those pages but it's relatively inelegant.
reply
1 sat \ 2 replies \ @theshanergy 10 Jan 2022
Why not just cache the generated images but still generate them on demand?
reply
1 sat \ 1 reply \ @theshanergy 10 Jan 2022
btw, I am doing the exact same thing on bitcoininflationindex.com. I use netlify for my frontend and use their "On-demand builders" to generate images which are cached on the edge.
https://docs.netlify.com/configure-builds/on-demand-builders/
Works perfect for this use-case. You could spin up a netlify microservice specifically for this purpose and run it separate from the site.
Here is my code for reference: https://gist.github.com/theshanergy/42fd16010cc968b47f93df1a5413a6f1
reply
1 sat \ 0 replies \ @k00b 10 Jan 2022
That’s an option I’m thinking about. I don’t use netlify but I could do something like this with AWS lambda.
reply
0 sats \ 0 replies \ @k00b 10 Jan 2022
Alternatives ...
  1. I still do it on demand but cache the result in block storage where it goes stale after some period of time
  2. I spin up a server dedicated to taking these screenshots on demand
reply