Apparently parsing and SSL are the bottlenecks when crawling the internet these days. I wonder how Cloudflare's pay-to-crawl policy will change things.
pull down to refresh
102 sats \ 0 replies \ @beameduplol 17 Jul
There is a tool I read up online to mitigate crawlers/scrapers, it's called uncaptcha, FOSS (if I am not wrong) and the way it works is makes browsers do a small proof-of-work to access websites. Usually takes 4 - 6 seconds on a normal computer, but adds magnitudes of delay to crawlers/scrapers
reply