pull down to refresh

Crawling a billion web pages in just over 24 hours, in 2025andrewkchan.dev/posts/crawler.html
85 sats \ 1 comment \ @Scoresby 17 Jul tech
related
What kind of bot would you like to see on SN?
1116 sats \ 28 comments \ @south_korea_ln 17 May 2024 meta
The three types of AI bot traffic and how to handle them - Vercelvercel.com/blog/the-three-types-of-ai-bot-traffic-and-how-to-handle-them
324 sats \ 2 comments \ @Scoresby 15 Aug AI
shreddit: Delete your Reddit datagithub.com/andrewbanchich/shreddit
1296 sats \ 27 comments \ @kepford 23 Apr 2024 meta
Perplexity is using stealth, undeclared crawlers to evade website no-crawl direcblog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
177 sats \ 2 comments \ @0xbitcoiner 4 Aug AI
f banks
749 sats \ 7 comments \ @cryotosensei 5 May econ
The Fox
200 sats \ 0 comments \ @bitcoingraffiti 18 Oct 2024 art
Just fucking use HTMLjustfuckingusehtml.com/
2674 sats \ 32 comments \ @k00b 14 May devs
Google just updated its algorithm. The Internet will never be the samewww.bbc.com/future/article/20240524-how-googles-new-algorithm-will-shape-your-internet
458 sats \ 13 comments \ @ch0k1 25 May 2024 news
[SN Feature Request] Follow feature
358 sats \ 16 comments \ @jerry 16 Jul 2022 bitcoin
Measuring minrelaytxfee across the Bitcoin networkdelvingbitcoin.org/t/measuring-minrelaytxfee-across-the-bitcoin-network/1989
1627 sats \ 8 comments \ @0xbitcoiner 18 Sep bitcoin
The Penal Preserve, Chapter 23
1335 sats \ 1 comment \ @cleophas 18 Aug 2024 BooksAndArticles
How do Search Engines Work?
521 sats \ 2 comments \ @Azik 11 Apr 2024 devs
Reddit blocks Internet Archive to end sneaky AI scrapingwww.theverge.com/news/757538/reddit-internet-archive-wayback-machine-block-limit
359 sats \ 7 comments \ @Coinsreporter 12 Aug news
Lets talk online anonymity: A few links and suggestions.
4977 sats \ 14 comments \ @Michelson_Morley 9 Oct 2023 tech
Ads are inevitable in AI, and that's okay - Rohit Krishnanwww.strangeloopcanon.com/p/yes-ads-are-inevitable-in-ai-its
289 sats \ 1 comment \ @Scoresby 29 Jul AI
Dragon's Deflationary Spiral: China's Economic Warning Signs
391 sats \ 4 comments \ @GhostofTruth 11 Nov 2024 econ
“mockingbird”
261 sats \ 10 comments \ @cryotosensei 18 Apr BooksAndArticles
The "true GDP" of the Bitcoin economy?
1627 sats \ 20 comments \ @ls0 5 Jul 2023 bitcoin freebie
Economic Crossroads: U.S. Labor Market Reveals Fragility
227 sats \ 3 comments \ @GhostofTruth 1 Nov 2024 econ
Road-To-Suave Fabs / Part 3.
2024 sats \ 27 comments \ @Fabs 3 Sep 2024 ideasfromtheedge
A deeper look at AI crawlers: breaking down traffic by purpose and industryblog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/
100 sats \ 0 comments \ @jakoyoh629 29 Aug AI