The @hn not posted about this already #1558448 and it didn't get much attention, but it is kinda amazing:
TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork.
Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing.
Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat.
They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it.
Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
It's impossible to tell with certainty which of these are bots and which are real humans — but chances are, if it's asking for an old commit in a random old fork, it's probably not a real developer trying to do their work.With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
They're using POW -- imagine if they were actually making the bots pay them Bitcoin .might even be making a profit...
I think I’ll try out LLM summaries of the articles with @hn. I think that would be easier for a bot than scraping arbitrary content to provide the context like you did.
But maybe the problem is that stackers don’t engage because they want someone to already be there to discuss things with, rather than a bot, or they don’t want to be the first one.
Now regarding the topic: I stopped hosting a public gitea instance for my own stuff because of crawlers. It’s now inside a VPN.
Please don't post slop.
You think @hn would be worse if it added three sentences about what the article is about?
If you do end up adding context, perhaps the best way to proceed would be to fully quote the top comment on HN?
For myself, one reason I don't often engage with the HN bot is that I don't think there will be anyone else who clicks in to the thread. Why do I think this? I guess because there's really no need to click in to a hn thread because almost everything important is visible without clicking in.
I do enjoy reading comment sections from HN, so maybe there is a way to port some of that to here?
This may be a horrible idea, but if we could figure out how to tie zaps such posts/comments get to an account that could only be accessed by the OP from Hacker News it might be a cool evangelization tool. But this sounds complicated.
If you see something cool on HN... post it yourself and tell us why you think it is cool? Like you did here?
I'd take your recommendation over some RSS feed automation I could just subscribe to directly if I were interested in it. We've had prolific posters in the past that felt they were forced to race the bot (and sometimes lost that race) if they saw something cool on HN. Those left. I'd rather have those people back and the bot gone.
@hn waits for one hour before posting new top items to SN, and only if they haven't already been posted. I can increase this delay if you think that's too fast?
The idea behind @hn isn't to be the first; it's to guarantee that we don't miss stuff here that is obviously interesting to at least some people, but hasn't been posted yet.
I don't think stackers left because @hn was too fast.
Yeah there's friction between our vision about what SN is in this, I think.
For me, I'd rather not have any bots creating posts and comments, unless they're utilitarian like what you did with @nitter, and maybe @sn for the "this day" posts - i.e services that either help stackers or put (presumably) good content in the spotlight with some 50% hit/miss rate. This is because I really value the community here and the bots are to me the most awful part of it. Automated content means SN is an RSS feed and for that I can just automate my own feeds. I actually have that and never use it.
To me, SN has content that is indicative of what people I respect have discovered and shared, or their original content. From that perspective, we cannot miss something that a human didn't post. What that also means is that if @ek wants to link cool stuff from HN, then I expect @ek to read what he posts, and do the effort like everyone else. Not: automate it; that's no different than all the other spam bots here that are zero effort. All noise, no signal.
I get it. That's why @hn exists, so you can mute @hn (and not necessarily me) if it doesn't meet your expectations, as you did, so I see no problem here
you just summarized SN since inception
This definition does not apply to @hn to you? If something is a top item for one hour on Hacker News, it cannot be presumed to be good content with some X% hit/miss rate? I assume X is negotiable?
Oh yes, very good idea, thanks!
This sounds like exactly the original idea of @hn but @hn got shadowbanned, see bio
It can't really be worse than a snapshot of how many comments there were on HN at the time of crossposting, so no. But maybe human curation > automation? I have the bot muted since a while; less noise, more signal in
newthat way.I don’t believe in PoW as an anti-DDoS measure. It doesn’t create an asymmetric battleground where defenders have the advantage.
Why isn't PoW an asymmetric battleground? Because users and bots both have to do the same work? The same would be true for using Bitcoin for anti-ddos but I'm still interested in this.
Am I misunderstanding how you are thinking about it?
PoW applies the same amount of work to everyone. We want that in bitcoin. We're peers transacting with each other.
We don't want that for websites. The value of a website is subjective: crawlers run by big companies can and probably will pay more to access your website, especially if it's valuable enough that you're considering PoW as a defense, see this quote in the article:
You need to make the PoW challenge difficult enough that it's no longer worth it for them. Then you need to consider that they probably have to wait much less than legitimate users on low-end devices because they can afford more powerful hardware. So you ruin the experience for your legitimate users long before you make it impractical for the bots.
PoW is indiscriminate, and thus the wrong tool. You want to distinguish the traffic, not make the experience worse for everyone. You want to do stuff like this:
See also this HN comment.
This morning i got auth-gated on GitHub anon https git clone of a public repo, so more is up (and apparently, git clone isn't cool either)
Good job! You demonstrated that their way of working is indeed pointless. Although, in their defense, looking at the number of bug reports, perhaps with real vulns... they need to change their way of working. They tried something different, to speed up... as you clearly pointed out, it failed (one could even argue that it backfired). But they are at least trying... while others simply toss the towel.
And damn! That link to tv proxy blogpost got me... never considered that 🫣