Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
by Waterluvian - Both groups exist. Scraper DDoS is likely to burn 100% of your CPU on git diffs if you host git. But static file sites are unlikely to notice it.by inigyou
- > Is the CPU cost actually worse than mildly annoying everyone
> What is the cost of 20% and is that cost worth “mildly annoying” everyone?
from the articled:
> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.
by colinsane - Not sure if you mean the solution, or the problem they were trying to solve.
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
by BowBun - this is an increasingly common situation. it goes something like:
the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?- i have a free, niche resource - it becomes too popular - i make it more efficient - now it's really popular, and people are "abusing" it - let's make them proof-of-work - ... and proof-of-work harder - but now "legitimate" users can't use it - ???which is called l402/x402. micropayments' day has come
by ynniv - What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones.
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
by inigyou - I mean generally what you said is true. But who is actually going to look at the linux kernel from their phone (probably just the maintainers in extreme circumstances)by desterothx
- Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)by easton
- It's the point that surprised me the most! We always used shallow clones, to speed the CI, I didn't knew that it got that much impact server side!by ygouzerh
- Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.by jacobvosmaer
- Ironically, defense by obscurity may be the way to go here.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
by sgsjchs - It doesn't matter what the hash is if it is inherently cheaper for a bot farm to compute the hashes than it is for a human to do it on their device. The human pays a greater cost in annoyance, wasted time, battery, and that means the PoW has failed its function. The bot farm owner does not care.by chr15m
- It's not really defense by obscurity (the JavaScript is public), more like defense by... being different?by afdbcreid
- I was thinking about this. Would it be possible for Anubis to have a code gen process that would make every deployment of itself sufficiently unique such that it's "annoying" to work around?
I do wonder about the premise as well: are people special casing for Anubis?
by rtpg - Render page in headless browser with JS, cache challenge solution, re-use on all your non-browser requests. Won't really change anything and it will auto-adapt to any new challenge that is being thrown.by dewey
- > Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
by slipknotfan - I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.by kdowns
- That's really funny. It's always been kind of amusing to me that Anthropic has this air about them of trying to be the most ethical AI company, but really exhibits the same behaviour as all the others.by myng111
- I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.by virgoerns
- You could also publish a list of IP addresses.by inigyou
- I really enjoyed the writing style in this article.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
by jdnier - It goes beyond mirrors, it's just something criminals have been doing since forever, to abuse all websites.by TZubiri
- It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.by atq2119
- Tavis Ormandy called this, about Anubis, almost exactly a year ago:
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
by tptacek - Yep, all security is about making attack more expensive than defense. If they have a farm of other people's devices, as alleged in the article, then it costs them less to calculate PoW than it costs ordinary users. Ordinary users have to sit there waiting, whereas bots are somebody else's problem.by chr15m
- I disagree. The kernel finds it effective - 66% of scrapers are turned away directly.
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
by nneonneo - If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
by harshreality - I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
by Demiurge - bzflagby incognito124
- What is "formally popular"?by rankdiff