Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Anubis does not stop me from browsing a site (not scraping, browsing with a human at the wheel)
On the other hand, Cloudflare and the others stop me dead. "enable javascript and cookies." Ok. "your browser is too old". (I have an old OS with the newest firefox esr that supports it).
sigh.
by m463 - Anubis also forces cookies and javascript, which deters me from many of the sites that use it that require neither.by nonamesleft
- Well, my call on this thing being useless waste of time of everyone involved was correct. Compute wasted on AI tokens alone is probably far more than some cpu for token solving, better invest time into caching it well (or blocking agentic traffic entirely if that's your jam)
- perhaps the author, instead of the "I'm so clever" theoretical arguments from the client side, actually implemented Anubis on the server side and observe the results.
I have set it up on a few sites being relentlessly hammered by clearly idiotic bot traffic, and it drops bot the traffic levels from insane to manageable. I don't even care if the traffic is AI or bots, if the bot has gone to the same effort as the author has the bot may even just access the site in a responsible manner and that's fine.
by patchtopic - We don't know who is doing the really dumb global scraping attack, but that's who it's meant to stop.by inigyou
- Funny, I'm working on a simple tool that pulls the atom feed of the latest patches from lore with Python, and I just slip a User-Agent header into the requests.get() and it works great.by cjd8
- Exactly, it's the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.by satvikpendem
- No, this is not the same as Cloudflare fascism. You're free to use any JS runtime to solve the challenge.by mike_hock
- It's getting to the point that the only way to get past the bot filters is to have bots do everything.by dlcarrier
- A world without cloudflare does not necessarily mean a world where anyone can access any website. It could very well be a world where anyone can point a DDOS at any website and make it inaccessible to everyone.by pibaker
- Cloudflare got it's early business from sites being taken down by DDOS attacks. The early users of Anubis were often sites taken down by scrapers.
Not a fan of either solution, but the sites simply going offline does not promote an "open web" either.
by nitwit005 - Open to whom?
There can't be a truly open web if the guys with all the resources in the world have the incentive to absolutely crush you by draining all of your resources.
They won't crush you out of malice, but by accident, like an ant.
by grim_io - Maybe free market principles apply here: if Anubis fails to reduce scraper/bot load on servers, or blocks too many desired users, then the sites that adopt it would probably scrap it.
If they’re keeping it even after all these posts, it must be stopping _some_ sort of undesirable traffic without costing too much desirable traffic.
by object-a - I was curious about the name:
> Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots.
* https://anubis.techaro.lol/docs/
> The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice responsible for maintaining order in the universe. The heart was the seat of the life-spirit (ka). Hearts heavier than the feather of Ma'at were rejected and eaten by Ammit, the Devourer of Souls.
* https://en.wikipedia.org/wiki/Weighing_of_souls#Ancient_Egyp...
by throw0101d - >>I was curious about the name:
That's knowledge usually learnt in primary school.
by aboardRat4 - I'm not familiar with the firewall utility, and my initial thought, in response to the question, was "anyone who's heart weighs more than a feather".by dlcarrier
- This mistakes the goal. It's not to block the scrapers, but to discourage excessive (and costly) scraping.
The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.
by nitwit005 - The cost is nothing. The browser implementation is too slow, mobile devices are too slow, and hash algorithms with hardware acceleration are too fast. You can’t balance these three constraints in a way that only keeps out the bad guys.
The hashing is just elaborate obfuscation. Anubis uses SHA256 which isn’t ASIC-resistant, and thanks to Bitcoin you could probably buy one off the shelf.
by cedws - There are multiple goals at play. They're easy to find in HN comment sections whenever these topics arise.
One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all.
Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web.
These are not the same goals.
by ssl-3 - > The exact adversary Anubis targets defeats it trivially.
Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive.
Anubis was not built to stop individual users with llms.
by yellow_lead - Claude Opus can make an optimized version of the proof of work solver that’s more than 10000x faster than the javascript one, in about 5 minutes time. Who is this stopping exactly? It’s the wrong tool in the wrong place.by drum55
- Anubis's primary goal is to prevent web scrapers from DDoSing a website. It's not meant to be an unbeatable challenge or only allow humans like Google's more privacy-invasive captchas.
You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers.
It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to not be too annoying for humans, to not block real users, and not be privacy invasive.
by novafunc - >tries not to be annoying
The little anime girl is pretty off putting. I bounce when I see it.
by what - Hmm, I don’t think I agree. The author is claiming that Anubis is meant to stop individuals using LLMs, but I don’t think that’s its purpose. I believe its purpose is to reduce mass scraping of data that puts excessive load on systems. It does that by increasing the cost of scraping, not by preventing it entirely.
Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors.
(That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)
by jdlshore - There is no distinction between the two. If one can do it the other can just as easily. I know multiple people right now that can scrape any a site very easily.
I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.
by ddtaylor - Except it does not actually increase the cost of scraping meaningfully. Compute is really cheap. The compute for minting an Anubis cookie will cost less than a thousandth of a cent even assuming the attacker uses the same JS implementation of proof of work rather than an optimized native implementation. That cookie can then be used for hundreds of requests.
How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. Just the bandwidth from a residential proxy will cost the scraper orders of magnitude more.
Proof of work just isn't a viable counter-abuse challenge, even for something as low-yield as scraping. (It might be economically viable as a counter against some types of DDoS attacks, since they're even lower yield. But in practice it just moves the attack surface to the proof of work validation service.)
by jsnell - It does it by blocking the scrapers because they are really stupid scrapers. go-away blocks them too, by seeing if they load images, a much quicker test.by inigyou
- There's plenty of arguments that mass scrapers have compute to spare but it seems to me that if Anubis makes it 100x more expensive to scrape then, for any given scraping budget, that means you get scraped 100x less. Which is the difference between your server buckling under the load or continuing to serve reliably.by TomatoCo