Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I know people have opinions about Cloudflare but why not use it here, at least as a stop gap? Stopping bot traffic is one thing it does very well.by paxys
- Cost?by NetMageSCW
- It tried it. Hundreds of “genuine” visitors per day on a new website with no search engine presence and no links. That’s a very leaky fence..by spiderfarmer
- I set Cloudflare up a couple of months ago specifically to block bot traffic. It didn't do anything for me. Dumb bots were still hitting every special link on my wiki fast enough that the server was continually swamped running Lua scripts. 65% of the traffic for my English-language site was coming from Vietnam. But I didn't want to block Vietnam altogether, because my hobby site has genuine users from there too.
I eventually just shut down my mediawiki instance. I couldn't find a way to keep it online and still run on an affordable VPS.
by philipkglass - There’s an easy solution: Bruce should place large bets on the prediction markets right before publishing the relevant data. It is legal, ethical, and, for him, risk free.by rodarmor
- The real story here is that prediction markets were banned for a reason and loosening the rules is causing chaos just as was expected. AI plays little role in this story, hacking by humans would also be motivated by financial returns, unless the element is that AI hacking is cheaper and the returns are not so big.by tehjoker
- AI companies really do socialize the costs and privatize the profits.
Sites like The Numbers have to take on the cost of surviving the AI onslaught and the AI companies return nothing back to them.
by breve - worse; it isn't just no return, it is also cost.by jambalaya8
- To a point; as the post mentions, they added instructions for LLMs to their website and increased their licensing inquiries tenfold (the article didn't state anything about actual licensing payments being made though).
I don't think there would be any issues per se if the scrapers just paid licensing fees to get the good / complete data. But the issue was that they started to try and find exploits to get to data earlier.
by Cthulhu_ - I have small website with archive of older radio broadcasts mostly in Russian. Lately, 90% of traffic comes from USA ;)
Previously it was like 5%.
URL is http://radar.lv btw
Thanks God i can serve up to a terabyte per month of traffic easily, otherwise it would be a catastrophe. I am not against bot scraping, but I worry about stability for meat visitors.
So, I think of enabling payments for website visits cloudflare recently developed.
I also have the problem of old technology like the mentioned site. While my personal blog uses static generator, archive website uses ancient Drupal version, which has no security patches for many years already.
by theragra - Just FYI, the bigger companies all allow you to block crawlers via robots.txt:
# Block Anthropic (Claude) User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / # Block OpenAI (ChatGPT) User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / # Block Perplexity User-agent: PerplexityBot Disallow: / # Block Google's AI Training User-agent: Google-Extended Disallow: / User-agent: Google-Extended-Factual Disallow: / # Block Microsoft's Search & AI Crawler User-agent: Bingbot Disallow: /by djoldman - I won't eat your lunch if you put a sticker on your lunch box telling me not to.
- Claude Bot still (at least last month, and it's been doing it for over a year now) seems to have a bug when traversing (at least my sites), wherein it drops the trailing slash of a directory (which is present in the a href tag), then makes the request to the subdirectory without the slash I put in the link, then Caddy automatically responds to that (via the built-in File handler) with a redirect telling it to add the trailing slash, and Claude Bot then makes the request again with the trailing slash that should have been there in the first place.
So most subdirectory URLs get two requests from Claude bot, the first one needless because that wasn't the URL in the tag.
by pixelesque - This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...
I presume it would also cut you off even more from referral traffic.
by frereubu - 90% of bot traffic on my network of websites is through headless Chrome, via residential bots nowadays. Impossible to block. Not even for Google, as they inflate my Adsense numbers as well.by spiderfarmer
- This only works for crawlers that play fair, unfortunately there's many that don't. Including ones that use consumer devices like TVs [0] and mobile phone apps that offer incentives to consumers (if they're even open about it) to use their internet connection to e.g. crawl websites. There's probably an army of hacked toothbrushes and the like doing the same thing.by Cthulhu_
- What a throwback. Back in 2015 I have started Applaudience, which at the time was the only provider of real-time cinema ticket sales data. I still remember comparing our numbers against TheNumbers.com as part of calibration. I have since moved on to other businesses, but this remains one of my favorite pieces of technology that I have developed. Would love to bring it back one day.by gajus
- One thing I’m wondering is whether we need an open source set of technical patterns and libraries for dealing with this changing traffic mix.
Millions of small sites and creators don’t have the ability to design their own protections against large scale automated access. If useful content now attracts aggressive crawler traffic, many sites will be too expensive or unreliable to run.
Is part of the answer a community response? Perhaps a community-maintained toolkit, based on traffic data, that host sites can apply? It could include standard agent identification, rate limiting, traffic classification, access policies, caching, challenge mechanisms, logging, attribution and usage control etc etc.
In effect, we need much stronger road rules for today’s automated traffic, available as open technical patterns and libraries rather than every site owner having to invent this alone (they won’t).
by baud9600 - I don't think there is a technocratic solution to this problem. Very rich people are using their money to DDoS the internet, for no good reason. We just need to identify these people and fine or imprison them until they stop doing it. Residential proxy providers would be a good start.by danlitt
- Yes, a community effort against the botnets would be great.
The "good bots" which identify themselves and follow the rules are easy enough to block, so not a problem. It is the "bad bots" which pretend to be real users and hide behind residential proxies, and so are almost impossible to block at the moment, which are the problem.
Given that there are big companies openly (i.e. on the clearweb, not even darkweb) selling access to these residential proxy botnets of compromised smart TVs[0] and mobile phones and other devices, can we not simply get a database of the these IPs and block access from them? I would venture that almost every single one of those residential users are unaware that they have devices in their homes which have been compromised and are being abused in this way, so if they were to start seeing messages from more and more sites along the lines of "Access to this site has been blocked because unusual traffic has been detected from your computer network. Please check all devices on your network and remove any malware which may be routing this traffic." then maybe we could start addressing the problem at the source.
by m-i-l - The community response was respecting robots.txt, but since that was more of a gentleman's agreement than a legal requirement, hungry AI scrapers disregarded it.
Which brings us to the old fashioned flood control mechanisms. That is, the toolkit you propose already exists and has been used for decades in various iterations to protect against various forms of attack (slashdotting, DDOS attacks, overzealous search engines, and now AI scrapers).
Have you looked into those before? Companies like Cloudflare have been at the forefront of this field for a long time now.
by Cthulhu_ - At the risk of oversimplifying things from a distance, this site -- especially the free, public-facing part of it -- seems like it would be an ideal candidate for a rewrite using static site generator/framework. That, coupled with a bot-aware CDN should keep them online at a reasonable cost for many years to come.
Otherwise, I'm very curious to know more about their old and new architecture and what sorts of mitigation/scaling strategies they've started using to keep the site online.
by ethagnawl - Or just use a cache.by RobotToaster
- Yeah this seems like one of the easiest access models to create an excellent security model for -- basically just static publishing. Sounds like they were in need of a rewrite anyway!
It probably seems daunting but to be honest this feels like a weekend's work at this point with LLM assistance. Not to be glib!
by tengada1 - Ya this problem was solved 20 years ago with Varnish cache and Coral cache/CDN.
I think what's really going on is that bots expose how underpowered web servers has gotten in recent years. In the 2000s, even poorly-architected PHP sites tended to serve about 200 requests per second, with 1000+ being common for static sites. I remember when Node.js came out and claimed that it could serve more like 100,000 RPS due to its cooperative threading model. But today sites have a remarkable slowness to them, running many hundreds or thousands of database queries due to ORMs and N+1 problems, so that response times can be 500 ms or more and even 1000 simultaneous users stresses servers.
What went wrong is that nobody solved stuff like Russian doll caching in a general way separate from the programming language and database. We should have had ways to make dependency graphs using Etag headers as keys with real cache invalidation of dependent data. I went down that rabbit hole 10 years ago using touch events in Laravel with callbacks to handle cache invalidation when class model data was saved to the database. Also a query cache using Redis which I think might have been handled better at the database level anyway. After that experience, I can honestly say that cache invalidation is so difficult to get right that it's effectively an open problem. Meaning that programmings should use a package instead of rolling it by hand, and it should be a major concern from the start (along with sharding by user id or using something like Firebase).
Don't get me started on how the web should have been a P2P content-addressable memory anyway. Nearly everything should be available from a nearby edge peer, similarly to BitTorrent. But nobody bothered to solve how to make that work with HTTPS/SSL. I suspect that has to do with early flaws in the browser security model where the whole page has to be behind HTTPS or warnings appear. So it was never clear what was personally identifiable information (PII) or merely public data being served over HTTPS. To really solve that, we probably need real trust networks and maybe even zero-knowledge proofs.
Since these problems are so challenging to fix, and big companies can't be bothered to do it since they pulled the ladder up behind them, we're probably stuck with banal "are you human" challenge screens for the foreseeable future.
by zackmorris - A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.
Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
- That’s a traffic design problem. You should be happy that your work is valuable and also protected it against excessive requests. Simple.by x3haloed
- Building on AWS is a financial time bomb.
- Do you have any view on why the AI scrapers resulted in a heavier load than existing crawlers from eg search engines?
Where they more exhaustive or more frequent?
by BeeOnRope - the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint.
No, they decided that would be a great way to convince you of the narrative and persuade you to pay for "security" services that further the incumbent browser monopoly.
They're not "AI scrapers", they're DDoS'ers manufacturing consent.
by userbinator