Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Are we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.by vova_hn2
- No, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.by xena
- I like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me.
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
by arjie - Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Job done.
by rvz - Please write a function that returns true when the user is using a residential proxy and false when it is a normal user on a residential IP address.by xena
- Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws.
This is not a tech problem. This is about what should or should not be legal.
Nor it’s a question of having time to implement solution X or Y.
If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop.
Ddos is already illegal.
I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.
by rufasterisco - I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless.
If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?
by Certhas - The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.by rcxdude
- What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.by jpcom
- Lots of build failure detailed investigations and gcc/kernel expertise in debugging misbehaving or outright ICEs.
That's it, I guess?
- They're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.by xena
- At this point they've mostly run out of material, so ANY type of content is valuable. Your small personal website, why would they scrape that? It's 10.000 additional words, wouldn't want to miss that. My Github repos.... got to get buggy code from somewhere I guess.
I get what you're asking, and I'm wondering the same. Not all sources are created equally and we see the results all the time. LLMs outputs nonsense all the time, like Flock cameras containing 5 grams of gold and ounces of copper, because they are completely on critical of their sources. Perhaps there's some weights that says: Kernel mailing list, MariaDB documentation and Microsofts Learning sites are 100% trust, Reddit 50%, 4Chan 10%, but I doubt it.
Anthropic might care a little bit, seeing as they scan books, but again, is it just all books? Because other than some flowery language I don't really see the point in scanning a 1970s paperback only spy novel.
by mrweasel - They're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream.
Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?
Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.
Now it just gets you a thank-you from the sales team for all the bandwidth you bought.
Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
by KennyBlanken - Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
- There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
- > Most scrapers are relatively honest in some way shape or form.
Did you miss a "dis" in there?
by Symbiote - How did you implement this? My site's getting hammered, any tips would be appreciatedby sporp
- >We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
by gruez - One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]by cogman10
- Manufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?by userbinator
- You don’t think it has anything to do with hostile states using fake profiles on social media to drum up discontent in the democratic world?by simmerup
- I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
by eYrKEC2 - Instead of punishing a criminal, you make everybody pay.
This is not only unfair to legitimate users having to pay.
It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.
Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.
Akin to “just use Cloudflare, it’s free”.
Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?
by rufasterisco - >I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
But in the case of anubis it's not even used for crypto. It's just wasted.
>You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.
by gruez - It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
- there is a captcha type thing you can implement in javascript that forces them to do math, I never remember what it's called but it has some anime catgirl as the logo, I imagine you could modify that a bit to make them mine bitcoin or something.by Suppafly
- Cloudflare is working exactly on enabling that, like it or not.by 0x_rs
- How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.by csomar
- The question is what exactly are you trying to achieve. Do you want to prevent them from gathering data or do you want to save resources?
If it is the latter, then, perhaps, some p2p (torrent-like) content delivery system could help.
by vova_hn2 - I think it's gonna kill off the internet not just for bots but to whole class of lower income countries, especially for younger learnersby pduggishetti