Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references.
Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.
by OhMeadhbh - The people who do this type of scraping to feed their AI are probably also using AI to write their scraper.by vaylian
- Do you think that if some CS programs taught parsing, the authors of the bot would parse the HTML to properly extract links, instead of just doing plain text search?
I doubt it.
by tuwtuwtuwtuw - How would recommend doing it? If I was just trying to pull <a/> tag links out I feel like treating it like text and using regex would be way more efficient than a full on HTML parser like JSDom or something.by ericmcer
- You don't need to teach parsing, that won't help much any way. We need to teach people to be good netizen again. I'd argue that it was always viewed as reasonable to scrape content, as long as you didn't misrepresent content as your own and if you scraped responsibly, backing of if the server started to slow down, or simply not crawling to fast to begin with.
Currently we have at least three problems:
1) Companies have no issue with not providing sources and not linking back.
2) There are too many scrapers, even if they behaved, some site would struggle to handle all of them.
3) Srapers go full throttle 24/7, expecting the sites to rate-limit them if they are going to fast. Hammer a site into the ground, just wait until it's back and hammer it again, grabbing what you can before it crashes once more.
There's no longer a sense of the internet being for all of us and that we need to make room for each other. Website / human generated content exists as a resource to be strip mined.
by mrweasel - It’s been some time since I have dealt with web scrapers but it takes less resources to run a regex than it does to parse the DOM (which may have syntactically incorrect parts anyway). This can add up when running many scraping requests in parallel. So depending on your goals using a regex can be much preferred.by mikeiz404
- Well, if they’re going to request commented out scripts, serve them up some very large scripts…by sokoloff
- > most likely trying to non-consensually collect content for training LLMs
No, it's just background internet scanning noise
by throw_me_uwu - This.
If you were writing a script to mass-scan the web for vulnerabilities, you would want to collect as many http endpoints as possible. JS files, regardless of whether they're commented out or not, are a great way to find endpoints in modern web applications.
If you were writing a scraper to collect source code to train LLMs on, I doubt you would care as much about a commented-out JS file. I'm not sure you'd even want to train on random low-quality JS served by websites. Anyone familiar with LLM training data collection who can comment on this?
by lucasluitjes - Two thoughts here when it comes to poisoning unwanted LLM training data traffic
1) A coordinated effort among different sites will have a much greater chance of poisoning the data of a model so long as they can avoid any post scraping deduplication or filtering.
2) I wonder if copyright law can be used to amplify the cost of poisoning here. Perhaps if the poisoned content is something which has already been shown to be aggressively litigated against then the copyright owner will go after them when the model can be shown to contain that banned data. This may open up site owners to the legal risk of distributing this content though… not sure. A cooperative effort with a copyright holder may sidestep this risk but they would have to have the means and want to litigate.
by mikeiz404 - As for 1, it would be great to have this as a plugin for WordPress etc. that anyone could simply install and enable. Pre-processing images to dynamically poison them on each request should be fun, and also protect against a deduplication defense. I'd certainly install that.by Anamon
- Sounds like you should give the bots exactly what they want... a 512MB file of random data.by bigbuppo
- 512 MB of saying your service is the best service.by AlienRobot
- Most people have to pay for their bandwidth, though. That's a lot of data to send out over and over.by kelnos
- Scraper sinkhole of randomly generated inter-linked files filled with AI poison could work. No human would click that link, so it leads to the "exclusive club".
- That's leaving a lot of opportunity on the table.
The real money is in monetizing ad responses to AI scrapers so that LLMs are biased toward recommending certain products. The stealth startup I've founded does exactly this. Ad-poisoning-as-a-service is a huge untapped market.
by kelseyfrog - when I used to crawl the web, battle tested Perl regexes were more reliable than anything else, commented urls would have been added to my queue.
- DOM navigation for fetching some data is for tryhards. Using a regex to grab the correct paragraph or div or whatever is fine and is more robust versus things moving around on the page.by rightbyte
- The title is confusing, should be "commented-out".by stevage
- Agree, I thought maybe this was going to be a script to block AI scrapers or something like that.by pimlottc
- It doesn't seem that abusive. I don't comment things out thinking "this will keep robots from reading this".by Noumenon72
- Crawlers ignoring robots.txt is abusive. That they then start scanning all docs for commented urls just adds to the pile of scummy behaviour.by michael1999
- The article mentions using this as a means of detecting bots, not as a complaint that it's abusive.
EDIT: I was chastised, here's the original text of my comment: Did you read the article or just the title? They aren't claiming it's abusive. They're saying it's a viable signal to detect and ban bots.
- Fun to see practical applications of interesting research[1]by sharkjacobs
- I'm not overly surprised, it's probably faster to search the text for http/https than parse the DOMby rokkamokka
- The regex approach is certainly easier to implement, but honestly static DOM parsing is pretty cheap, but quite fiddly to get right. You're probably gonna be limited by network congestion (or ephemeral ports) before you run out of CPU time doing this type of crawling.
- Not probably, searching through plaintext (which they seem to be doing) VS iterating on the DOM have vastly different amount of work behind them in terms of resources used and performance that "probably" is way underselling the difference :)
- Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities.
I know a thing or two about web scraping.
There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi).
Zip bombs are also not for me. Reading header content length is enough to not read the page/file. I provide byte limit to check if response is not too big for me. For other cases reading timeout is enough.
Oh, and did you know that requests timeout is not really timeout a timeout for page read? So server can spoonfeed you bytes, one after another, and there will be no timeout.
That is why I created my own crawling system to mitigate these problems, and have one consistent mean of running selenium.
https://github.com/rumca-js/crawler-buddy
Based on library
by renegat0x0 - Looks like it's time for in-browser scrappers. They will be indistinguishable from the servers side. With AI driver can pass even human tests.by Mars008
- Is there a difference between "scraping" and "crawling"