Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Unconscionable.by CivBase
- The authors are proud of facilitating the theft of intellectual property. Disgusting.by il-b
- Did you rage equally against the machine when wget --mirror came out?by phoghed
- Things like this make me wish that we have to pass ethics courses to work in tech.by xena
- Reminds me of this old joke: No ethically trained software engineer would ever write a destroyBaghdad function - they’d write a destroyCity function to which you could pass Baghdad as a parameter.by m12k
- Does Anubis still work? (against this?)by BLKNSLVR
- For agent/browser automation, getting blocked is only one part of the problem. The other hard part is knowing whether the page you got back is the real page, a degraded version, or some silent challenge page.by Avery29
- Is it possible to give negative stars to a GitHub repo? This one would be a prime candidate for that feature.by zombot
- > Bot detectors flag automation by reading the browser fingerprint; Fortress corrects that fingerprint inside Chromium's C++, so the browser presents as an ordinary Chrome install.
This does not seem like it would work against anything but the most basic bot protection.
by xnx - I think that instead of trying to prevent web scraping, websites should try to make it easier so that it generates less traffic. As long as any user is allowed to view the website, there will always be a way to scrape it anyway. If there were simply a monthly updated torrent available on a standardized subpage, such as example.com/scrape, scraping would be much less harmful.by RandomGerm4n
- This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.by dclaw
- Sites that are backed by a billion dollar company typically have better antibot protection, and lot of expertise in this.
Also, patched browsers have existed since long ago, although they were not open-source.
by codedokode - And those bot protection mechanisms and ad-enforcement layers that terrorize humans are okay or what? Yes, I accept that many pages don't want me there and just don't use them anymore, but it sucks.
I am not saying that forks like this are a good idea, but I have enough frustration with said techniques that I sympathize with the effort.
by arendtio - Making a network request from a script is not abuse, consuming excess bandwidth is. Scrapers already spoof browser reputation and cycle IPs while abusing bandwidth. Recently I wanted to convert the results of an Autotrader filter into a table so I could supplement it with data they don’t track (towing capacity). It was two pages of results, but requesting it from a script was aggressively blocked by browser fingerprinting. I had to port it to JS and run it in my browser console manually to get the data out, wasting my time.by deckar01
- Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.by TrevorFSmith
- It's most definitely unethical.by LoganDark
- I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.by cr125rider
- I suppose it's like trying to enforce no picture rule in public.by maxglute
- > Is it ethical to scrape when a site has explicitly blocked bots
Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.
by est - Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.by lemagedurage
- Man, this sucks. I doubt there’s anything that I can say to get people to stop doing things like this, but the eventual outcome here isn’t going to be freedom for you to scrape sites that are trying to avoid being DDoSed by bots, but instead that we all end up in a world where device attestation is required to do practically anything online. And for what?by csnover