Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • > Sorry, you have been blocked You are unable to access patronview.com Why have I been blocked?

    > This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

    Thank you, from Vietnam.

    by landgenoot
  • I see popular blog posts, click it and see its blocked from India.
    by aitchnyu
  • > There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.

    Kinda sounds like we're missing a peer to peer network here.

    Instead of downloading the same data over and over again we can just download it once and then share it.

    Wouldn't that be better for everyone involved?

    It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)

    by andai
  • The problem there is trust
    by amazingamazing
  • Have you checked out IPFS?
    by potamic
  • Common Crawl?

    https://commoncrawl.org/

    by NicuCalcea
  • Anyway, Google doesn't send traffic to your site any more. Only important sites and obvious scams seem to get indexed.
    by inigyou
  • I made a small booking site for a local dutch canal boat, which has a calender function. A simple PHP app. I checked the Apache logs recently, and it had THOUSANDS of claudebot and other AI UserAgents flooding the logs every day, apparently because the scrapers keep hitting the 'next month' button on the calendar in a an infinite loop, all day, everyday!

    This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.

    Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.

    by rsolva
  • It's vibecoders all the way down. I'm sure Anthropic et al. would like to reduce their resource waste if they could, seeing the insane amounts of money they're bleeding. But I had to come to the conclusion that these LLM companies have not a single developer good enough to implement a decent scraper. Their tools produce garbage code and they don't know enough about programming to realise it, or do anything about it.
    by Anamon
  • I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.

    It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.

    However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).

    by mgbmtl
  • My solution was similar. Anything coming from the ASN of a major cloud provider gets a CAPTCHA with a little nuance to allow Google and Bing to index. That seems to do a pretty good job. Also, anything coming out of China or Singapore also gets a CAPTCHA as my site is not popular in those regions and many Chinese bots seem to show up as a Chinese mobile provider. So far, the bots have never attempted to solve the CAPTCHA.
    by IMSAI8080
  • This will resonate with anyone who operates a public-facing website and doesn't work at a big tech company. In the past few years I have also been down this exact same road. I'm desperately trying to avoid resorting to Cloudflare, but I'm running out of time and patience to keep tweaking nginx and firewall rules every few weeks.

    I had initial success with https://git.gammaspectra.live/git/go-away as a more powerful and more reasonable self-hosted alternative to Cloudflare than Anubis, but it seems to have gone unmaintained. There is also https://github.com/dgl/haphash if you run HAProxy, though I have not tried it.

    On that note, does anyone know what happened to Ted Unangst aka tedu? He was a prolific OpenBSD developer and blogger, and he had developed one of his own simple solutions https://humungus.tedunangst.com/r/anticrawl, but all his web properties seem to have gone away recently...?

    by aorth
  • I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
    by ddxv
  • But what do these bots gain from this?
    by hskalin
  • I shutdown all my little informational hobby projects this year that I've tinkered with for decades

    They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes

    I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots

    by fooey
  • Can someone help me understand the underlying motivation behind this?

    It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.

    What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?

    by varenc
  • The most recent data on the internet for advertising, intelligence, etc.

    And a lot of bad scrapers.

    by esseph
  • My laymans understanding is that poorly written scripts get executed and then owner comes back occasionally to check that there is 'content' in their database. They're not sitting there troubleshooting their thing beyond 'it's getting data' because, for similar reasons to their scripts being poorly written, their strategy is 'get data'.

    Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.

    by BLKNSLVR
  • Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it.

    It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.

    by phillmv
  • They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.
    by bigbuppo
  • > And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
    by qbane
  • Live by the scraper, die by the scraper
    by alansaber
  • Who scrapes the scrapemen?
    by ethersteeds
  • Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
    by aeturnum
  • There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
    by ihuman
  • Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.

    [1] https://anubis.techaro.lol/

    by johnorourke
  • Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.

      pow_server  http://127.0.0.1:8080   backend avx512-x16
  ──────────────────────────────────────────────────────────
  uptime   00:03:12
  solver   ● BUSY  difficulty 9, 0.3s
  queue    [####################............] 5/8   peak 12
  ──────────────────────────────────────────────────────────
  accepted 1240        solved 1180
  503 shed 48      504 timeout 2      4xx/5xx 10
  ──────────────────────────────────────────────────────────
  last     difficulty 5  nonce 645376  in 9 ms  (101.6MH/s, avx512-x16)
  hashes   3.90GH total   avg 65.3MH/s
  Ctrl-C to stop
    Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.
    by drum55
  • make your only legit users mine fake crypto to access your site, only costs them 5% battery on an android device
    by czk
  • Anubis sucks because CPU is cheap for scrapers and hard for humans.
    by wbl
  • I'm assuming a bot running a headless browser instance can still get past it?

    It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.

    by marklar423