Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution.

    On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others.

    I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well.

    Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.

    by exiguus
  • I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
    by boznz
  • 410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!

    Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

    by binaryturtle
  • If you are unable to read this, there is an archived copy at https://archive.ph/d3236
    by CqtGLRGcukpy
  • You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP.

    Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.

    > One class of bots always claims to come from the root page of google. Obviously this is a lie.

    This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.

    by thayne
  • I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
    by fxtentacle
  • You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.

    Block on traffic volume and request frequency if that's causing a problem.

    (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)

    by userbinator
  • I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?

    WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.

    Are you using lambdas or something like that to serve a blog?

    Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?

    by AussieWog93

Explore Birbla archives