Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.
The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
by FabCH - Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.by Bender
- and all valid traffic too, judging by these HN comments (and my own attempts to connect).
- The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.by Bender
- I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.by billfor
- load balancer / proxy, since you need a way to get the real ip
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
by Bender - In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution.
On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others.
I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well.
Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.
by exiguus - I agree whitelisting is great. I have used that in B2B communications by setting up network to network VPN's. Anyone not on the VPN can not reach the servers. Employees can use their corporate VPN to also access the servers.by Bender
- I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.by boznz
- Techno-Thriller/Sci-Fi/Mystery novels
Nice, I will have to take a look at some point.
by Bender - 410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!
Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
by binaryturtle - I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.by gruez
- Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.
People can see a few headers here [2]
[1] - https://caniuse.com/?search=sec-fetch-mode
[2] - https://nochan.net/.env
by Bender - If you are unable to read this, there is an archived copy at https://archive.ph/d3236by CqtGLRGcukpy
- I wonder why you'd be unable to read it? :-)by hdjrudni
- I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.by ACCount37
- Thanks. Regretful implementation*, being a normie on iOS Safari:
https://i.ibb.co/vCDH79d0/IMG-0303.png
…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)
*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
by Barbing - You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP.
Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.
> One class of bots always claims to come from the root page of google. Obviously this is a lie.
This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
by thayne - Author has indicated he's perfectly fine with blocking all sorts of human users. I wouldn't take any of his advice.by inigyou
- I blocked AWS incoming traffic a few years ago. I wrote a blog post about it. I get maybe dozens of actual visitors a week; I got around 15,000 real people views of the post in about three months. Amazing how quickly things like that drop below the memory horizon for people.
I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly.
As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case.
This is fun.
by m3047 - For what it's worth I block all of Google's ASN's entirely. Just me personally for my silly hobby and tinkering sites, I have not received any good traffic from them as of late. Even their search has gone to pot in my opinion.
I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake.
Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
by Bender - An unfortunately large amount of people unknowingly having their network resold for residential vpns too.
Blocking http1.1 is reasonable imo, its been over a decade since just about every browser has supported it. Anyone on a browser that old has most the mainstream web broken to them at this point, a broken (personal) site wouldnt be an exception to them, but the norm.
- I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.by fxtentacle
- I don't see why AWS would run fail2ban on IMDS. Do they not trust their implementation? Do they want lawsuits from their big customers?by inigyou
- When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.by Bender
- You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
by userbinator - The user agent string is harmful in general. Any new browser should just copy the chrome user agent.by Gigachad
- Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.by inigyou
- I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.
On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.
by Bender - I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
by AussieWog93