Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://developers.google.com/crawling/docs/crawlers-fetcher...

    Seems like the crawler companies would be incentivized to not want to take responsibility for people spoofing their user agents.

    by wilg
  • Yup, in fact most of them are already. That's one of the ways this data is verifying whether the visits are spoofed or not: https://knownagents.com/insights#spoofing-and-security
    by gavinhking
  • Just in case any of the authors read HN, I'm getting a pretty crazy rendering bug on this page, where a bunch of the contents are redrawing up and down by a few pixels. It seemed to go away with resizing the width a few times, but I didn't look into it too hard. My page width was probably small on first draw. Incredibly distracting though and hard to read with the text moving. Using latest chrome, and it occurred on more than one page refresh. I didn't dig in beyond that though.
    by kevin_nisbet
  • I'm seeing the same thing on Firefox on Linux. It almost looks like the page scroll is jiggling up and down a tiny amount constantly when it's supposed to be stationary.
    by saghm
  • Interesting thought: what if the idea of an open internet is over.

    What if we're now moving into a world of strictly KYC. The same way "The Facebook" generated massive revenue by creating a KYC world.

    by blobbers
  • Fake Googlebot visits are #1 in website logs I've been working on. At the beginning I was fighting with them using Cloudflare ASN block rules or their managed Bot Fight mode but it appeared to be not only pointless, but also harmful for my websites. Bot Fight mode randomly started blocking real Bing / Google / OpenAI crawlers what wasted crawling budget and discouraged crawlers to revisit updated pages.

    Sometimes it's better to not fight with bots actively but harden environment and only react for the worst offenders.

    by ChillyCapy
  • With Cloudflare you can set a rule to block traffic that identifies as Googlebot but is not a "verified bot", ie is not from the proper IP range.
    by Jskewel
  • For Google it's pretty straight forward to throw away fake crawlers by just only allowing their published list of crawler IPs so you don't accidentally allow someone from a random GCP IP to crawl you if unwanted (https://developers.google.com/crawling/docs/crawlers-fetcher...).
    by dewey
  • Very similar experience here. Started July 30, sustained through August 6, when it started a significant ramp-up in volume (5x or so).

    Most of the traffic is originating in GCP. We're seeing ~70k req/min sustained from Google Cloud IP space (AS396982). Reported to GCP Abuse, they've been non-responsive so far.

    The main distinguishing factor is the reuse of a bunch of legit AI-training bot UserAgent strings. It's clear that the traffic is under the same centralized control because of how it changes volume across thousands of IP addresses simultaneously.

    by oasisbob
  • If how they’ve handled Gmail abuse is any indicator, they’re not likely to do anything. They’re still getting paid for the server time
    by mcmcmc
  • Seems like you’re part of the group represented in this dataset trend then, many of these visits are also from (compromised) Google servers in that same ASN.
    by gavinhking
  • I recently blogged about some Cloudflare Workers I developed to combat this type of traffic: https://code.backwater.systems/blog/#2026-06-29T23:40:00.000...
    by nate-gehringer
  • Using a normal page per blog entry would go a long way to making your site more indexable, readable, shareable and seo-able. (Good article btw).
    by sethops1
  • Worth saying out loud: user-agent is not identity. Verify AI crawlers by reverse DNS or the provider's published IP ranges - the ones worth letting in all publish them.
    by tomveber
  • Why would you voluntarily pretend to be a AI bot, when those have already a much higher chance of being blocked? Seems holly unproductive.

    Best hypothesis I can come up with is to somehow make the AI companies look bad, but they seem to be doing an excellent job at that themselves already by scraping everyone hundreds of times per hour over and over.

    by Tharre
  • Googlebot and other web scrapers are how you get visible, how you get adwords, ways to boost your traffic, etc.

    bots hitting your site aren't problems per se

    by red-iron-pine
  • Most websites don't have an incentive to block AI bots to their main sites. Think businesses, government and community websites, nonprofits, etc.
    by xgulfie
  • Because businesses dont want them blocked, that would be a very stupid thing for most of them to do given its becoming a vital traffic source now that people are using chatbots instead of google.
    by esskay
  • Mass automated vulnerability scans have been a very common thing since years before the advent of this in 2001:

    https://en.wikipedia.org/wiki/Code_Red_(computer_worm)

    I remember when 'code red' spread and it had the effect of crapping up the contents of my apache server logs. Fun times.

    such as:

    GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0

    by walrus01
  • Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.
    by Bender
  • Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
    by gavinhking
  • Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
    by codegeek
  • Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not.

    From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.

    Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.

    by bflesch
  • I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world.

    We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.

    Here's a small snapshot of the dashboard:

    https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358...

    This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.

    I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.

    by cullenking
  • On average about 100 (TCP) requests hit my home router per minute doing various probing and scanning. Lots of checking for the telnet port obviously. Sometimes you can see a swarm of entirely different IPs scanning the full port range (probing the ports one-by-one).

    You'll see a lot of deepfield, censys-scanner, visionheight.com, shadowserver.io, etc., but also the usual suspects of Chinese or Russian IPs.

    With OpenWRT I use something like this: `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'`, or alternatively `tcpdump -i pppoe-wan 'inbound and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and not port 44000'`, if we have some torrent client running (e.g. here at port 44000) which would mess up the result. I'm not sure it's the best way to handle this, but it's definitely enlightening what bounces off on the router.

    by binaryturtle
  • fail2ban?
    by hahahaa
  • This is one reason I don’t mind that I’m behind CGNAT.
    by heywire
  • I can corroborate visionheight and shadowserver from my firewall logs.
    by impish9208
  • I have ubiquity UniFi for this reason (amongst others). OpenWRT is a good choice as well. Most home router software is such junk, might as well leave the door open..
    by sroussey
  • The easiest way to deal with the usual suspects is to just block the entire countries network range(s). There really is no reason they should be connecting to your home router anyway, and you lose nothing from blocking them.

    Sure their packets will still hit your router, but if they are dropped immediately at least you're not wasting a syn-ack on them.

    by miah_