Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Which website platform are you using and hosting provider to support your need?by Era24UK
- Assuming your site has no dependency on Amazon EC2:
Make sure that gives you CIDR blocks, then add it to a script and use a for loop tofetch_aws() { curl -s \ -H "Accept-Charset: UTF-8" \ -H "Accept-Encoding: gzip, deflate" \ -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \ -H "Accept-Language: en" \ -H "User-Agent: Mozilla/5.0 (Windows NT 11.1; rv:102.0; ) Gecko/20100101" \ -H "Cache-Control: 0" \ -H "Host: ip-ranges.amazonaws.com" \ -H "Connection: keep-alive" \ --referer https://ec2.amazon.com/ \ -o /dev/shm/aws.json \ --url "https://ip-ranges.amazonaws.com/ip-ranges.json" grep ip_prefix /dev/shm/aws.json | awk -F "\"" '{print $4}' | sort -n | uniq rm -f /dev/shm/aws.json; };
Google and Cloudflare if you need them:for AWS in $(fetch_aws);do ip route add blackhole "${AWS}" 2>/dev/null
This assumes you have no need to connect to or get connections from Amazon on your web server. If your server is an instance in Amazon that will break DNS resolution unless you are using something outside of AWS for DNS. The gateway should still work just fine. Obviously test from an out of band console if that is an option. This is easier to maintain if you remove DNS records for IPv6 and eventually disable IPv6 listeners.get_google() { for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :) do dig +short txt "${line}" done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq } get_cloudflare() { curl -A Mozilla "https://api.cloudflare.com/local-ip-ranges.csv"|grep -Ev "::|/32"|awk -F "," '{print $1}'|sort | uniq }If you paste a couple of lines of the bots from your access logs I can offer more suggestions in the event they try from outside of Amazon.
If you want to have some fun, add a hidden link only the bot will see that points to http://cpanel.yourdomain.tld/ after adding a DNS record for cpanel that points to 169.254.169.254 so they start scraping the AWS cloud-init IP.
by Bender - Ah, if its only Amazon, then you are in a good position :-))
I hosted a small website with some newfeeds and it got killed by all the AI scrapers in the end.
- I haven't personally seen it, but I think us operators need to take entire registered ranges and hook 'em up to fail2ban. Maybe maintain lists of offenders and document it and block these orgs.
That is, I haven't personally seen a bad actors list that gets used in a fail2ban-like setup.
by bellowsgulch - It's not abnormal for people to disrespect robots.txt
Meta not only ignored my robots.txt but they hit me with 1k requests per second sometimes.
BingBot is doing something weird. Based on the logs I see, it feels like their crawler is trying to reverse engineer my public facing APIs and scrape those directly.
by leros - Ban Singapore that solved my Amazon traffic issues and probably others.by ipaddr
- It is not just Amazon (although they're #4 on my list of bad actors)[1].by schappim
- Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their crawler blocker.
You could go down the rabbit hole and try to find someone at Amazon to tell their crawler to chill. Don’t expect success but would make a fun blog post.
https://www.cloudflare.com/learning/ai/how-to-block-ai-crawl...
by toomuchtodo