Firewalling AWS

[Joshua Smith]

I have a small site that I run on a not-for-profit basis.

Periodically I need to update robots.txt or add firewall rules to shut down bad actors who beat the crap out of the site running up my instance costs.

Lately, I've been getting slammed by instances running on AWS. They are mostly making HEAD requests, which makes me think it's some kind of crawler, but it uses regular browser user agents and doesn't respect my robots rules.

There's no legitimate reason for AWS to browse my site, so I just add a firewall rule, right? Trouble is, AWS has 6,462 different IPV4 address ranges, and this crawler is constantly jumping between them.

Any costs that I have are paid out of my own pocket. So I'm looking for suggestions that don't require MORE subscriptions (like CloudFlare or something).

Any ideas?

-Joshua

[George (Cloud Platform Support)]

Hello Joshua, 

You have done the right thing with robots.txt. App Engine firewall offers these 3 options: Allow only traffic from within a specific network, Allow only traffic from a specific service, and Block abusive IP addresses. You have considered these in your post, so there is no other option, except based on eventual other subscriptions.