dos.yaml -- does google use this information for anything else?

[James Gilliam]

I am using the dos.yaml file to block ip addresses which are going to the same url repeatedly, like 15 times per sec.  It works well. After a day or so most of the ipaddrs will stop the abusive conduct but new ones emerge regularly.  I assume they are trying to bring down my site but it has little effect.  Grateful gae has dos.yaml.

Question -- when I put an ip addr in the dos.yaml to block their access, does google do anything with this information for their own purposes?

Thanks

[Barry Hunter]

Question -- when I put an ip addr in the dos.yaml to block their access, does google do anything with this information for their own purposes?

Like what? 

What use could they make of it?

They have no idea why you've added an address to the list. The description is optional. The block maybe nothing related to a real DOS. Maybe using to block a competitor from viewing your website, or have a sensor net feeding data to appengine for storage, and one of the sensors has malfunctioned, so you just block it to keep its records out. Or maybe you just using it block a crawler you dont like, but is acceptable to everyone else. 

In short the data is meaningless without context of why something is listed. 

[James Gilliam]

1. The data is hardly meaningless.  One app reporting an abusive ip address has limited value, but what if 100 apps do, or 1000.

2. The report of a abusive ip addr is not in isolation ... they count the number of requests also.  The cases I speak about are when one ip address, access the same url, as many as 15 times a second for hours at a time.

[Vinny P]

Hello James,

I'm going to agree with Barry here.

On Wednesday, April 24, 2013 10:09:52 AM UTC-5, James Gilliam wrote:

1. The data is hardly meaningless.  One app reporting an abusive ip address has limited value, but what if 100 apps do, or 1000.

But there is no metadata for why that IP is blocked. That IP could be malicious, or it could be a misconfigured system, scraper, etc. Just because you have reason to block that IP, or several other people do, doesn't mean that the IP isn't also sending out legitimate traffic.

Also Google has its own anti-DOS systems, which seem to be perfectly capable of blocking bad traffic; if you search these forums you'll see a few instance of the anti-DOS system mistakenly blocking Cloudflare traffic.

On Wednesday, April 24, 2013 10:09:52 AM UTC-5, James Gilliam wrote:

2. The report of a abusive ip addr is not in isolation ... they count the number of requests also.  The cases I speak about are when one ip address, access the same url, as many as 15 times a second for hours at a time.

The number of requests coming from an IP address can be misleading. For instance, MIT has a whole /8 to itself, which is far more than enough to give every computer on its network an IPv4 address (an /8 block is 16 million + IP addresses). I did my undergrad at U of Wisconsin @ Madison, and I know for a fact that there were labs with ~100 computers on them which shared a single IP. There are many companies and organizations that do the same. Do you really want to block such an organization just because some dumbass accidentally left his script running? The PR storm alone would be ugly. (as an aside, there were quite a few undergrads at UWM that weren't - to put it mildly - the sharpest tools in the shed. I could easily see them making such a mistake). 

Also, 15 times a second for hours is not very much. A HTTP client that doesn't support HTTP pipelining ( http://en.wikipedia.org/wiki/HTTP_pipelining ) can easily do far more than that, especially on a rich web page. Now multiply that by many different users.. RSS feeds for important news and financial services can have many clients hammering away at it. There's an RSS feed for Google Hot Trends: I wouldn't be surprised if it gets hundreds of hits a second.

The TL;DR of it is, there's just no reason to take your dos.yaml list. Google has its own anti-DOS systems already.

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

My Go side project: http://invalidmail.com/

 

[James Gilliam]

You probably run malicious bots, and even bad bots probably have some valid traffic. But i don't care if they are bad. I get alerts when an IP address abuses and then i block.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/zb-73YV3o-g/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.