Flooded with traffic, and it appears to be local IP addresses... ?

[Clark Van Oyen]

Hi everyone. I'm stumped by something I'm seeing happen to my app on Google App Engine - tried asking on Stack Overflow with the App Engine tag but my question wasn't received well because it's not a programming question. I'd be very happy if anyone can shed some light on this though :)

I'm receiving a huge number of requests from IP addresses starting with 10. Afaik, this is a local IP. So, my question is why would Google's network be swarming my app with requests? It's costing me quite a few bucks in extra quota, so is cause for concern. I've attached the log I'm looking at. Thanks in advance if anyone has any idea what's going on!

[Stuart Langley]

Without the app_id it's hard to take a peek at the logs and try and deduce what it might be.

[Clark Van Oyen]

Thanks for your replies. In my panic I seemed to miss some obvious evidence, and only had to look at the user agent to see that the requests are coming from Google's own crawlers. Adding the url to robots.txt seems to put an end the local requests. I'm still getting some from "Feedfetcher-Google", apparently an RSS reader of Google's that ignores robots.txt as it's used by humans. I've started responding to these requests with a 410 as suggested in Google's docs for Feedfetcher. Here's the log for one of them:


66.249.92.99 - - [03/Feb/2015:12:39:42 -0800] "GET /rss?page=http%3A%2F%2Fwww.torrentman.net/commnet.html&opts=8498&num=20&mod=%241%24wq1rdBcg%24Z17SROlxqlo6.pL8QVJi0.&v=1421906055999 HTTP/1.1" 200 344 - "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=15992684286155802798)" "www.htmlcommentbox.com" ms=149 cpu_ms=21 cpm_usd=0.000038 app_engine_release=1.9.17 instance=00c61b117c6945bbc0f8c36160688fcec149d2

[Clark Van Oyen]

I forgot to mention, htmlcommentbox.com is actually the site I'm running! It's a comment service (like Disqus) that started around the same time but was just a small side project of mine to try out GAE.

On Monday, 2 February 2015 12:05:04 UTC-8, Clark Van Oyen wrote: