Re: [google-appengine] unusual traffic from your computer network

[Jeff Schnitzer]

If you've been reading about my troubles with this issue in the past,
you're going to laugh at my suggestion:

Use CloudFlare. CF's IP blocks are apparently whitelisted by Google
now and won't trip Google's alarms. You can disable CF's threat
monitoring and response system - and even better, you get metrics so
that you have some idea when/why it's being tripped when it is
enabled.

This seems like a silly way of routing around Google's undocumented
and unwanted "service", but it should get the job done.

Client -> Client's Proxy -> CF -> GAE

Jeff


On Mon, Apr 1, 2013 at 2:26 PM, Peter Warren <pe...@treehouselogic.com> wrote:
> I see posts about this issue going back years, so sorry if I'm kicking a
> dead horse, but I haven't been able to find any resolution.
>
> (I’ve posted this message twice on a new account, once 5 days ago and once 3
> days ago, and neither message has actually made it into the forum. So I’m
> trying my old account. Sorry if this post ends up getting duplicated.)
>
> We have a paid app on app engine we've been using to serve a commercial web
> app for 3 years. The app is mapped to a custom domain via Google Apps. I
> think that’s the crux here.
>
> We have one application that serves different content for different clients.
> Each of our clients has reverse proxy set up on their web server to fetch
> the content from our custom domain on app engine. We use reverse proxy
> simply to mask our domain to the clients' domains. There is no caching, and
> the reverse proxy is Apache2 with out of the box configuration.
>
> On March 26, after 2 years of happily serving content to a particular
> client's server, Google for some reason decided that this server was
> violating its Terms of Service and started denying content to that client's
> reverse proxy, redirecting users to the www.google.com/sorry/misc page with
> the message that: "Our systems have detected unusual traffic from your
> computer network." This of course caused our application to be totally
> unusable. We sent requests to Google for more information and heard nothing.
> The next day App Engine decided that particular server was ok again and
> resumed serving our content to the problem server.
>
> Then again on March 30 Google decided to ban this particular server.
>
> Our app is very low volume, averaging about .05 requests/second. There were
> no traffic spikes that day. There were no configuration changes to the
> reverse proxy or any of our infrastructure.
>
> The only information I can find on the issue is here:
> http://support.google.com/websearch/bin/answer.py?hl=en&answer=86640&rd=1.
>
> That page suggests that the client's server was doing one of these things:
>
> • Sending automated queries
> • Using software that sends queries to Google to determine how a website
> or webpage ranks on Google for various queries
> • 'Meta searching' Google
> • Performing 'offline' searches on Google
>
> I could find no evidence of any requests being sent to Google search. There
> were open requests to one of Google's nameservers, presumably to look up our
> app's ip from its Google Apps custom domain. Surely that isn't a violation
> of Terms of Service. We found no malware on the machine. So at this point we
> have no idea why Google stopped serving the content to that particular
> server, or why it resumed service. Additionally all our other clients'
> reverse proxies continued to work fine. There was even another reverse proxy
> successfully fetching the same content that Google was denying to the other
> proxy.
>
> Switching to the yyy.appspot.com domain from our custom domain seems to fix
> the problem, so I really suspect the problem is with the domain mapping.
>
> I sent a support request to Google Apps, and of course they said they
> couldn’t look into it, stating: “You are correct that the custom domain
> mapping is created in the Google Apps Control Panel and is handled there
> however any issues with the mapping of Google App Engine apps needs to be
> investigated and supported by the App Engine team.”
>
> So I’m left wondering why Google has denied requests from this particular
> server after 2 years when nothing has changed. And yet Google continues to
> happily serve our other clients who are using the exact same proxy settings
> on other machines.
>
> Searching through previous posts, the best information I can gather is that
> maybe our proxies headers are malformed and Google doesn't like them. Why
> would Google randomly complain after 2 years of happily serving content to
> this same proxy with the same headers?
>
> Previous posts described this problem as a landmine, where stepping in the
> wrong place can trigger it. Seems more like a surprise missile attack to me
> because we were simply walking the same path we'd walked every day for 2
> years when everything blew up.
>
> Obviously this is totally unacceptable. We can't very well offer a
> commercial service to clients with the caveat that it might blow up at any
> time, and we have no idea when or why.
>
> I also don't understand the connection between Google Search's Terms of
> Service and my paid App Engine app. Why does Google deny service to my paid
> application when it thinks some machine is violating its search policies???
> Even if that machine were violating its search policies, if I want to serve
> content to the violating machine from my totally un-Google-search-related
> web app, I should be able to. Granted a DOS situation could be a valid
> reason for denying service to my app engine app, but violating Google's
> search policies is totally unrelated to my app engine app, and I should be
> able to serve content from my paid application to whomever I want.
>
> Can Google or anyone here on the forum shed some light on why this might
> have happened and what I can do to prevent it? Will turning on PageSpeed
> help, since requests will presumably be served mainly by edge caches and not
> hit the app engine app as frequently?
>
> This issue has been around for years and clearly is still a huge problem.
> I'm sure many app engine users would love some transparency on this issue.
>
> Thanks for any help,
> Peter
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at http://groups.google.com/group/google-appengine?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Nacho Coloma]

Funny, because in the past CloudFlare was getting banned quickly for unusually high traffic. I suppose they have been whitelisted since.