App Engine DoS protection ("unusual traffic" error) and reverse proxies

[Alexander Konovalenko]

Dear App Engine developers,

I think it should be easier to debug App Engine's automatic denial of
service protection (that's the thing that redirects your users to an
error page at sorry.google.com when it detects suspicious activity).
That's really important when many requests to your app intentionally
come from the same IP address (a reverse proxy you run).

Detailed use cases and feature requests are here:
http://code.google.com/p/googleappengine/issues/detail?id=5239
Please star that issue or comment on it if you're interested.

Cc'd to Matthew Blain and Sean Lynch from Google who have participated
in some discussions of the "unusual traffic" errors. Bcc'd to several
App Engine developers who have suffered from those errors.

Thanks.

-- Alexander

[Alexander Konovalenko]

I wrote:
> Cc'd to Matthew Blain and Sean Lynch from Google who have participated
> in some discussions of the "unusual traffic" errors.

Oh, Sean Lynch no longer works at Google. Sorry for the confusion.

-- Alexander

[Martino A. Sabia]

Great!

I've starred your issue on the Appengine issue-tracker, let's hope to have some response ;).

I'll put a link here of our previous discussion just for the records:

https://groups.google.com/forum/#!topic/google-appengine/4D1IGqCh4LA

Thank you,

Martino.

[Brandon Wirtz]

The problem only exists for ONE company.   I use Akamai DSA with out issue with AppEngine, and I have several setups using a Squid in front.   Their problem not Goog’s.  The problem is that CloudFlare “Double Bags” and messes with headers. And then falls out of compliance.

 

Sorry if I’m cold hearted, but -1 for “help my free app run even more free with a free RCP” doesn’t seem like a priority.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/21jXmXNcUycJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Alexander Konovalenko]

Brandon Wirtz <dra...@digerat.com> wrote:
> The problem only exists for ONE company.   I use Akamai DSA with out issue
> with AppEngine, and I have several setups using a Squid in front.   Their
> problem not Goog’s.  The problem is that CloudFlare “Double Bags” and messes
> with headers. And then falls out of compliance.

I see. In fact, I don't use CloudFlare and have no plans to use it.
I'm concerned with the possibility that when my nginx reverse proxies
get suddenly banned by some overzealous DoS protection mechanism, it
will cause many hours of downtime before someone from Google can
manually fix it. That's why I'm requesting the whitelist feature.
Problems like that happen periodically. False positives are hard to
avoid completely when trying to draw the fine line between legitimate
and malicious traffic.

> Sorry if I’m cold hearted, but -1 for “help my free app run even more free
> with a free RCP” doesn’t seem like a priority.

Honestly tried to understand this part but failed. What does RCP stand
for? Rich client platform? Perhaps the big blue font makes me more
stupid. Also, I'm not sure why free apps are an issue at all, because
originally I thought mainly of paid App Engine apps. If someone is
paying to host their reverse proxies somewhere, they definitely can
afford some $9/month for App Engine.

-- Alexander

[Brandon Wirtz]

I don't think you will have the problem if your Proxy passes the appropriate
headers. Also, I don't remember where it is, but there is a Google
WhiteList for Proxies not tied to AppEngine that is used for Schools and
large businesses who get blocked on the Google Homepage for too many users
for one IP.

Have you had the problem in the past?

RCP = Reverse Caching Proxy

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of Alexander
Konovalenko
Sent: Thursday, June 23, 2011 12:35 PM
To: google-a...@googlegroups.com
Cc: Matthew Blain; Brandon Wirtz
Subject: [google-appengine] Re: App Engine DoS protection ("unusual traffic"
error) and reverse proxies

-- Alexander

--

You received this message because you are subscribed to the Google Groups
"Google App Engine" group.

[Francois MASUREL]

In fact, CloudFlare was the only "simple" solution I found to make my site reachable from the P.R.C.

But I'm open to any other solution that doesn't cost an arm.

Francois 

[Brandon Wirtz]

Oh, so you are using it to violate the GAE TOS.  Even more reason to fix it.

[Francois MASUREL]

Brandon,

Could you precise in what way I am violating the GAE TOS by using CloudFlare ?

I really don't want to be in some kind of illegal situation.

Thanx for your help.

Francois

[Martino A. Sabia]

Brandon,

are your statements about CloudFlare based on your sole opinion or is it supported officially by Google somehow? 

In that case why don't you share with us your sources so we can delete definitively our accounts from CloudFlare without any doubt that the problem we're talking about is their and only their fault? 

What i found "interesting" is that nobody from google is saying a word about this and even CloudFlare people (who was involved in our previous discussion) are silent. For now i've deactivated my account on their service 'cause to much strange things happened in this period on my website. So even if I don't have any evidence on what happened I preferred to remove anything that can 'cause problems, even because no solutions were proposed.

Martino.

[Brandon Wirtz]

I’m not a Google employee, but putting a service in place to bypass Geographic access restrictions seems like a violation of TOS.

[Philip]

Brandon, I doubt you are an expert regarding US software export laws.
App Engine TOS say: "Your use of the Service must comply with all
applicable laws, regulations and ordinances, including any laws
regarding the export of data or software."

I am not a expert either but I think if Francois "exports" his service
to the cloudflare servers he does not violate any export laws. What
happens at the cloudflare servers is something that has no effect to
App Engine TOS.

On Jun 23, 10:23 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
> I'm not a Google employee, but putting a service in place to bypass
> Geographic access restrictions seems like a violation of TOS.
>
> From: google-a...@googlegroups.com
> [mailto:google-a...@googlegroups.com] On Behalf Of Francois MASUREL
> Sent: Thursday, June 23, 2011 1:10 PM
> To: google-a...@googlegroups.com
> Subject: Re: [google-appengine] R: Re: App Engine DoS protection ("unusual
> traffic" error) and reverse proxies
>
> Brandon,
>
> Could you precise in what way I am violating the GAE TOS by using CloudFlare
> ?
>
> I really don't want to be in some kind of illegal situation.
>
> Thanx for your help.
>
> Francois
>

> On Thu, Jun 23, 2011 at 22:05, Brandon Wirtz <drak...@digerat.com> wrote:
>
> Oh, so you are using it to violate the GAE TOS.  Even more reason to fix it.
>
> From: google-a...@googlegroups.com
> [mailto:google-a...@googlegroups.com] On Behalf Of Francois MASUREL
> Sent: Thursday, June 23, 2011 12:47 PM
> To: google-a...@googlegroups.com
> Subject: Re: [google-appengine] R: Re: App Engine DoS protection ("unusual
> traffic" error) and reverse proxies
>
> In fact, CloudFlare was the only "simple" solution I found to make my site
> reachable from the P.R.C.
>
> But I'm open to any other solution that doesn't cost an arm.
>
> Francois
>

> On Thu, Jun 23, 2011 at 21:03, Brandon Wirtz <drak...@digerat.com> wrote:
>
> The problem only exists for ONE company.   I use Akamai DSA with out issue
> with AppEngine, and I have several setups using a Squid in front.   Their
> problem not Goog's.  The problem is that CloudFlare "Double Bags" and messes
> with headers. And then falls out of compliance.
>
> Sorry if I'm cold hearted, but -1 for "help my free app run even more free
> with a free RCP" doesn't seem like a priority.
>
> From: google-a...@googlegroups.com
> [mailto:google-a...@googlegroups.com] On Behalf Of Martino A. Sabia
> Sent: Thursday, June 23, 2011 4:16 AM
> To: google-a...@googlegroups.com
> Cc: Matthew Blain
> Subject: [google-appengine] R: Re: App Engine DoS protection ("unusual
> traffic" error) and reverse proxies
>
> Great!
>
> I've starred your issue on the Appengine issue-tracker, let's hope to have
> some response ;).
>
> I'll put a link here of our previous discussion just for the records:
>
> https://groups.google.com/forum/#!topic/google-appengine/4D1IGqCh4LA
>
> Thank you,
>
> Martino.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visithttps://groups.google.com/d/msg/google-appengine/-/21jXmXNcUycJ.

> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com

> <mailto:google-appengine%2Bunsu...@googlegroups.com> .
> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com

> <mailto:google-appengine%2Bunsu...@googlegroups.com> .
> For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

[Brandon Wirtz]

I can’t speak for Google.  I enjoy a very good relationship with some of the people in the various teams at Google, and when problems arise for big clients I get answers. 

 

If you need validation Holding a big sign outside the Schwerma Place just off campus in Mt. View that reads “Will Trade Lamb for Answers On External Rate Limiting” would probably result in an off the record response from someone on the team I know who eats there most Wednesday’s.     Beyond that, I don’t have a resource I can site, and I don’t give contact info for people at Goog for fear they will stop answering my own questions.

 

I encourage you to go to meetups, and events where you can get off the record advice, and knowing nods of affirmation about issues you suspect you know the answer to but need confirmation.  Google People are People, and if you want answers being nice to them, and looking them in the eye helps…. Or spend lots of money in their group that works too…

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/RbKXQoyIu9wJ.

[Francois MASUREL]

I don't think Google restricts access from PRC but it's rather the opposite in fact :-)

[Martino A. Sabia]

Brandon,

hahah it's a nice joke, but i thought that Internet was to speak to people even you're geographically distant. If i can afford an intern-continental flight to go speak with some google employee i surely don't even have the problem to afford a non-free caching service, don't you think.

By the way, besides kidding, and don't think that there is no way to have some official response from google for a service like AppEngine. Yeah i can subscribe to the new 500$/month appengine account to speak with someone...

Martino.

[damoncloudflare]

Using CloudFlare wouldn't run afoul  of any TOS with Google Apps (our IPs have been whitelisted with Google). If there's some sort of issue we need to address with our contacts at Google, or if you are experiencing some sort of issue with Google Apps while using CloudFlare, please contact us with the details here: https://www.cloudflare.com/contact.html. We're more than happy to work on any issues with Google.

[Ikai Lan (Google)]

There are lots of legitimate uses for proxies. Using a reverse proxy is not a violation of the terms of service. For instance: you may work for a company that requires use of a proxy for outbound internet access. Generally speaking, using App Engine from your corporate network would not be a violation of the terms of service (but you may get into trouble with your own corporate compliance if you are doing something you're not supposed to be doing).

I cannot interpret the terms of service for you, however, the bit that is being discussed is this:

"2.2. Your use of the Service must comply with all applicable laws, regulations and ordinances, including any laws regarding the export of data or software. You agree not to use the Service in the design, development, production, or use of missiles or the design, development, production, stockpiling, or use of chemical or biological weapons. You agree not to use the XMPP API to operate or to enable any telecommunications service or in connection with any applications that allow users to place calls to or receive calls from any public switched telephone network."

If you believe your application may depend on a reverse proxy (or ANY technology, for that matter) to circumvent a data export law, you should speak to someone who can interpret the laws and the terms of service for you (not me).

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

On Fri, Jun 24, 2011 at 5:17 AM, damoncloudflare <dbil...@gmail.com> wrote:

Using CloudFlare wouldn't run afoul  of any TOS with Google Apps (our IPs have been whitelisted with Google). If there's some sort of issue we need to address with our contacts at Google, or if you are experiencing some sort of issue with Google Apps while using CloudFlare, please contact us with the details here: https://www.cloudflare.com/contact.html. We're more than happy to work on any issues with Google.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/I-FzH3l7d7MJ.