Google blocks traffic to appengine application

[Martino A. Sabia]

Hi,

I've a website on Google Appengine and a custom domain via Google App for domain. For the first time today i, and other users of my website, got an error message that prevents me to get access to my website:

"Our systems have detected unusual traffic from your computer network. 
Please try your request again later. Why did this happen?... "

I've tried to access the website through different browsers and different connections (even with my iphone on 3g network) - so with different ip addresses - but i get the same message anytime.

I can access my website through the appspot domain, but not via the domain (and my website traffic is dropped dramatically), so I think that this happens for all my site traffic users. I even use CloudFlare to cache my website, but they say that there have no issues with AppEngine. 

Someone have some clues to share? Please help :O

Martino.

[Brandon Wirtz]

CloudFlare Lied.  They are your problem.   But nice of them to Blame GAE.  Turn Cloud Flare off it will work just fine. 

If you need a Cache for GAE check www.cdninabox.com (this is my project) I apologize for the spam, but I only do it when caching comes up.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Wk12WXZPSm5GdzhK.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Martino A. Sabia]

Problem disappeared without any change made either on my site or in CloudFlare, so I don't think it depends on their service. 

It would be great to know by Google what happened, since the message cames from them, but i suppose it will not happen :(. My only idea is that it has something to do with Ipv6 day, but i have no evidences to prove this.

Brandon, I don't know if CloudFlare lied to me, I don't understand why they had too, but i have appreciated the fact they even responded to me even if i have only a "free" account on their service... enough said? 

The great trouble in all this is, in my point of view, that there is no way to have some kind of response from google even if i'm an early adopter of appengine and pay for their service. That scared me a lot.

Martino.

[Brandon Wirtz]

The issue is that cloudflare uses headers that don identify the ip adress they are making the request on behalf of so gae sees them as a dos attack when their round robin requests don't round robin enough.

this will happen a lot when bing bot or google bot comes though to index your site.

Also if you have a site with more than 5000 pages you will find cloud flare often results in the cloudflare captcha being indexed.

Sent from my Samsung Epic™ 4G

>--
>You received this message because you are subscribed to the Google Groups "Google App Engine" group.

>To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/I0gte2ag83QJ.

[Brandon Wirtz]

Chris at cloudflare responded to me offlist that I am wrong. I am on my phone at hpdisover in vegas and can't forward his reply easily. If he wants to respond on list it would allow public discussion of why this issue cropped up.

Chris may be correct that ip headers forward, but that wasn't my experience. I think their proxies aren't always configured the way they think they are and are often double bagged

Sent from my Samsung Epic™ 4G

[cdata]

I apologize, I hit the wrong reply button. Here is my message in full:

Actually, we use the industry standard X-Forwarded-For header to relay
the original IP address. This is used by most reverse proxies to allow
these issues to be solved in a practical and transparent manner.

Chris
Engineer, CloudFlare

On Jun 8, 8:09 pm, Brandon Wirtz <drak...@digerat.com> wrote:
> Chris at cloudflare responded to me offlist that I am wrong. I am on my phone at hpdisover in vegas and can't forward his reply easily. If he wants to respond on list it would allow public discussion of why this issue cropped up.
>
> Chris may be correct that ip headers forward, but that wasn't my experience.  I think their proxies aren't always configured the way they think they are and are often double bagged
>
> Sent from my Samsung Epic™ 4G
>
>
>
>
>
>
>
> Brandon Wirtz <drak...@digerat.com> wrote:
> >The issue is that cloudflare uses headers that don identify the ip adress they are making the request on behalf of so gae sees them as a dos attack when their round robin requests don't round robin enough.  
>
> >this will happen a lot when bing bot or google bot comes though to index your site.
>
> >Also if you have a site with more than 5000 pages you will find cloud flare often results in the cloudflare captcha being indexed.
>
> >Sent from my Samsung Epic™ 4G
>
> >"Martino A. Sabia" <ezu...@gmail.com> wrote:
>
> >>Problem disappeared without any change made either on my site or in
> >>CloudFlare, so I don't think it depends on their service.
>
> >>It would be great to know by Google what happened, since the message cames
> >>from them, but i suppose it will not happen :(. My only idea is that it has
> >>something to do with Ipv6 day, but i have no evidences to prove this.
>
> >>Brandon, I don't know if CloudFlare lied to me, I don't understand why they
> >>had too, but i have appreciated the fact they even responded to me even if i
> >>have only a "free" account on their service... enough said?
>
> >>The great trouble in all this is, in my point of view, that there is no way
> >>to have some kind of response from google even if i'm an early adopter of
> >>appengine and pay for their service. That scared me a lot.
>
> >>Martino.
>
> >>--
> >>You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> >>To view this discussion on the web visithttps://groups.google.com/d/msg/google-appengine/-/I0gte2ag83QJ.

> >>To post to this group, send email to google-a...@googlegroups.com.
> >>To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

> >>For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

[Martino A. Sabia]

I'm sorry maybe I gonna say something totally wrong, but the "real" problem here, IMO, is that there are NO EVIDENCE at all in any place we can access, to say what was the real problem. So we're talking about something that can be, but even can be not.

When Brandon said that Google see incoming traffic from CloudFlare as a DOS attack, for the header issue or whatever, I gone to appspot dashboard's Blacklist page to see if something was blocked. Well there was not, and there was neither an IP address from CloudFlare in the list of most active connections. I don't know if we can consider this an 'evidence' of something. Maybe Google have something in front of GAE that filters incoming traffic, but even for this, i have no evidence nor Google has declared this. Am I missing something here?

For sure i can i say that i have other GAE apps without CloudFlare and, during the issue, i was able to access them without problems. So I have some legitimate suspect, but i can't consider them the real problem 'cause i can't tell. The other think i can say is that, even if traffic comes from geographically same origin but on different connections/networks there was different behaviors. We had some users blocked for long time (3-5 hours), some who doesn't had any problem, some that had blocked for a short period of time (30min-1hour).

I have noticed that the blocking message was noticed especially by heavy user of the website (editors, owners, developers) and the more the user used the website in the past, the more was the blocking time.

I hope that there will not be a similar issue in the future, anyway the first think that i will do is temporarily disable CloudFlare to see if it's the real problem :D. In the meantime it will be 'nice' if the only one who knows what REALLY happened, i.e. Google, will tell us...

For Chris from CloudFlare: do you think CloudFlare will investigate or can you send some message to google (if you have some direct connection with them) on this issue, or simply for you there is no issue at all?

For everybody: what is the best practice in this case? What to do now? Open a ticket with google hoping they will investigate... some clues?

Martino 

[Brandon Wirtz]

Check you Logs and it will tell you a request was blocked. Sorry still on phone so I can’t remember how to search exactly but you get a warning or error and on there you can see the ip that hit you. When I played with cloud flare most the time it would present an ip, but not always, and some times the ip would be a local rather than public ip.  If you set your headers correctly on gae, they will do 75% of what cloud flare claims to do, with none of the risk of badness.  Based on your use case I will likely be building a caching library for gae that would make this issue go away for python and java users, an integrated cache which would be better and make your gae costs lower.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/pA-byZLu4UcJ.

[Brandon Wirtz]

Had a phone conversation with a friend who is  not a gae employee, but an IT contractor at Google, who suggested that google infrastructure often blocks large proxies that aren’t on their ISP whitelist, and that if this issue is to be resolved it isn’t necessarily a GAE issue but a “GOOGLE” issue and Cloudflare will need to get on that white list.  No clue how they do that, maybe a GAE employee can speak up, but this lends to CloudFlare needs to get some ducks in a row.

 

 

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Martino A. Sabia
Sent: Thursday, June 09, 2011 3:35 AM
To: google-a...@googlegroups.com
Subject: R: Re: R: RE: [google-appengine] Google blocks traffic to appengine application

 

I'm sorry maybe I gonna say something totally wrong, but the "real" problem here, IMO, is that there are NO EVIDENCE at all in any place we can access, to say what was the real problem. So we're talking about something that can be, but even can be not.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/pA-byZLu4UcJ.

[Martino A. Sabia]

Hi Brandon,

first of all thank you for your time to review this thread. As you suggested i took a deep look into my appspot logs but there was no errors logged for the behavior i described.

So i think that what happened was what your friend says on Google, they have some sort of filter in front of GAE. Do you have some suggestions on how i can submit my issue to google somehow - with some kind of probability to be read -?

For the caching suggestions: well i already have caching inside my app for my dynamic content. What i need is something that takes down my output traffic billing, which is the greatest voice in my billings from GAE.

Since i have written a custom CMS that saves images into my app on GAE that even serves dynamically the images, so i don't need to lock into statically defined image dimensions, the biggest issue - from an economic point of view - is that i need something external of GAE that can serve my static content in an affordable way.

The alternative, which is more complex and doesn't give me the same flexibility, is to store my images in fixed dimensions somewhere else (that gives me the same throughput, at a lower cost). But since Google made a great job with the image library on GAE i hope to not arrive to that decision.

So cames CloudFlare, that gives me external caching with no complexity on my app logic. It's a price issue for me, right now. When, and if, my website gets more money in, I can consider more complex solutions.

Thanks again,

Martino.

[Alexander Konovalenko]

On Fri, Jun 10, 2011, Martino A. Sabia <ezu...@gmail.com> wrote:
[...]

> So i think that what happened was what your friend says on Google, they have
> some sort of filter in front of GAE. Do you have some suggestions on how i
> can submit my issue to google somehow - with some kind of probability to be
> read -?

Martino, you could collect as much debugging information as possible
when this happens again and submit a Production issue to the App
Engine issue tracker. I guess the following information will help:

1) the request URL
2) the client's IP address
3) the exact time when the error occurred
4) the full error message or a screenshot
5) mention that the request came through CloudFlare
6) the outgoing CloudFlare IP address for that specific request
(extracted from CloudFlare logs, perhaps with the help of their
customer support)

Please post to the group if you find out anything new about what can
cause the error on the Google side and whether it can be related to
CloudFlare. I'm interested in the details, too, because I'm planning
to use a reverse proxy in front of App Engine.

-- Alexander

[Francois MASUREL]

I'm actually using CloudFlare in front of AppEngine for some low traffic websites (ex: www.filhot.com) and, crossing fingers, everything seems to be going fine.  But still very interested in any information about this potential problem.

Thanx in advance.

Francois

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Martino A. Sabia]

Thank you both for your interest on this issue. To update my 'problem' status, i had this kind of problem only once within one month of usage of CloudFlare and it happened only during the IpV6 day. Because it's a sort of startup website it has only 2.500 pageviews a day so i don't think it's a matter of traffic.

If it will happen again, and I sincerely hope that it doesn't :D, i'll post informations about this even here, i hope you will do the same ;). 

Finger crossed,

Martino.

[Brandon Wirtz]

As an SEO, not as an app developer.   I have billed more money to clients fixing issues caused by cloudflare than the client ever hoped to gain by using it.  When it screws up and serves a search bot with a captcha page your world ends.

 

As an App Developer, Google Edge Cache built in to Appengine does 70% of what CloudFlare does for scaling.  And just blacklisting bad bot IP’s and UserAgents will get you 25% of the way. 

[Waleed Abdulla]

Martino,

    I've had the same issue you had a few months ago, on two occasions. I have an external php server that calls my GAE server a lot to get data. One day, without notice, GAE started blocking all requests coming from my php server. The app works fine when I hit GAE from a different ip address. It was only the IP address of my php server that was blocked. 

    I had to bring my website down for a little while (5 minutes or so) so that my php server stops calling GAE. After that I tried again and it worked. It's not clear what GAE considers a DoS attack, and there is no way that I know of to add certain IP addresses to a while list (there is a black list, though). 

    One that I had to do was to prevent my php server from re-trying if requests to GAE fail. I suspect what might've happened was that GAE slowed down (happens from time to time), and then requests started timing out, so that caused my php server to keep retrying, and then GAE decided that my php server was doing a DoS attack. 

    Someone on this list suggested that if I make all requests to GAE with an authenticated admin user then that should solve the issue. Maybe. I didn't try it. But it seems that the issue happens before the request hits GAE, and that's why you don't see anything in your logs.

Waleed