APP DOWN due to some sort of undocumented Google security system

[Jeff Schnitzer]

Ok, this is fucked up. Visit http://www.voo.st/, and get this:

https://img.skitch.com/20120801-cd1h98pqwb8e8qryct9yjcqwgk.jpg

Something is triggering a false positive from a totally undocumented
Google security system. This is really, REALLY not ok. We are losing
sales and looking like total idiots to our customers:

-----
Our systems have detected unusual traffic from your computer network.
Please try your request again later. Why did this happen?

This page appears when Google automatically detects requests coming
from your computer network which appear to be in violation of the
Terms of Service. The block will expire shortly after those requests
stop.

This traffic may have been sent by malicious software, a browser
plug-in, or a script that sends automated requests. If you share your
network connection, ask your administrator for help — a different
computer using the same IP address may be responsible. Learn more

Sometimes you may see this page if you are using advanced terms that
robots are known to use, or sending requests very quickly.

IP address: 208.90.212.26
Time: 2012-08-01T18:02:00Z
URL: http://www.voo.st/
-----

We use CloudFlare as a reverse proxy. Wild guess is that some sort of
automated security system is cutting in and detecting CF's proxy as an
attack.

PLEASE TURN THIS OFF NOW.

Jeff

[Takashi Matsuo]

Hi Jeff,

I've escalated this issue. However, I don't think we can completely
turn this off.
What was the main reason that you're using CloudFlare?

-- Takashi

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>



--
Takashi Matsuo

[Kaan Soral]

There were many people experiencing the same problem before, after you spoke very highly of CloudFlare I asked them If they experience any problems with GAE, they said no, but I didn't believe

These are all offtopic, another offtopic question: Why don't you switch to GAE SSL? Is it the pricing?

[Jeff Schnitzer]

1) SSL at a reasonable price
2) Documented edge cache behavior
3) Real (as opposed to javascript-driven GA-type) traffic statistics
4) They do a number of convenient bits of processing (stripping
whitespace from html, etc)

Jeff

[Damon Billian]

Hi Takashi,

CloudFlare is a reverse proxy/CDN that provides services to hundreds
of thousands of websites. It would be nice to know what is triggering
this and you can contact me at my email address (damon@). My hunch is
that one domain on the service may be causing this trigger, so we
would really like to know what the cause is here.

I've checked a few of these cases and the IPs showing in the Google
report are not CloudFlare's in some cases....but the domains do appear
to be using us.

On Wed, Aug 1, 2012 at 12:06 PM, Takashi Matsuo <tma...@google.com> wrote:

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 12:10 PM, Kaan Soral <kaan...@gmail.com> wrote:
> There were many people experiencing the same problem before, after you spoke
> very highly of CloudFlare I asked them If they experience any problems with
> GAE, they said no, but I didn't believe

I'm pissed because this behavior is undocumented. Google doesn't say
"don't access your site through a shared reverse proxy because we may
shut you down with some sort of automated threat detection." This
isn't just a CF issue; any kind of proxy is subject to this totally
surprising behavior.

> These are all offtopic, another offtopic question: Why don't you switch to
> GAE SSL? Is it the pricing?

That is what I have done to bring the site back up.

Jeff

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 12:14 PM, Jeff Schnitzer <je...@infohazard.org> wrote:
>
> I'm pissed because this behavior is undocumented. Google doesn't say
> "don't access your site through a shared reverse proxy because we may
> shut you down with some sort of automated threat detection." This
> isn't just a CF issue; any kind of proxy is subject to this totally
> surprising behavior.

Another huge disappointment here is that the failure mode: A redirect
to http://www.google.com/sorry which produces a 200 OK response. Our
monitoring system interpreted this as "just fine" so we didn't get
notice of the downtime for a couple hours. A customer informed us of
the failure.

We have changed our monitoring system to grep for known content on the
pages... but this is incredibly dangerous. It's like a gigantic
hidden landmine perfectly positioned to cause maximum damage when
someone steps on it.

Jeff

[Damon Billian]

Hi Takashi,

I believe our CEO is going to reach out to our contacts at Google as well. If it helps in the interim, however, CloudFlare users the following IP addresses:

IPv4

204.93.240.0/24
204.93.177.0/24
199.27.128.0/21
173.245.48.0/20
103.22.200.0/22
141.101.64.0/18
108.162.192.0/18
190.93.240.0/20

 

IPv6

2400:cb00::/32
2606:4700::/32
2803:f800::/32

[Francois Masurel]

Getting the same really annoying error for a very very low traffic web site < 5 requests per minute : http://www.filhot.com

How can this be?  I'm using CloudFlare too.

Google, please respond.

Francois

[Francois Masurel]

Hi Takashi,

Do you mean that we can't use any kind of CDN with GAE?

I've been successfully using CloudFlare for a few months to make my websites reachable from China.

I don't have any other solution at the moment, at least in my budget.

It would be great to have a clear answer about this as soon as possible.

Thanx for your help.

Francois

> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Rick Mangi]

Same here! I'm on a thread with enterprise support right now and they are telling me that the "reverse proxy is to blame". WTF??

You guys have to stop changing things on us with no warning. This is NOT OK! We are paying customers (at least my company is). You're really giving us every reason to go over to AWS instead.

[Damon Billian]

Our CEO literally just shot one of our contacts over there some
additional information moments ago. I'll try to update you as soon as
I know more.

On Wed, Aug 1, 2012 at 3:05 PM, masterblaster <dan...@gmail.com> wrote:
> ditto. CloudFlare support recommended Pausing cloudfalre until google and
> CF work this out. Please help, Google.

>>>> > google-a...@googlegroups.com.

>>>> > To unsubscribe from this group, send email to

>>>> > google-appengi...@googlegroups.com.

>>>> > For more options, visit this group at
>>>> > http://groups.google.com/group/google-appengine?hl=en.
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Takashi Matsuo
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/qUKogHr3_wIJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Francois MASUREL]

Thanx Damon for keeping us informed.

Francois

[Damon Billian]

I'll do my best. Like I recommended to a few other folks that
contacted us already, pausing CloudFlare *might* help in the interim
(settings->pause CloudFlare). The only problem is that we will be
going direct at that point & no SSL:(

[Drake]

CloudFlare biting you in the ass.

It happens. Bonus...

Cloudflare doesn't form its proxy request headers correctly. Check the
archive this isn't "undocumented" I've mentioned a lot when helping other
people with the same error.

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 3:47 PM, Drake <dra...@digerat.com> wrote:
> CloudFlare biting you in the ass.

What stops me from crafting requests that Google will recognize as an
attack and running them through your CDN in the middle of the Survivor
season finale, shutting you down?

I mean, other than legal and ethical concerns.

This affects anyone that proxies GAE.

Jeff

[Joshua Smith]

Who had 3:47pm PDT in the pool for when Brandon would pipe up with his "I told you so"? Anyone?

:)

[Damon Billian]

Whatever, "Drake" (Brandon).

Hi Jeff,

We're in contact with our Google contact to find out what the issue
is. I'll post as I have any additional details to share.

[Drake]

Not if they form their request headers properly.

CF sends IPs that are malformed, nonsensical, and non-responsive. (and often
blank)

My CDN solution says "Request on behalf of X" the right way.

Oh, and CF has already tried doing that to me. More than once.

[Damon Billian]

Hi,

We have heard back from our contact at Google and this is being worked
on (don't have an eta for a full fix yet & will let you know...but
hoping within the next day or less).

It also looks like our IPs are going to get whitelisted to prevent
these issues in the future.

[Drake]

That's what y'all say every 9 months...

https://groups.google.com/forum/?fromgroups#!topic/google-appengine/om1DF_61
Xl4

[Jeff Schnitzer]

What evidence do you have to suspect that Google is treating CF as an
attack because of the way they form headers?

That doesn't even make sense - if CF is mangling headers, presumably
they've been doing it for a while, and it wasn't being blocked before.
More likely there's some sort of real traffic going through CF is
being recognized as an attack (false positive or not) and that is what
turned up the defense screens.

If so, you are just as vulnerable.

Jeff

[Damon Billian]

Hi "Drake",

We have already had the confirmation from the Google Security Engineer
that is our contact that he will look into this. It is probably best
not to comment on discussions that you are not involved in & can't
read minds.

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 4:07 PM, Drake <dra...@digerat.com> wrote:

Brandon, this is why I can't trust anything you say anymore. That
thread is unrelated to this issue, and either you don't understand why
or you are being deliberately disingenuous.

FWIW, wwwizer is almost certainly subject to this same problem. As is
any other proxy, including you.

Jeff

[Drake]

Instead of whitelisting your IPs Fix your Proxy. Google is not the only
place with these issues. Don't lie about the IP the request is on behalf of,
don't create IPs that don't exist or are in unassigned ranges. Don't try to
piggy back session headers that aren't from the same session. Google smacks
you for a reason. Band-aiding is what causes your users to get de-listed,
de-ranked, and have very bad days.

I don't have to read minds. I can see the headers, and I have helped LOTS of
customers after CF ruined their lives.

It's still Brandon, but I put Drake on because I get so much spam from bots
scraping lists.

[Damon Billian]

"you are being deliberately disingenuous."

I vote for this, personally.

Hi "Drake".
I'm pretty sure our engineering team, given their background at major
internet companies and the like, are probably pretty familiar with
creating properly formed headers.

[Drake]

This is a mirror of the conversation from 9 months ago and the one from 9
months before that, and the conversation on the Amazon forum, and the
Rackspace forum and the Google Webmasters forum...

https://groups.google.com/forum/#!msg/google-appengine/om1DF_61Xl4/cRtKFojKc
BMJ
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CFgQFjAB
&url=http%3A%2F%2Fproductforums.google.com%2Fd%2Ftopic%2Fwebmasters%2FsILDRV
zPo6U&ei=38cZUMO_L4mMqwGk6oCICg&usg=AFQjCNHk6YVJTj6-hj_2NYP1OBBIuSFQXw&sig2=
dRQ9_wSJ8bDC4nzzS6IQVQ


CF does headers wrong. CF doesn't play well with Google bot. Not news to
anyone.

CF "Double wraps" its proxy some times. Don't know why. When this happens
Google Freaks. CF doesn't report what IP the request is made on behalf of
sometimes. Google Freaks out. CF reports that it is proxying on behalf of a
Google Internal IP sometimes GOOGLE REALLY FREAKS OUT.

You guys can be assholes, but I'm telling you where your problem is. I told
you 2 years ago, I told you 9 months ago. I am telling you now. You can
stick your fingers in your ears, but this is not Google doing the wrong
thing, it is CF. You won't have this problem if you use Akamai's DSA/PLC
solution. Which is what I would recommend to anyone who needs international
access on GAE, or who wants to leverage enterprise class caching. It's what
we are using for our top tier deployments.

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 5:05 PM, Damian Menscher <mens...@gmail.com> wrote:
>
> Google automatically blocks IPs sending attack traffic. If you decide to
> have your entire userbase come through a handful of IPs (in this case the
> CloudFlare proxies), then any time a single one of your users attacks
> Google, it's possible all of your users will be blocked. Hopefully none of
> this is particularly surprising.

It surprised the hell out of me, and from other comments in this
thread, I don't think I'm the only one. This behavior is not
documented anywhere. It has *dramatic* repercussions for anyone
running on a reverse proxy to provide SSL or edge caching or access
from China or whatnot. It means that anyone with a few cleverly
designed requests can immediately DOS your app, and possibly a number
of other apps too.

I absolutely do NOT want this protection on my app. There is
currently one live thread on this mailing list about someone whose app
is getting DDOSed and Google's "protection" does nothing, yet I'm
getting shut down for a "false-positive". Combined with zero
analytics, zero alerting, and Google's sluggish response to inquiries
(downtime measured in hours, not minutes) and it's hard to see this
undocumented "feature" as anything other than a HUGE liability.

FWIW, I like CloudFlare's attack protection feature because I can
leave it off. I only care about it if I'm getting attacked. Oh, and
they give me nice charts and graphs so I can get a window into what's
actually going on.

> That page produces a 503, not a 200:
>
> bash-3.2$ curl -I 'www.google.com/sorry/?continue=http://www.voo.st/#'
> HTTP/1.1 503 Service Unavailable
> [snip other response headers]

You are totally right - sorry. Not sure why our monitoring system
didn't pick that up. Must have been confused by the redirect. At any
rate, we're grepping for specific content on the check pages now.

Jeff

[Damon Billian]

The headers would be coming from nginx. Unless nginx is passing along
malformed headers, it wouldn't be caused by us.

"CF does headers wrong"
Please provide something specific that is malformed. We're more than
happy to check on it both internally and with nginx. To date, you just
keep making claim after claim without providing anything to
substantiate it.

"CF doesn't play well with Google bot"

Also incorrect. We have hundreds of thousands of sites using us
without issue. In addition, crawl errors that people write on the
forums may or may not be related to CloudFlare and we do investigate
these complaints. Things that could go wrong with crawling (a small
sample):

1. user changed something in robots.txt recently
2. bad .htacess rules
3. security plugins (not CloudFlare) blocking googlebot

If we find an issue that we're absolutely causing, we fix it. If we
find that we're having a problem with a service (like Google today),
then we reach out to them to find out what the issue is and fix it.
The internet is a big place & a lot of things can happen that are not
to blame on any one company.

I'm sure some of your customers have issues with your service as well
that may/may not be related to your product I don't jump in and go:
"SEEEEE (done on purpose) what can go wrong when you use XYZ?" I also
don't make any claims about how your service works or any other
weakness in your product. That is the absolute worst way to market
your product.

[Jeff Schnitzer]

On Wed, Aug 1, 2012 at 5:29 PM, Drake <dra...@digerat.com> wrote:

> This is a mirror of the conversation from 9 months ago and the one from 9
> months before that, and the conversation on the Amazon forum, and the
> Rackspace forum and the Google Webmasters forum...

I'll pretend for a moment that you're being thick instead of
deliberately disingenuous:

> https://groups.google.com/forum/#!msg/google-appengine/om1DF_61Xl4/cRtKFojKc
> BMJ

This was back when CloudFlare offered full-mile SSL to GAE. They did
it by issuing a second ssl request to ghs.google.com with a Host
header override to xyz.appspot.com. Apparently that worked for a
while, until Google stopped accepting ssl requests on ghs.google.com.
CF was doing the wrong thing (xyz.appspot.com != ghs.google.com) but
this problem is totally unrelated to today's issue.

Apparently GAE wasn't a big enough market for them to devote enough
engineering resources to fix that particular problem, so the feature
got removed. The reason I know what's going on here is because I
lobbied (unsuccessfully) to get their platform lead to bring the
feature back.

> http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CFgQFjAB
> &url=http%3A%2F%2Fproductforums.google.com%2Fd%2Ftopic%2Fwebmasters%2FsILDRV
> zPo6U&ei=38cZUMO_L4mMqwGk6oCICg&usg=AFQjCNHk6YVJTj6-hj_2NYP1OBBIuSFQXw&sig2=
> dRQ9_wSJ8bDC4nzzS6IQVQ

Who knows what's going on here, but it isn't related to anything that
happened today. It's not even running on GAE or subject to GAE's
attack prevention.

> You guys can be assholes, but I'm telling you where your problem is. I told
> you 2 years ago, I told you 9 months ago. I am telling you now. You can
> stick your fingers in your ears, but this is not Google doing the wrong
> thing, it is CF. You won't have this problem if you use Akamai's DSA/PLC
> solution. Which is what I would recommend to anyone who needs international
> access on GAE, or who wants to leverage enterprise class caching. It's what
> we are using for our top tier deployments.

Are you so certain of this that you're willing to grant me free
license to DOS your CDN at a time of my choosing? I have some
reasonable suspicions as to how to tickle Google's defenses, and
shutting down your entire business in the middle of prime-time would
be a really spectacular way to prove someone wrong on the internet.

Jeff

[Drake]

Here from when I was working with the guys at Google about proxying requests
when I was consulting for Akamai.

Your requests will be blocked if you cross a thresh hold of requests that
meet the following criteria.


PROXY reports request on behalf of Google Internal IP address. (or any IP
assigned to Google) (CF does this a lot)
You can check that.

PROXY Reports Request on behalf of a session/token which has expired. (CF
does this a lot)
You can check that.

PROXY reports request on behalf of invalid IP. (octet missing)
You can check that.

PROXY reports request on behalf of invalid IP. (127 or 169 address range)
(CF does this a lot)
You can check that.

PROXY Requests SLURPbot on behalf of non-Yahoo IP Address. (CF does this a
lot)
You can check that.

PROXY made request on behalf of User Agent which is unavailable on platform
indicated in the request. (CF does this a lot)
You can check that.

[Drake]

> Are you so certain of this that you're willing to grant me free license to
DOS your CDN at a time of my choosing? I have some reasonable suspicions as
to how to >tickle Google's defenses, and shutting down your entire business
in the middle of prime-time would be a really spectacular way to prove
someone wrong on the >internet.

One of the CF guys does this everytime I bash them, Why would it be any
different that I would expect you don't play too?

The resume transfer by bytes of a file on the next to last byte of the file
was a neat trick. I was impressed.

I only have one CDN client at the moment. As I have stated before I don't
have any desire to be in the business, I got in to it because so many
clients who adopted CF had their livelihoods ruined.

[Jon Stevens]

On Wed, Aug 1, 2012 at 6:03 PM, Drake <dra...@digerat.com> wrote:
> I only have one CDN client at the moment.

I think this says it all.

jon

[Jeff Schnitzer]

If I read this right, you're telling me that anyone can DOS your CDN
by making a lot of requests with bogus User-Agent strings (which,
incidentally, was one of my guesses). Forget CloudFlare for a moment.
You aren't the least bit concerned about that?

Or are you saying that you filter User-Agents to ensure that you only
give Google "safe" ones. Are you sure you got them all? Sure enough
that you won't mind having to write an apology letter to your biggest
customer explaining what went wrong and what you are changing so that
it never happens again? Like I just did?

Jeff

[Drake]

> If I read this right, you're telling me that anyone can DOS your CDN by
making
> a lot of requests with bogus User-Agent strings (which, incidentally, was
one
> of my guesses). Forget CloudFlare for a moment.
> You aren't the least bit concerned about that?

If your CDN is worth their salt they are blocking the people doing this "to
you".

That is a feature they advertise.

[Jon Stevens]

You bring up headers, but once again, you try to deflect the
conversation to something irrelevant. It is absurd to think that a CDN
should be responsible for not passing through headers. That is like
playing an endless whack-a-mole game. In fact, I'd hope that my CDN
would mess with the headers in the least amount possible. I'd hate to
be on the tail end of debugging an issue where data is randomly not
making it to my origin.

The main thing 'wrong' with this CloudFlare situation is that they
didn't have a whitelist agreement setup with Google already. I find
this rather shocking because as a huge CDN, you'd think they'd be
whitelisted with all the major players. That is like setting up a mail
service and not making deals to SPF/DNSBL/whatever whitelist all the
email you're sending.

jon

[Drake]

I never wanted to be in the CDN business. As a load balancer and traffic
router I have a number of clients, but I can make a LOT more money on the
affiliate programs and markup from other peoples products, and I hate
customer support. Because people don't seem to bother to RTFM.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-

> appengine+...@googlegroups.com.

[Drake]

>whitelist / headers

Playing with the headers sucks.
CF plays with theirs a lot, so if you are in the business already, be in the
business.

If you Cache you kind of have to, because Referrer often isn't going to be
what is expected.

CF runs on other people's IPs so they have less control over being white
listed. (if this is outdated I apologize)

NGinX isn't designed to run quite the way they run their setup, and there
are some weird issues that arise from that. This is one of the hardest
things about being a CDN, you are trying to do the best fetching possible
and minimize load, but you have to muck with stuff because if Shiela in
Australia requests a page, and Bobby in the UK requests a page you want to
serve the latter out of cache, but that means that when Bobby request the
next page his sessions, and referrer is going to look strange to the target
server.

At this point... There is almost no need for a CDN on GAE. Page Speed does
this MUCH better than CF. Almost as well as CDN In A Box does, but with
fewer "gotcha's" than either (CIAB never liked Authenticated users as much
as I wanted it to... and it has had trouble with Japanese and Korean
characters from time to time, (so does CF))

For mom and pop, use Page Speed by Google it is a one click bolt on to GAE.
For Enterprise Go with Akamai.

[Jeff Schnitzer]

There are a variety of reasons people run reverse proxies against GAE.
Someone in this thread has already mentioned they do this to support
access from China. That poor developer getting DDOS'd would probably
like to use CloudFlare right now. Folks using wwwizer and custom
solutions for SSL ($1200/year is very painful to bootstrappers us... I
can't even imagine what that would be like for a small developer in
India).

These proxies are all subject to being DOSed (deliberately or not) by
one script kiddie on dialup. Basically, GAE does not safely support
any kind of reverse proxy. This needs to be a huge red warning label
in the documentation, but there is nothing. The subject has not been
brought up on this list in the last three years, through many
discussions of running proxies to get around things like the GFW.
This is the first time it has come up.

It's an armed landmine. It really should be defused for appengine,
because as soon as this thread fades into the archives otherwise savvy
developers are going to step on it again. Or script kiddies are going
to figure it out and GAE is going to get another huge black eye on
Hacker News.

Jeff

[Drake]

Jeff,

Your CDN should have the conversation, it should never hit the list. Google
is not the only place that CF bumps heads on the network edge. There have
been similar discussions on other cloud providers.

Google is doing some things with security through obscurity. Which sucks.
They are also good about telling the people they have peering arrangements
the rules.

Akamai won't bump in to this because they are whitelisted.

Limelight, same deal.

Also since a CDN should be handling DDOS detection, they should be keeping
you from encountering this.

The hoops I had to go through to get on the Azure white list... And I was a
former MSFT employee, working on making that work for a Tier 1 MSFT partner
at the time. (3 letters start with an A ends with a T)

As to DDoS-ing via the CDN... that worries me less on a daily basis than the
things I know about ways to crush a GAE instance that don't require a DDoS
because a single machine (or another appengine instance) can generate 10k
times more load than it puts on the attackers machine... That bothers me.
(and the same attack works even better on CF because the same flaw exists in
them, it is not in LimeLight, Akamai you can fix it if you set things
"wrong")

[Francois Masurel]

Problem is not GAE specific as it impacts Blogger too :

https://support.cloudflare.com/discussions/suggestions/2007-blogger-hosted-sites-on-cloudlflare-receiving-error-messages-from-google 

On Wednesday, August 1, 2012 8:13:42 PM UTC+2, Jeff Schnitzer wrote:

Ok, this is fucked up.  Visit http://www.voo.st/, and get this:

https://img.skitch.com/20120801-cd1h98pqwb8e8qryct9yjcqwgk.jpg

Something is triggering a false positive from a totally undocumented
Google security system.  This is really, REALLY not ok.  We are losing
sales and looking like total idiots to our customers:

-----
Our systems have detected unusual traffic from your computer network.
Please try your request again later. Why did this happen?

This page appears when Google automatically detects requests coming
from your computer network which appear to be in violation of the
Terms of Service. The block will expire shortly after those requests
stop.

This traffic may have been sent by malicious software, a browser
plug-in, or a script that sends automated requests. If you share your
network connection, ask your administrator for help — a different
computer using the same IP address may be responsible. Learn more

Sometimes you may see this page if you are using advanced terms that
robots are known to use, or sending requests very quickly.

IP address: 208.90.212.26
Time: 2012-08-01T18:02:00Z
URL: http://www.voo.st/
-----

We use CloudFlare as a reverse proxy.  Wild guess is that some sort of
automated security system is cutting in and detecting CF's proxy as an
attack.

PLEASE TURN THIS OFF NOW.

Jeff

[Rick Mangi]

That's what we did as well.

To answer the question of why did we "switch" to cloudflare ssl, and was it the price. The answer is that google's ssl is a brand new product, we were on cloudflare before it was an option with google. And yes, it's a lot cheaper. Plus they have all the other features that Jeff mentioned (threat control, edge caching with a purge option, non javascript page tracking, flexible dns configuration, etc.)

On Wednesday, August 1, 2012 6:05:31 PM UTC-4, masterblaster wrote:

ditto.  CloudFlare support recommended Pausing cloudfalre until google and CF work this out.  Please help, Google.

On Wednesday, August 1, 2012 5:48:03 PM UTC-4, Rick Mangi wrote:

Same here! I'm on a thread with enterprise support right now and they are telling me that the "reverse proxy is to blame". WTF??

You guys have to stop changing things on us with no warning. This is NOT OK! We are paying customers (at least my company is). You're really giving us every reason to go over to AWS instead.

On Wednesday, August 1, 2012 5:06:49 PM UTC-4, Francois Masurel wrote:

Hi Takashi,

Do you mean that we can't use any kind of CDN with GAE?

I've been successfully using CloudFlare for a few months to make my websites reachable from China.

I don't have any other solution at the moment, at least in my budget.

It would be great to have a clear answer about this as soon as possible.

Thanx for your help.

Francois

On Wednesday, August 1, 2012 9:06:05 PM UTC+2, Takashi Matsuo (Google) wrote:

Hi Jeff,

I've escalated this issue. However, I don't think we can completely
turn this off.
What was the main reason that you're using CloudFlare?

-- Takashi

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

--
Takashi Matsuo

[Kaan Soral]

Indeed CF looks great on paper, but no one should expect it working properly with GAE anymore

That's how big companies operate, since CF and GAE/Google are potential competitors, logically speaking they would disrupt their service as much as they can with minimal responsibility

I would guess their internal contacts replying positively to them, halting them a bit, say they are working on it etc. but overall they will be stalled and most of you will be forced to switch to GAE SSL

This is all logical, but Google should up their game, improve their products more swiftly if they are going to bombard third parties like this, otherwise like you said, people will switch to alternative services (AWS etc)

[Chris Ramsdale]

Jeff, et al.--

We have verified that a configuration change on our side led to certain requests being denied / redirected.  The rollback of this change started earlier this morning and should be completed shortly.  We are actively looking into measures that we can take to ensure that issues like this are caught prior to rolling out to production.

If your application continues to be impacted please contact me directly. 

-- Chris 

Product Manager, Google App Engine

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Cesium]

I hate it when Mommy and Daddy fight.

[Michael Hermus]

+1 Internets for Cesium posts. They usually make me laugh.

[Jon Stevens]

If you are upset by Google's pricing for SSL, please star this issue:

http://code.google.com/p/googleappengine/issues/detail?id=7932

jon

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Jon Stevens]

Here is another issue to star... SSL is too difficult to setup.

http://code.google.com/p/googleappengine/issues/detail?id=7933

jon

[Francois Masurel]

Will we have an explanation from Google or CloudFlare about what went wrong?

Everything seems to be back to normal at the moment but for how long?  Is GAE CDN incompatible?

Thanx for your answers.

Francois

On Wednesday, August 1, 2012 9:06:05 PM UTC+2, Takashi Matsuo (Google) wrote:

Hi Jeff,

I've escalated this issue. However, I don't think we can completely
turn this off.
What was the main reason that you're using CloudFlare?

-- Takashi

--
Takashi Matsuo

[Damon Billian]

Hi Francois,

A Google employee already posted about the issue.

"Jeff, et al.--

We have verified that a configuration change on our side led to
certain requests being denied / redirected. The rollback of this
change started earlier this morning and should be completed shortly.
We are actively looking into measures that we can take to ensure that
issues like this are caught prior to rolling out to production.

If your application continues to be impacted please contact me directly.

-- Chris

Product Manager, Google App Engine"

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>> >
>>
>>
>>
>> --
>> Takashi Matsuo
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/hAyrgn0xZQwJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Rick Mangi]

You could start by being a little more transparent about what you're doing. Publishing release roadmaps are the norm for almost every software company in the world. You guys seem to love to leave us (the users) in the dark about everything you do until it's released. This is the 2nd time our site has been taken down for days by something which we could have told you was going to break it. 

sorry, but we're bailing on appengine as soon as we can.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Chris Ramsdale]

Is GAE incompatible with other CDNs?  No.  As mentioned earlier there was an issue on our side that that resulted in Google incorrectly blocking traffic from services such as CloudFlare.  For security reasons, we cannot provide greater detail.  We did identify the root cause and have rolled back the associated changes.  We are also working with our own internal testing teams and CloudFlare to better ensure that this does not happen again.

-- Chris

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2xbF2m-Q5a0J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Francois MASUREL]

Thanx Chris for your answer.

François

[Chris Ramsdale]

Rick,

We publish upcoming features and functionality within the "App Engine Features" section of our developer site:

https://developers.google.com/appengine/docs/features

In regards to bailing on the platform, it is unfortunate to hear.  If you have time, I would like to understand what we overlooked and what information you had that would have allowed us to avoid the current issue, as well as why you are planning on leaving the platform.  Thanks...

-- Chris

On Mon, Aug 6, 2012 at 8:04 AM, Rick Mangi <Ri...@broadcastr.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2xbF2m-Q5a0J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Jon Stevens]

Chris,

I've tried to setup billing on my appid three times now. I set it up
and then it seems to turn off on its own without even sending me an
email or any notification.

Every few days, I check and it keeps saying that I have a new past due
bill. When I go to turn it on again, it asks me for a new shipping
address (even though I already have two duplicate addresses in the
select box... one was from the last time I tried to set it up and I'm
not even shipping anything!?!).

Now that I'm on the checkout page trying to *pay you money*, I can
also see that there is a javascript error on the page, which is
preventing any clicks on buttons from working.

This is seriously the work of amateurs, I really expect more quality
and testing from you guys before you put stuff like this into
production. Please, let me know when you have billing for my appid
fixed. If you can't figure it out, feel free to contact me privately.

jon

>>>> To post to this group, send email to google-a...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengi...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/2xbF2m-Q5a0J.
>>
>> To post to this group, send email to google-a...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengi...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

[Jon Stevens]

Sorry, I apologize for the amateur remark, that was a bit much.

jon

>>>> To post to this group, send email to google-appengine@googlegroups.com.

>>>> To unsubscribe from this group, send email to

>>>> google-appengine+unsubscribe@googlegroups.com.

>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/google-appengine/-/2xbF2m-Q5a0J.
>>

>> To post to this group, send email to google-appengine@googlegroups.com.

>> To unsubscribe from this group, send email to

>> google-appengine+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Cesium]

Chris,

I am using GAE for my startup and it's just freakin' awesome.

I don't use frameworks. I don't use CDNs (whatever those are!?). I don't use SSL.

Just MVP and Objectify, and it flat out rocks.

Daily, I pee in my pants with excitement.

My customers are thrilled. (Both of them).

David

(Thought we needed a little 'balance' on this thread.)

[Chris Ramsdale]

Would you mind sending me (via direct email) your app ID(s) and we'll look into this?  We're aware of Checkout/Wallet issues and are improving the user experience...I promise.

-- Chris