Google App Engine's Team Dishonesty

[trilok]

Google app engine hello,

Let me first specify that I am a paying app engine user for about 1.5
years. We, at my company, have developed an online restaurant takeout/
delivery ordering service running completely on the appengine. We
currently serve over 50 restaurants in my home country, and are now
expanding abroad with restaurants in Canada, Hungary, Belgium, UK, and
more.

Ever since the appengine's release from production a week ago, there
has been 3 (!!!) major disruptions - On 7th for 45 minutes, yesterday
for 30 minutes, and right now. I understand that failures occur, but
specifying a "99.95%" and being so far from it is to me a major
failure on the part of Google.

To make matters worse, we, AppEngine's paying users, NEVER receive any
explanations or descriptions of the cause of the failure, the solution
and Google's efforts to prevent its returning occurance. Not by any
means to compare, but EC2's team constantly admit and report ALL of
the failures and their debriefing!

And now for the "cherry on the top", and the reason I used the word
'dishonesty' - You remove any note of the disruption from System
Status. For example, yesterday there was a disruption causing 40 secs
(!!!) of latency in response. Today viewing the System Status,
yesterday is marker with "No significant issues". That to me is
dishonesty and a clear cut lie.

Unfortunately, our service is now so deeply connect to the AppEngine
framework that leaving this service is currently not an option, but I
would definitively not advise or recommend anyone to use the AppEngine
today, and my next product will definitely not run on the AppEngine.

Regards,

- Yoav.

[Brandon Wirtz]

Are you on High Rep or Master Slave? I have not had an outage in something
like 10 months.

Google app engine hello,

Regards,

- Yoav.

--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengi...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

[Robert Kluin]

I've seen at least two or three *major* disruptions on high-replication apps since GAE went out of beta.

[Renzo Nuccitelli]

This input about System status is True. I have seen the system status
indicating problems in some days and some days later just "erase" that
the problem ocurred.

[Olivier]

+1 for a more honest and consistent system status

as for the issues, since we migrate to HRD, everything has run quite
smoothly for us which is nice.
Before that, it was really, really bad on MS.

[Andrius A]

+1

Can somebody from Google find the respect and comment here?

[WallyDD]

I have always wondered the same thing. One minute there is an issue, a
few days later it never happened.
Google is far from alone with such issues which is why there are
websites/services that monitor cloud status.

It may be a little unfair calling the app engine team dishonest.
Trying to change something in a large organization can be a very
unrewarding experience.

[Gregory D'alesandre]

Hello all,

Trying to show an accurate and honest representation of the status of a massive distributed service is a really hard technical challenge but an even harder conceptual one.  While your app might be showing higher latency or errors that doesn't indicate a systematic issue with the whole service.  For instance, the main reason we are encouraging customers to move to HRD is because M/S is dependent on single BigTable tablets, this means you can have lots of issues when there is absolutely nothing wrong systematically with GAE.  There was a small service disruption (on the order of minutes) in some HRD apps recently when a datacenter was having systematic issues so we have to re-ruote that traffic.  But, it didn't show up on the status site because it was a short disruption that only affected a small portion of users.

The upshot of this is that our status site gives a general sense of how App Engine is running but that doesn't show whether your app is experiencing issues or not, it just shows whether the probes we are using to generate the information are having issues.  So, at times, when it says there is a problem initially and then it disappears it is usually because a prober app was having an issue but it was not a large-scale issue for all of GAE.  We are not trying to hide issues, quite the contrary when there is a large-scale systematic issue we have a policy of doing post-mortems and posting them publicly,  So, if your app is having an issue and the status site looks fine, this is probably not a lie but rather an artifact of how we show status for the system.

We are looking into ways to improve this to be more useful but I hope that helps clarify why you see what you see.

Yoav, I never saw a response as to whether you are using HRD or not, the 99.95% SLA only applies to HRD because we know M/S is going to have issues.  As always, thanks for the feedback!

Greg D'Alesandre

Senior Product Manager, Google App Engine

[Andrius A]

Thank you Gregory,

You say it affected a small portion of users and you later remove issue notification from GAE status page which makes your status history and availability counter look better as well for the new customers comming to GAE and checking status page.  Is this honest?

you never know, maybe next time your app will be within that small portion of incidents..

[trilok]

Hello,

Me again :)

Yes, we are using HRD. Indeed since we moved to it (at least 4-5
months ago), things became stabler... stable enough? Good question.

I have a monitoring SW (running on EC2) making a request every minute.
In the past week this monitoring system gave me at least 2 errors per
day and sometimes more (500 - Internal Server Error... and no, the
request never reached our app. It fails before us). I know it seems
like a very low number, but still I'd like to have one day without an
error. (2 disappointed clients a day for me, as a very young start-up,
can cause some very bad brand reputation).

Regarding the dishonesty issue - It still amazes me that at times I
see an "Investigating" or "Elevated" sign in the system status,
sometimes 45 minutes of a very high Java latency, and the next day "No
significant issues" on the previous day. I, and I'm guessing the rest
of the people here, would really appreciate some kind of
acknowledgment from Google that you have seen the issue, and didn't
just "let it disappear" but rather investigated it, found the cause,
and are performing steps to make sure it does reappear.

Basically what I'm asking, and what I think everyone is asking here,
is to know that there is someone to talk to. If you look back at the
'issues' site, you'll see many 'production' issues from people like me
crying for help during downtime. These issues have gone unanswered
even now, months after the issues. If you'll look at other monitoring
sites you'll see that there is some kind of description of the issues
as they happen. Now, I fully understand that during times of
disruptions you guys are amazingly busy in trying to solve them, but
perhaps just a word from a human-being and not an automated SW to show
that we have someone there helping us, and perhaps, just perhaps an
ETA on a solution?

Thank you again, and sorry for the long posts - It is just frustrating
having nothing to do during down times other then refreshing the
status monitor and prying (last downtime, 45 minutes, I pryed to every
religions' god - anything that can work, I don't discriminate during
down times :) )

- Yoav

[Jeff Schnitzer]

FWIW, I've met a lot of the GAE team at various local events (the privilege of living in the SF bay area) and I don't think any of them would do something genuinely "dishonest".  By and large they seem like normal developers like you and I.

"Never attribute to malice what is adequately explained by stupidity" is a good principle - if the reporting dashboard is not behaving correctly, it's most likely a bug, not a willful conspiracy.  Bringing it up on this mailing list is good - creating an issue in the issue tracker (with specifics) is better.  Inflammatory subject lines don't help.

Jeff

[Adrian Scott]

If I were "King of GAE", I would want to be very, very communicative about issues that affect even only a portion of apps and users, e.g. having a detailed hosting environment performance log (even if there is no in-depth post-mortem, even just a one-line log item like: "There was a small service disruption of less than 5 mins in 1% of HRD apps recently when a datacenter was having systematic issues so we have to re-ruote that traffic"). That would be a Huge! selling point to me in selecting and staying with a particular cloud provider.

What does not get measured...

And whether or not within Google a GAE customer-affecting issue was because of the GAE Team's tech or some other Google team, it should still be reported.

My $0.02,
-A

Adrian Scott, Ph.D.
CEO, Founder
CoderBuddy
http://www.coderbuddy.com/ <-- Create a Facebook or Google App Engine app in a minute without installing anything

[Brandon Wirtz]

We may just have different locations for where our apps are... But we have
uptime monitoring on several dozen domains and they get checked every 5
minutes, so we are checked something like every 15 seconds for the uptime of
our primary app. We've had 90 seconds or so of downtime, and during that
time static files still served and requests that took less than 20 seconds
to fill were served.

We have seen issues where long time to fill requests had high fail rates.
We have also seen that if we set the application settings to have too few
idle instances that we got a LOT of 500 errors.

Do you have your app set to automatic? Or have you clamped your Apps number
of idle instances. How long does a typical request take to fill? How about
a "long" request.

Some of your downtime may be your own fault, not GAE's. Don't know that for
sure, but when my multiple apps don't exhibit a behavior I assume that the
issue isn't system wide, but localized to something a given user is doing.

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of trilok
Sent: Saturday, November 19, 2011 3:42 PM
To: Google App Engine

[trilok]

OK, ok, ok, so we don't talk in theory (and to answer one question -
the errors are of requests that don't reach our app. They fail
before):

Example 1
----------------
http://code.google.com/status/appengine/detail/serving-java/2011/11/15#ae-trust-detail-helloworld-secure-get-java-error_rate

- First look at the general console, you will see no mentioning of
this event.
- OK. Failures happen. Now, why? what is done? Who is working on it?
During the failure, when will it be fixed? Any kind of online
information so I won't feel alone in the dark! Trust me, I'd feel
better knowing that someone brought his pet and it cut the power cord,
and it will take 30 minutes to come back rather then nothing at all!

Example 2
----------------
http://code.google.com/p/googleappengine/issues/detail?id=6274&can=5&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

This was during a 45 minute down-time (BTW, marked again as 'under
investigation' after being marker as 'no significant issues' so kudos
on the remark!) between the 5th and the 6th phone call I've received
from restaurants wanting to leave my system. I think this was after I
tried Tweeting app engine, and right before I was looking for Mr.
Page's personal email address :)

- Is there anyone to talk to during these times or do I HAVE to sign
up for the $500 program just to have someone tell me when will the
problem be solved?

I don't think GAE teams is anything but very very professional. I
don't think I would ever manage to really set up such a wonderful
service. I just think that their customer-relationship needs a bit of
work (which will definitely result in more people joining and
staying!).

- Yoav.

[Brandon Wirtz]

You avoided my question. Have you limited max idle instances.

Failure before reaching the app can be that you have no available instances
to handle the request which was why I was asking if you had set max idle
instances. If those are set wrong you will get 500 errors up the wazoo and
never see a single error in your log. I have helped several people who
thought they were saving money by setting those to very low numbers only to
find out they were losing 10% of their traffic to errors.

[trilok]

Mr. Wirtz,

Thank you for your recommendation!

Unfortunately I would really not want this conversation to move to the
'why I get so many errors' realm, and as you can clearly see the
examples I have given above are system wide daily errors rather then
application specific errors. My original post specified 3 major system
wide disruptions, but the rest of my post talked mainly of GAE team's
handling of these disruptions.

I perfectly know that there are errors caused by my company's system,
and by our configuration - I am sure there are still many things we
can do to make them better (Thank god we are far from 10% error rate
on our account). I have no doubt that 99% of the time errors are
caused by users' bugs rather then GAE's production issues.

I would really want this conversation to move something in Google to
make their handling of the system wide errors as transparent and
honest as possible, to help us communicate as much information to our
clients as we can. (Mr. D'alesandre, are you reading this? Am I
achieving my goal somehow? :) )

I hope you understand why I am 'avoiding' your question, as I simply
think it can take us out of the path of to the discussion I am trying
to advance (Yes, I am limiting my max number of instances, and yes I
know it can cause errors but I almost always have at least 2 idle
instances, and my errors are at times where traffic is not at its
maximum).

- Yoav.

[Brandon Wirtz]

You sited outage numbers greater than what Greg did. If your app was down
for your own fault then it is NOT a google issue.

If less than 1% experienced an outage of less than 5 minutes, and those apps
were idle then the number of "lost" pages was likely even lower.

You said they were not meeting SLA, and Not Reporting. But if your numbers
are wrong, then we don't know that is the case. You can't call Google out,
AND not diagnose if the problem is YOUR fault.

[trilok]

Mr Wirtz,

You have not read my response.

All of the examples I have given are Google issues since they are
presented in the system status.

I do know when my system is down for my fault and when it is down on
Google's fault. If you dive into the system status you will see that
even by their account they do not meet the SLA (Current availability
is 98.92%. FAR from 99.95%).

Again, my real issue is that Google not only does not meet their SLA,
but that they do not communicate the issues transparently to their
users.

Facts are these:

1. Google do not meet their SLA (again, look at their stated current
availability in the system status).

2. When they are down on their fault, I have no way of knowing when is
the problem going to be fixed, and how are they preventing it in the
future. If I do not know the answers to those questions, how can I
communicate stability to my clients? What do I tell my paying clients
when they ask me 'when is the system going to be up again'?

I am begging you again, do now make it a "who's fault is it that my
system is down" issue! Sadly I already think that you've diverted the
conversation to a point where I won't get any answers now

Bottom line I would like Greg to answer: How can I continue using the
GAE when I do not get transparency regarding Google related down times
(during or after)?

- Yoav.

[Brandon Wirtz]

I read your response.

I don't think you read the graphs you sent.
For 45minutes they were at 30% errors. That only counts as 15 minutes of
downtime. And since it appears that only polls once every 5 min or so I'm
not sure what that means for SLA.

You spoke of downtimes you had that weren't reported elsewhere. And you
said that you know it was GAE's fault because they didn't hit your App. I'm
saying that your "downtime" may have been misconfiguration.

Also even if you are correct that 99.5 isn't being met since they are at
98.2 or something along those lines, it is early in the yearly average.

I'm all for calling people out when they have made mistakes, but my uptime
thus far is north of 99% and is likely north of 99.75% Amazon had nearly a
week of down time, they will be 2 years getting that back to an average of
99.5.

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of trilok
Sent: Sunday, November 20, 2011 12:03 AM
To: Google App Engine
Subject: [google-appengine] Re: Google App Engine's Team Dishonesty

Mr Wirtz,

Facts are these:

- Yoav.

--

[trilok]

Although I repeatedly asked you not to take this conversation to
"Who's fault is the downtime", you still do so!

Again, my problem is that when it is Google's fault, there's no one to
talk to - Can you argue anything relevant to this point please? Amazon
had a week downtime, but THERE WAS SOMEONE TO TALK TO! Whenever Amazon
is done I receive an email from them telling me why, and how they
solved it.

I would be happy to continue the conversation of "is google meeting
their SLA" in private. But read the subject of this email - You can't
argue that on the 11th of November there was a serious issue which is
not represented in the system status, and that we have not been
informed any of!

Regarding the points you have mentioned:

1. For 45minutes, 30% of errors, is to me 45 minutes of downtime.
Since it happens during evening time when thousands of people enter my
site to order lunch, and 30% of them need to refresh the site to see
the welcome page and the rest suffer latency, I see it as 45 minutes.
(I agree, it could be worse, but it is still bad for my brand name).

2. By definition you cannot have the uptime you talked about when
Google themselves say that they have a 98.92% up time! Unless you do
not have traffic during those times, which is cool for you, but we
have traffic to our site all of the time.

- Yoav.

[Brandon Wirtz]

Yes. Because the thread is "GAE team is dishonest"

If their honesty is in contention then who's fault your app is down is
relevant.

Yes I can have higher than 98.92 because not everyone was down during those
times. Plus Edge Cache can serve up to 60% of my traffic for short periods
of time, so even during an outage I can be partially up. Also Static files
haven't had any downtime.

As to the "someone you can call" if everything is down, calling someone
doesn't help. Even if you had a premier app, the conversation would go "Hey
we are down" "Yep we lost all of the datacenter you are in" "can you get me
back up?" "yep when everyone else comes back up"

What a support account gets you is things like "Hey I'm on Python 2.7 and
Thread Safe seems to be giving me 2+2 = 5 and it takes 50 seconds to
calculate it" Or the ability to call someone and say "hey I need to alias a
bunch of apps on to each other and merge the data, is there a best practice
for that?"

Would an in dashboard SLA counter be awesome? Yes. But you are delusional
if you think Amazon has ever been forth coming about issues.

[Brandon Wirtz]

Oh, and as to destroying your thread. I don't expect that Greg and Ikai and
Nick, and the rest of the Google team stopped reading the thread because I
commented on it.

I get the distinct impression that your Downtime was higher than "stock"
because you had something misconfigured. I suspect that I hit the nail on
the head when I said your latency and instance settings were wrong, and you
are annoyed because I pointed this out.

And I don't feel bad about destroying your thread (if I have) the tone of
this thread was not appropriate. If you call someone stupid. It's an
insult, but it is not a moral judgment. If you say someone is not
transparent, it is just a statement of fact or opinion with no moral
judgment. When you call someone dishonest you are passing moral judgment.
When you judge someone's morals you should do so without question of them
being wrong.

You will see me "troll". Tell people they are wrong. I have even gone so
far as to say that an employee who has cost the community LOTS of money and
trouble should quite possibly be fired. I have NEVER judged those people on
a personal level. Call me an Ass, a Troll, an Idiot. Great I might do
something to deserve that, but the people from Google we have interacted
with on this list have NEVER done anything deceitful. They have discussed
with us changes in price, policy, features, and terms of service. They have
always been candid with us about legal issues. In terms of support you
could hardly ask for a better group. A group where every email has the
address of the person who sent it, so many of us have interacted off list.

Your thread is off base. If the GAE Team isn't hitting uptime, call them on
it. As you pointed out they fess up to that. Don't like the data on the
status or downtime board, point out that you want higher resolution.
Calling the team dishonest. Where I grew up calling someone a liar is the
kind of thing that gets your nose bloodied.

[trilok]

Brandon, read my original post - the dishonesty is regarding the fact
that one day they say that the service has disruption, and the next
they show that they don't. Not who's fault is that my app is down.

When I call support I want to know how long will it take them
approximately to solve the problem so I have something to tell my
clients. I'm a bloody paying customer, I deserve that, like my paying
customers deserve it from me. It is clear to me you don't run a B2B
business, so please don't comment on things you do not understand.

Since you fail to read my posts and understand them (read the first
line of this post) I can only ask you to mind your own business and
stop wasting my time with unrelated answers to my posts.

Can we SOMEHOW return to the subject at hand - Transparency? Or is
this a lost cause because of people like this guy?

[trilok]

Oh, and you are not an "Ass, a Troll, an Idiot". I respect your
excellent ambition to 'protect the innocent'. I agree my subject line
was inflammatory to get a response, which it got. I have nothing but
respect to the GAE team (as I said again and again and again).

I am sorry to tell you but I rechecked the configurations and all the
500's I get and non of them are our fault. But lets put that aside.

Now, Brandon, please tell me what will it take to return this
discussion to the transparency issue? Respect my problems with Google,
like people respect yours.

[WallyDD]

Could this be threatening behaviour? I'll let someone else decide.

Both of you. Knock. It. Off.

[Yoav Amit]

Excellent! Can I now plllllllllllllllllllease get someone at Google to comment on the transparency issue?

Is there any chance that next time GAE is at a very elevated status, as seldom as that may occur, we the paying users can get a description of the problem and a rough estimate of how long will it take to solve the problem during real-time? Just something so small to hang on to and pass to our frustratingly angry clients?

I am literally on my knees begging. I'll upload a pic of me begging if I can get 10 +1's on this :)

 - Yoav.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
 - Yoav

[Gregory D'alesandre]

Hi Yoav,

I don't feel this thread hasn't been hijacked, but I'm not sure what your question is at this point.  We typically do post an update when there is a system-wide issue and we typically do a post-mortem as well.  So if you question is "Can we post something when there are system-wide issues", the answer is, yes, we typically do and we will continue to do so.  

It seems that the question is around the definition of "system-wide issue", while you might've seen 3 issues that does not mean they were widespread problem.  The status dashboard is run from a small set of probers and so there are times it will show issues on the status dashboard when it was a very isolated problem or worse it was merely an issue with the prober.  Should we fix this so that the status site is accurate?  As I said previously, yes, we need a better way of doing this.  We have started offering Premier Accounts (for $500/month) that provide operational support so if you feel it is critical to your business to talk to someone when there is an issue, you should sign up for a Premier Account.  The fee for the premier account essentially pays for the people that you can contact about issues (this is not a way for us to increase profits, its a way for us to provide better service).

To be clear we are not going to post every single time there is any sort of disruption to any application.  It isn't feasible to do so for a massive multi-tenant distributed system running over 300,000 applications.  We will continue to work on trying to find the appropriate level so that everyone stays informed when there are issues.  But in the end there will be times that you feel as if you have an issue, maybe slightly elevated latency, and no one else is seeing a problem, this is what happens in large multi-tenant distributed systems.  It sometimes is something you've done and sometimes something going on with the serving infrastructure, if you want someone to contact who will reply within a certain amount of time, get a Premier Account.  That being said lots of App Engine team members monitor this list constantly (well, when we're conscious) and you might sometimes get a reply.

The reason that the status site says the number that it does is because it includes M/S and HRD apps so it is showing uptime for apps not under SLA.

Jeff is correct, whenever you assume we are insidiously trying to trick you, its probably a better assumption that there is a bug, or a misconfiguration or, even, very rarely, someone made a mistake, we are human.  Frankly, anyone assuming that we are trying to lie in order to hide issues says more about that person than it does about the App Engine team. 

Brandon is also correct in that you don't need to accuse us (or in this case, me) of flawed morals in order to get attention.  There are plenty of people we talk to all the time simply because they have a question.  If you accuse me of lying to you I will most likely take a more defensive stance than I would otherwise and ask for your evidence, again, I'm human.

Lastly, I sent a similar response about 24 hours ago, I thought I had answered your question at that point, so I'm not sure if this answers it or not.

TL;DR want someone to talk to when you are having issues, pay for a premier account.

Greg

[Andrius A]

Hi Greg,

can you please provide more information on you recent statement:

"The reason that the status site says the number that it does is because it includes M/S and HRD apps so it is showing uptime for apps not under SLA."

Does that mean that premier accounts get better uptime?

Thank you,

Andrius

[Brandon Wirtz]

No. It means that all the OLD M/S apps bring down the average.  (aka if you care about uptime don’t use M/S)

 

Premiere grants you support, AND if you have 300 apps instead of paying the per app minimum fee you pay for usage so if you have 300 apps that would cost $2 a month instead of the $10 a month minimum of ($3000) you pay the $500 + $600 in usage fees.   See you save $1900.

[Marcel Manz]

I would suggest that Google splits the current availability percentage in the system status into two values.

One related to applications using MS, the other related to applications using HR. A single value for the hole system somehow doesn't make sense, as this number won't be accurate to anyone, since no application can use MS and HR datastore at the same time.

One could now argue that it further should need to be split according the language that is been used (Java, Python, Go). I'm sure the Googlers find a way to better present an accurate number for certain application groups.

People who are using HR would like to see an over-all system status related to the HR datastore and not a much lower number just because MS apps are taking this value down. If the MS value is separately displayed, it should also motivate more people to migrate to HR.

Marcel

[Yoav Amit]

Gregory hi,

I'm sorry, I still don't see how next time an event like on the 8th of November happens (which, correct me if i'm wrong, was a system wide issue), I will have any tools to confront it. Your suggestion to pay the $500 fee, seems to be the only solution (also suggested here by Google's BizDev representatives in the GDD and later discussions). However, this and the custom domain SSL price makes GAE a very (very, very) expensive solution compared to EC2.

Also please note that there is still no information (even not just the basic of basic information) regarding the 8th of November spike. Transparency helps making your clients feel more assured of the product you are giving them.

If you wish to discuss this further, you may email me. However, needless to say, we have recently started working on transporting our system to EC2, so we have more control when such events happen (yes, on the price of handling our own IT).

Thank you all,

 - Yoav.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/k3J3OowyrUcJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
 - Yoav