More detailed informations about outages

[Flips]

If you want to know more about outages please star this feature
request:

http://code.google.com/p/googleappengine/issues/detail?id=2867

[Ikai L (Google)]

We have the downtime notify list for this specific purpose:

http://groups.google.com/group/google-appengine-downtime-notify

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
http://googleappengine.blogspot.com | http://twitter.com/app_engine

[Flips]

I want to know *more*. It is not very satisfying if you just confirm
the outage, wait an hour and tell us that there are new problems. Half
an hour later you announced that all applications are in read-only
mode and the next status update is that the problem is solved.

Specifically I want you to notify me if you know what the cause of the
problem is and tell me how much time you approximately need to fix it.

Problem Switch to the seccond datacenter

On Feb 24, 7:50 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> We have the downtime notify list for this specific purpose:
>
> http://groups.google.com/group/google-appengine-downtime-notify
>
>
>
>
>
> On Wed, Feb 24, 2010 at 9:28 AM, Flips <p...@script-network.com> wrote:
> > If you want to know more about outages please star this feature
> > request:
>
> >http://code.google.com/p/googleappengine/issues/detail?id=2867
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

[Nick Johnson (Google)]

Hi,

We've already committed to publishing a full postmortem on the most recent outage, as we did for our other major outage in July last year (https://groups.google.com/group/google-appengine/msg/ba95ded980c8c179?pli=1). Such reports take a while to prepare, however, and in the middle of a crisis, it's obviously necessary to prioritize fixing the issue over documenting it.

-Nick Johnson

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[Eli Jones]

I demand you post videos to You Tube of the GAE team frantically cursing, angrily casting blame, and furiously banging on their keyboards while investigating yesterdays outage.  Maybe grappling on the floor or throwing things.

"I TOLD you that relying on mulitmastermode hotspot failover for transcontinental datacenter backup sites WOULDN'T WORK!  Who unplugged all 200 superredundant fiber routers just to plug in their hot plate?"

And then possibly, personal essays from everyone involved describing their feelings during the event..  How did it make you feel?  Were you professionally detached and getting the job done (Determined.. like steel)?  Were you worried that this outage implied you've become a total failure, and it brought up all of these issues with your father and how he was going to call you up and say, "You're just like your mother!  Good for nothing... (grumble..).. not highly available.."?  Were you angry at someone else who you just knew was the reason this outage occurred?  

What did you eat that morning? (Was it morning, late night... evening?)  Were you eating right then?  Were you thinking about maybe eating?  Did you get an e-mail alert indicating System A was down?  And you're like dammit.. and then you got an e-mail alert saying Systems A and B failed in the failover process for System A.. and you're like stopping eating and going to a computer.. and then you get the phone call that all Backup Systems Ai, Bi for Primary Failover Systems Ai, Bi have failed and there is a wild hobogoblin loose in one of the offices and he's eaten the head of the Team App Engine Disaster Recovery (TEADR)? And you're like.. "Why didn't we purchase that Anti-Hobogoblin Sytem?  That's what I said.. I said, 'You never know... you never know when the hobogoblins strike... ..until its too late.."

And yes.. I want real time updates next time.. possibly a Disaster Recover Webcast.. maybe you could hire someone to follow team members around and live blog the whole thing.

I think, if you devote more of your resources to these things... real live customer service (instead of wasting time fixing things and improving the backend.. you need to shift a lot of these scarce resources to white glove customer appreciation b.s.)  We need more of that facade of direct eye contact, "Yes I understand your concerns"-reassurances instead of real work.  So.. when the backend doesn't get better because all of the resources slowly move towards this kabuki play business model.. and things start breaking.. We can feel better about it because the GAE team will be right there, telling us that.. they understand our frustrations.. and the issue is being taken very seriously indeed and everything is being done to fix the problem and don't worry it'll be okay.

Yes, yes.. I know that the reasonable, serious and well-considered response from the GAE Team is and will be:  "We can do better. And we will be even more transparent next time.  And we will keep working just as hard on the back end while now devoting even more energy to responding to people's e-mails about 'how am i supposed to sign up if I don't have a mobile phone'  or 'how do I do a join in GQL.. can I use a different database with appengine?' or maybe 'Why doesn't GAE support PHP? GAE must support PHP!' "

Anyway.. I look forward to the Post Mortem.. it's always instructive to read them.. and if you read between the lines, there's always a little dramaturgy that you can fill in.

[Juan Pablo Guereca]

LOL, I nerdly agree..

> <mailto:ika...@google.com>> wrote:
> > We have the downtime notify list for this specific purpose:
> >
> > http://groups.google.com/group/google-appengine-downtime-notify
> >
> >
> >
> >
> >
> > On Wed, Feb 24, 2010 at 9:28 AM, Flips
> <p...@script-network.com <mailto:p...@script-network.com>> wrote:
> > > If you want to know more about outages please star this
> feature
> > > request:
> >
> > >http://code.google.com/p/googleappengine/issues/detail?id=2867
> >
> > > --
> > > You received this message because you are subscribed to
> the Google Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to
> google-a...@googlegroups.com

> <mailto:google-a...@googlegroups.com>.

> > > To unsubscribe from this group, send email to
> > > google-appengi...@googlegroups.com

> <mailto:google-appengine%2Bunsu...@googlegroups.com><google-appengine%2Bunsubscrib
> e...@googlegroups.com <mailto:e...@googlegroups.com>>

> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
> >
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App
> Enginehttp://googleappengine.blogspot.com
> <http://googleappengine.blogspot.com>|http://twitter.com/app_engine
>
> --
> You received this message because you are subscribed to the
> Google Groups "Google App Engine" group.
> To post to this group, send email to
> google-a...@googlegroups.com

> <mailto:google-a...@googlegroups.com>.

> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com

> <mailto:google-appengine%2Bunsu...@googlegroups.com>.

> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>
>
> --

> Nick Johnson, Developer Programs Engineer, App Engine
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
> Number: 368047
> --
> You received this message because you are subscribed to the Google
> Groups "Google App Engine" group.
> To post to this group, send email to
> google-a...@googlegroups.com

> <mailto:google-a...@googlegroups.com>.

> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com

> <mailto:google-appengine%2Bunsu...@googlegroups.com>.

> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --

[Manny]

nerd rage is hilarious

> >https://groups.google.com/group/google-appengine/msg/ba95ded980c8c179...).

> >> > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

> >> e...@googlegroups.com>
> >> > > .
> >> > > For more options, visit this group at
> >> > >http://groups.google.com/group/google-appengine?hl=en.
>
> >> > --
> >> > Ikai Lan
> >> > Developer Programs Engineer, Google App Enginehttp://
> >> googleappengine.blogspot.com|http://twitter.com/app_engine
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-a...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> > --

> > Nick Johnson, Developer Programs Engineer, App Engine
> > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> > 368047
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to