Since 7:53am PST, App Engine has been experiencing an unexpected outage affecting the majority of App Engine applications. The team is working quickly to correct the cause and will have an ETA on the fix shortly.
Please watch this thread for updates. We sincerely apologies for the inconvenience.
We are still actively working on the on-going outage. We've also experienced a problem with our backup datacenter. We will continue to provide status updates on this thread every thirty minutes.
On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> wrote:
> Since 7:53am PST, App Engine has been experiencing an unexpected > outage affecting the majority of App Engine applications. The team is > working quickly to correct the cause and will have an ETA on the fix > shortly.
> Please watch this thread for updates. We sincerely apologies for the > inconvenience.
> We are still actively working on the on-going outage. We've also > experienced a problem with our backup datacenter. We will continue to > provide status updates on this thread every thirty minutes.
> On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > Since 7:53am PST, App Engine has been experiencing an unexpected > > outage affecting the majority of App Engine applications. The team is > > working quickly to correct the cause and will have an ETA on the fix > > shortly.
> > Please watch this thread for updates. We sincerely apologies for the > > inconvenience.
As of 10:09am, all application should be serving traffic and successfully reading/writing from the datastore. Again, apologies for the inconvenience of the outage. We will follow up with a post- mortem.
On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> wrote:
> As of 9:48am, all applications should now be available in read-only > mode.
> On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > We are still actively working on the on-going outage. We've also > > experienced a problem with our backup datacenter. We will continue to > > provide status updates on this thread every thirty minutes.
> > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > outage affecting the majority of App Engine applications. The team is > > > working quickly to correct the cause and will have an ETA on the fix > > > shortly.
> > > Please watch this thread for updates. We sincerely apologies for the > > > inconvenience.
As of 1:00pm PST, we are seeing lingering write issues with a small percentage of datastore entity groups. As well, a small percentage of cron jobs are not running correctly. We believe both issues are linked to the unforeseen outage earlier today and we are treating them as high priority. We will update this thread hourly with more information going forward.
On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> wrote:
> As of 10:09am, all application should be serving traffic and > successfully reading/writing from the datastore. Again, apologies for > the inconvenience of the outage. We will follow up with a post- > mortem.
> On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 9:48am, all applications should now be available in read-only > > mode.
> > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > We are still actively working on the on-going outage. We've also > > > experienced a problem with our backup datacenter. We will continue to > > > provide status updates on this thread every thirty minutes.
> > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > outage affecting the majority of App Engine applications. The team is > > > > working quickly to correct the cause and will have an ETA on the fix > > > > shortly.
> > > > Please watch this thread for updates. We sincerely apologies for the > > > > inconvenience.
> As of 1:00pm PST, we are seeing lingering write issues with a small > percentage of datastore entity groups. As well, a small percentage of > cron jobs are not running correctly. We believe both issues are > linked to the unforeseen outage earlier today and we are treating them > as high priority. We will update this thread hourly with more > information going forward.
> On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 10:09am, all application should be serving traffic and > > successfully reading/writing from the datastore. Again, apologies for > > the inconvenience of the outage. We will follow up with a post- > > mortem.
> > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 9:48am, all applications should now be available in read-only > > > mode.
> > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > We are still actively working on the on-going outage. We've also > > > > experienced a problem with our backup datacenter. We will continue to > > > > provide status updates on this thread every thirty minutes.
> > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > outage affecting the majority of App Engine applications. The team is > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > shortly.
> > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > inconvenience.
> As of 2:00pm PST, we are continuing to investigate the lingering > datastore and cron issues. We will continue to update this thread as > we progress.
> On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 1:00pm PST, we are seeing lingering write issues with a small > > percentage of datastore entity groups. As well, a small percentage of > > cron jobs are not running correctly. We believe both issues are > > linked to the unforeseen outage earlier today and we are treating them > > as high priority. We will update this thread hourly with more > > information going forward.
> > On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 10:09am, all application should be serving traffic and > > > successfully reading/writing from the datastore. Again, apologies for > > > the inconvenience of the outage. We will follow up with a post- > > > mortem.
> > > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > As of 9:48am, all applications should now be available in read-only > > > > mode.
> > > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > We are still actively working on the on-going outage. We've also > > > > > experienced a problem with our backup datacenter. We will continue to > > > > > provide status updates on this thread every thirty minutes.
> > > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > > wrote:
> > > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > > outage affecting the majority of App Engine applications. The team is > > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > > shortly.
> > > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > > inconvenience.
> As of 3:00pm PST, we have determined the datastore writes and only > reads within transactions are affected. We are working on a fix for > the issue.
> On Feb 24, 2:13 pm, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 2:00pm PST, we are continuing to investigate the lingering > > datastore and cron issues. We will continue to update this thread as > > we progress.
> > On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 1:00pm PST, we are seeing lingering write issues with a small > > > percentage of datastore entity groups. As well, a small percentage of > > > cron jobs are not running correctly. We believe both issues are > > > linked to the unforeseen outage earlier today and we are treating them > > > as high priority. We will update this thread hourly with more > > > information going forward.
> > > On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > As of 10:09am, all application should be serving traffic and > > > > successfully reading/writing from the datastore. Again, apologies for > > > > the inconvenience of the outage. We will follow up with a post- > > > > mortem.
> > > > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > As of 9:48am, all applications should now be available in read-only > > > > > mode.
> > > > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > > > wrote:
> > > > > > We are still actively working on the on-going outage. We've also > > > > > > experienced a problem with our backup datacenter. We will continue to > > > > > > provide status updates on this thread every thirty minutes.
> > > > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > wrote:
> > > > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > > > outage affecting the majority of App Engine applications. The team is > > > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > > > shortly.
> > > > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > > > inconvenience.
> As of 6:00pm PST, this issue has been limited to a very small set of > existing entity groups. This thread will be updated once we have made > the fix.
> On Feb 24, 3:01 pm, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 3:00pm PST, we have determined the datastore writes and only > > reads within transactions are affected. We are working on a fix for > > the issue.
> > On Feb 24, 2:13 pm, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 2:00pm PST, we are continuing to investigate the lingering > > > datastore and cron issues. We will continue to update this thread as > > > we progress.
> > > On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > As of 1:00pm PST, we are seeing lingering write issues with a small > > > > percentage of datastore entity groups. As well, a small percentage of > > > > cron jobs are not running correctly. We believe both issues are > > > > linked to the unforeseen outage earlier today and we are treating them > > > > as high priority. We will update this thread hourly with more > > > > information going forward.
> > > > On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > As of 10:09am, all application should be serving traffic and > > > > > successfully reading/writing from the datastore. Again, apologies for > > > > > the inconvenience of the outage. We will follow up with a post- > > > > > mortem.
> > > > > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > > > > wrote:
> > > > > > As of 9:48am, all applications should now be available in read-only > > > > > > mode.
> > > > > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > wrote:
> > > > > > > We are still actively working on the on-going outage. We've also > > > > > > > experienced a problem with our backup datacenter. We will continue to > > > > > > > provide status updates on this thread every thirty minutes.
> > > > > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > wrote:
> > > > > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > > > > outage affecting the majority of App Engine applications. The team is > > > > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > > > > shortly.
> > > > > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > > > > inconvenience.
> Note that at 3pm, we disabled billing on all App Engine applications. > We will also not be charging for any of today's usage before the > outage.
> On Feb 24, 6:20 pm, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > As of 6:00pm PST, this issue has been limited to a very small set of > > existing entity groups. This thread will be updated once we have made > > the fix.
> > On Feb 24, 3:01 pm, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 3:00pm PST, we have determined the datastore writes and only > > > reads within transactions are affected. We are working on a fix for > > > the issue.
> > > On Feb 24, 2:13 pm, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > As of 2:00pm PST, we are continuing to investigate the lingering > > > > datastore and cron issues. We will continue to update this thread as > > > > we progress.
> > > > On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > As of 1:00pm PST, we are seeing lingering write issues with a small > > > > > percentage of datastore entity groups. As well, a small percentage of > > > > > cron jobs are not running correctly. We believe both issues are > > > > > linked to the unforeseen outage earlier today and we are treating them > > > > > as high priority. We will update this thread hourly with more > > > > > information going forward.
> > > > > On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > > > > > wrote:
> > > > > > As of 10:09am, all application should be serving traffic and > > > > > > successfully reading/writing from the datastore. Again, apologies for > > > > > > the inconvenience of the outage. We will follow up with a post- > > > > > > mortem.
> > > > > > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > wrote:
> > > > > > > As of 9:48am, all applications should now be available in read-only > > > > > > > mode.
> > > > > > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > wrote:
> > > > > > > > We are still actively working on the on-going outage. We've also > > > > > > > > experienced a problem with our backup datacenter. We will continue to > > > > > > > > provide status updates on this thread every thirty minutes.
> > > > > > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > > wrote:
> > > > > > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > > > > > outage affecting the majority of App Engine applications. The team is > > > > > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > > > > > shortly.
> > > > > > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > > > > > inconvenience.
We continue to work diligently in the wake of yesterday's unforeseen App Engine outage on a number of issues, so we wanted to provide you an update with our current list of tasks.
- Applications will not be charged for any App Engine resource usage that occurred during Feb 24th, 2010.
- We are still working on fix for writes and transactional reads affecting a small number of datastore entity groups. We have determined that approximately 25 application IDs have been affected by this issue. If you feel you are having difficulties with certain entity groups in your application, please respond to this thread with your App ID.
- We are working hard on putting together a detailed post mortem that we will be making public. We expect to have this available to you next week.
Again we sincerely apologize for yesterday's service disruption. We take App Engine's reliability very seriously and we are confident we can use the hard lessons learned from yesterday's event to improve our offering to you.
On Feb 25, 12:33 pm, App Engine Team <appengine.nore...@gmail.com> wrote:
> As of 11:00 a.m. PST, billing should be re-enabled and functioning > normally for all applications.
> On Feb 24, 7:18 pm, App Engine Team <appengine.nore...@gmail.com> > wrote:
> > Note that at 3pm, we disabled billing on all App Engine applications. > > We will also not be charging for any of today's usage before the > > outage.
> > On Feb 24, 6:20 pm, App Engine Team <appengine.nore...@gmail.com> > > wrote:
> > > As of 6:00pm PST, this issue has been limited to a very small set of > > > existing entity groups. This thread will be updated once we have made > > > the fix.
> > > On Feb 24, 3:01 pm, App Engine Team <appengine.nore...@gmail.com> > > > wrote:
> > > > As of 3:00pm PST, we have determined the datastore writes and only > > > > reads within transactions are affected. We are working on a fix for > > > > the issue.
> > > > On Feb 24, 2:13 pm, App Engine Team <appengine.nore...@gmail.com> > > > > wrote:
> > > > > As of 2:00pm PST, we are continuing to investigate the lingering > > > > > datastore and cron issues. We will continue to update this thread as > > > > > we progress.
> > > > > On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com> > > > > > wrote:
> > > > > > As of 1:00pm PST, we are seeing lingering write issues with a small > > > > > > percentage of datastore entity groups. As well, a small percentage of > > > > > > cron jobs are not running correctly. We believe both issues are > > > > > > linked to the unforeseen outage earlier today and we are treating them > > > > > > as high priority. We will update this thread hourly with more > > > > > > information going forward.
> > > > > > On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > wrote:
> > > > > > > As of 10:09am, all application should be serving traffic and > > > > > > > successfully reading/writing from the datastore. Again, apologies for > > > > > > > the inconvenience of the outage. We will follow up with a post- > > > > > > > mortem.
> > > > > > > On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > wrote:
> > > > > > > > As of 9:48am, all applications should now be available in read-only > > > > > > > > mode.
> > > > > > > > On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > > wrote:
> > > > > > > > > We are still actively working on the on-going outage. We've also > > > > > > > > > experienced a problem with our backup datacenter. We will continue to > > > > > > > > > provide status updates on this thread every thirty minutes.
> > > > > > > > > On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com> > > > > > > > > > wrote:
> > > > > > > > > > Since 7:53am PST, App Engine has been experiencing an unexpected > > > > > > > > > > outage affecting the majority of App Engine applications. The team is > > > > > > > > > > working quickly to correct the cause and will have an ETA on the fix > > > > > > > > > > shortly.
> > > > > > > > > > Please watch this thread for updates. We sincerely apologies for the > > > > > > > > > > inconvenience.
