is app engine down? I'm seeing a huge number of deadline errors and our app failing to server, hr datastore

[Kenneth]

nothing the status page.

the scheduler has given me about 10x my usual number of instances.

where's the sla form? I've put in a production issue.

thanks,

[Kenneth]

Seems to be back now, these thing always happen during important demos.  Nothing on the status page, must be my corner of app engine.

[Tom Willis]

I'm seeing it too actually. 

[ZeroCool]

My servers are also down.
Here is the app id: pe-server7

[Vivek Puri]

+1 . Complete app is down. Somone from AppEngine team, please look
into this asap.

[Felippe Bueno]

same here 

DeadlineExceededError: The API call datastore_v3.Put() took too long to respond and was cancelled.

Python 2.5/HR

On Fri, Jan 27, 2012 at 12:34 PM, Tom Willis <tom.w...@gmail.com> wrote:

I'm seeing it too actually. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uU9Sw0wDP1sJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kenneth]

Yeah, mine's gone again.

[Vivek Puri]

Come on guys, look into it and fix it. Also, please acknowledge that
someone from AppEngine team is looking into it.


On Jan 27, 9:46 am, Kenneth <kennet...@aladdinschools.com> wrote:
> Yeah, mine's gone again.

[Naresh Talluri]

Mine too, app-id : beta-sb

Thanks,
Naresh T

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Miroslav Genov]

+1

[peterk]

Having problems here for the last half hour. Can't update my apps, get
'Exception: Version not ready'. Apps are 500-ing after a long request
time with the warning that ' A serious problem was encountered with
the process that handled this request, causing it to exit.'

So, I don't think you're alone anyway. Problems still persisting here.

[Mark]

Seems like mine was down overnight (seems to be up now for me).
Looking at my dashbaord I only see activity for the last hour or so.
And I have far less new users than I should do. Not happy. I have
only been using appengine in production for the last 2 weeks or so.
Is this the kind of thing I should be expecting?

[Mark]

This is what my dashboard looks like http://tinypic.com/r/2yotlco/5 it
should have constant activity through 24 hrs.

My app id is bedbuzzserver

Please let us know what happened.

[ZeroCool]

Is there any compensation?
I'm asking because I didn't expect this when I'm paying $5000/month
for a service.
Do we have an SLA on HR based apps?

[pdknsk]

I've noticed this as well for about two hours.

[Mark]

I notice that now on the status page (http://code.google.com/status/
appengine) there is a green checkmark for today saying no significant
issues (is used to show an exclamation mark for Java). However if you
look at the charts for Java for today (http://code.google.com/status/
appengine/detail/serving-java/2012/01/27#ae-trust-detail-static-get-
large-nogzip-java-error_rate) you can see something serious happened.
To me, the green checkmark seems to be a bit deceiving.

Please can we have some comment from Google on this? Things seem to
be fixed now, which is good.. however my users received errors because
of this issue and I want to know what steps are being take to make
sure it doesn't happen again.

[dimi]

I've had a wake up call at 2AM (Australia) while my clients in Europe were affected by this problem.

If I recall it correctly, the community has had the same 'complaint' about very late information about a previous short down time. I guess the 'very late' is rather relative. 

I'd rather have correct information to provide to my customers, than incomplete info.

My main complaint is why you don't let this down time happen during the day (in Australia) so they don't have to wake me up.. haha :)

[Sean Kovacs]

My task queues are not getting worked. Just building up....using the
default queue. GAE is very inconsistent.

[Mark]

This seems to be happening again now. My dashboard has been reset
(only see activity from past 10 minutes and only have 1 instance
running) and you can see from the Java status charts (http://
code.google.com/status/appengine/detail/serving-java/2012/01/31#ae-
trust-detail-helloworld-get-java-latency) that something bad is
happening.

[Vivek Puri]

Its is happening again right now. All Task are stuck, and app serving
is also impacted.

[Kenneth]

I'm working fine although I did notice the dashboard graph detail was cut off. It seems have come back but with a missing hour.

If you are replying to this thread be sure to say if you're running MS or HR datastores.

[Olivier]

we also just had issues with two production applications using HR.

[Alexis]

Also get big issues simultaneously with all our apps 10 min ago.
Lot of errors, drop in requests/sec while peak of active instances.

http://code.google.com/status/appengine showed an "anomaly" for
Python, but now it's back to "Elevated" and I guess it'll disappear as
usually...

[Marzia Niccolai]

Hi,

Both today and yesterday, there was a networking issue in one of the datacenters serving High Replication applications that caused increased datastore latency and elevated error rates.  We responded by stopping app engine from serving out of the affected datacenter until the problem was resolved.

If you have a paid HRD app and you feel the SLA has been violated for your app, you can file a request for billing credits here:

https://docs.google.com/a/google.com/spreadsheet/formResponse?hl=en_US&formkey=dFk2SEVTd0lrS1d1M2I0S3Q5eFlWQnc6MQ

-Marzia

[Cezary Wagner]

I am feel that some week ago same was on M/S so maybe next version is
comming - whatever it should be more stable - I currently use M/S and
there is not problems planning to go to hrd but see that it is not
good time to do it in this week maybe :)

On Jan 31, 8:28 pm, Marzia Niccolai <marce+appeng...@google.com>
wrote:

> Hi,
>
> Both today and yesterday, there was a networking issue in one of the
> datacenters serving High Replication applications that caused increased
> datastore latency and elevated error rates.  We responded by stopping app
> engine from serving out of the affected datacenter until the problem was
> resolved.
>
> If you have a paid HRD app and you feel the SLA has been violated for your

> app, you can file a request for billing credits here:https://docs.google.com/a/google.com/spreadsheet/formResponse?hl=en_U...

>
> -Marzia
>
>
>
>
>
>
>
> On Tue, Jan 31, 2012 at 9:33 AM, Alexis <alexis.hanico...@gmail.com> wrote:
> > Also get big issues simultaneously with all our apps 10 min ago.
> > Lot of errors, drop in requests/sec while peak of active instances.
>

> >http://code.google.com/status/appengineshowed an "anomaly" for

[Phil Burk]

Hi Marzia,

Thanks for the status update. BTW, if the problem was so severe that
you had to shut down a datacenter and are offering billing credits,
then why does the status page today show no problems?

http://code.google.com/status/appengine

How bad do things have to get before they are listed on the problem
page? Maybe you could show a finer grained status. The two valued "No
significant issues" vs "Service disruption" does not communicate much.
Maybe you could use a series of emoticons that ranged from ecstatic to
happy, content, sad or suicidal. ;-)

Thanks,
Phil


On Jan 31, 11:28 am, Marzia Niccolai <marce+appeng...@google.com>
wrote:

> Both today and yesterday, there was a networking issue in one of the
> datacenters serving High Replication applications that caused increased
> datastore latency and elevated error rates.  We responded by stopping app
> engine from serving out of the affected datacenter until the problem was
> resolved.
>
> If you have a paid HRD app and you feel the SLA has been violated for your

> app, you can file a request for billing credits here:https://docs.google.com/a/google.com/spreadsheet/formResponse?hl=en_U...

>
> -Marzia
>
>
>
>
>
>
>
> On Tue, Jan 31, 2012 at 9:33 AM, Alexis <alexis.hanico...@gmail.com> wrote:
> > Also get big issues simultaneously with all our apps 10 min ago.
> > Lot of errors, drop in requests/sec while peak of active instances.
>

> >http://code.google.com/status/appengineshowed an "anomaly" for

[Brandon Wirtz]

My app was slower, but I didn’t have the 10% error rate specified by the SLA to qualify as downtime. Did you?  If you did, you can submit a ticket and I believe you will get a refund for the whole day’s billing.

 

Don’t hold me to that I haven’t read the SLA lately.

 

But you have to submit to receive a refund they aren’t automatic, and they  are for error rates over 10% of requests (I’m pretty sure)  so my app that is snappy just ran slow I don’t qualify.   If your app is slower it may.

 

 

Brandon Wirtz BlackWaterOps: President / Lead Mercenary
Work: 510-992-6548 Toll Free: 866-400-4536 IM: dra...@gmail.com (Google Talk) Skype: drakegreene YouTube: BlackWaterOpsDotCom BlackWater Ops Cloud On A String Mastermind Group

[JW]

I also experienced these issues, both last week and now (a few minutes ago).
This is how the requests / min chart looks in my dashboard:
http://i.imgur.com/q0Uax.png

It clearly shows how the cron job I am running once a minute wasn't
executed for 1 hour.

I am using the HR datastore.

2012/1/31 Olivier <o.mi...@gmail.com>:

> we also just had issues with two production applications using HR.
>

[Phong Luu]

Can anyone assist. My appengine instances are all down..from 21 to 1.

i still get Error: An error occurred for the API request datastore_v3.Get().

My app(yt-carousel-gadget.appspot.com) serves thousands of global users. Advertisers are asking me when this will be fixed.

thanks

[Phong Luu]

Is this problem reoccuring again today? Can anyone look into this ASAP as im serving thousands of users globally and it affects our advertisers.

app id(yt-carousel-gadget.appspot.com). Here is a Youtube channel where it exposes the error publically. http://www.youtube.com/user/danonecanal

thanks

  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
    return self.__get_result_hook(self)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1428, in __get_hook
    self.check_rpc_success(rpc)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1182, in check_rpc_success
    rpc.check_success()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 558, in check_success
    self.__rpc.CheckSuccess()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception 

[Phong Luu]

Hi,
Has these problems resurfaced? My app is down can anyone help urgently
with this?

Im getting lots of errors today. My appid is (yt-carousel-
gadget.appspot.com)

Error: An error occurred for the API request datastore_v3.Get().

On Jan 31, 7:28 pm, Marzia Niccolai <marce+appeng...@google.com>
wrote:

> Hi,
>
> Both today and yesterday, there was a networking issue in one of the
> datacenters serving High Replication applications that caused increased
> datastore latency and elevated error rates.  We responded by stopping app
> engine from serving out of the affected datacenter until the problem was
> resolved.
>
> If you have a paid HRD app and you feel the SLA has been violated for your

> app, you can file a request for billing credits here:https://docs.google.com/a/google.com/spreadsheet/formResponse?hl=en_U...

>
> -Marzia
>
>
>
>
>
>
>
> On Tue, Jan 31, 2012 at 9:33 AM, Alexis <alexis.hanico...@gmail.com> wrote:
> > Also get big issues simultaneously with all our apps 10 min ago.
> > Lot of errors, drop in requests/sec while peak of active instances.
>

> >http://code.google.com/status/appengineshowed an "anomaly" for

[Robert Kluin]

If this happens again in the future, you might want to file a production issue:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/45FXpuJI3jMJ.