Large latency spike, need assistance

[Nathan Skone]

Application: hs-hbo

Datastore: High Replication

Normal latencies: 50ms-200ms

Today's latencies: 5000ms-10000ms

Idle Instances: ( Automatic – Automatic )

Pending Latency: ( Automatic – Automatic )

Dear Appengine Team,

This morning the latency of my application saw a sudden spike that has made it unusable for my company's purposes. How can I get assistance with this problem? This is an urgent issue that is directly effecting our customers.

Thank you,

Nathan Skone

DYMO / Mimio - A Newell Rubbermaid Company

[Takashi Matsuo]

Hi Nathan,

I think it's OK now. Are you still seeing this?

BTW, this list is not a support channel with any kind of SLA. Now
we're offering premier support for that type of demand.
For more details about our premier support, please see:
https://developers.google.com/appengine/docs/premier/

Regards,

-- Takashi

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/O-GXusXXlzsJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



--
Takashi Matsuo | Developer Advocate | tma...@google.com

[alex]

This is not directly related to the original post, but speaking of SLA, I was just re-reading it (https://developers.google.com/appengine/sla) and this got me thinking:

"Eligible Application" means an Application that has been created by Customer using the High Replication Datastore setting.

Does this actually include *any* app, even free quota HRD apps? 

"Customer" term is not defined anywhere in the SLA (unless I missed it) so, I'm not really sure whether Customer means Billing Administrator (for a paid app) or it now relates to App Engine Premier Account only, or both.

Thanks,

Alex.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Strom]

Does it matter? 50% of zero is still zero, so you won't get anything.

On Apr 26, 10:58 am, alex <a...@cloudware.it> wrote:
> This is not directly related to the original post, but speaking of SLA, I
> was just re-reading it (https://developers.google.com/appengine/sla) and
> this got me thinking:
>

> "*Eligible Application*" means an Application that has been created by

> > > To post to this group, send email to google-a...@googlegroups.com.

> > > To unsubscribe from this group, send email to

> > > google-appengi...@googlegroups.com.

> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
>
> > --

> > Takashi Matsuo | Developer Advocate | tmat...@google.com

[Nathan Skone]

Takashi,

The latency spike stopped around 11:30pm PST. Can you tell what caused the high latency, and if it is likely to occur again in the future? Occasional spikes such as what happened yesterday would make the Google App Engine much less useful to my company.

I understand there is no SLA support channel for non-premium accounts. Does that mean that us paying customers that cannot justify the extra $500 monthly fee cannot depend on any support from Google when experiencing problems?

Thank you for your response,

Nathan Skone

DYMO / Mimio - A Newell Rubbermaid Company

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Takashi Matsuo]

Hi Nathan,

On Fri, Apr 27, 2012 at 1:17 AM, Nathan Skone <nsk...@headsprout.com> wrote:
> Takashi,
>
> The latency spike stopped around 11:30pm PST. Can you tell what caused the
> high latency, and if it is likely to occur again in the future? Occasional
> spikes such as what happened yesterday would make the Google App Engine much
> less useful to my company.

First, are you really sure that the cause is in our side? Do you have
any appstats results which show that any of your RPC calls don't take
longer time than usual? If your RPCs take time, there are several
things you can do to mitigate this. Do you set any deadline on your
datastore calls? If no, you may want to set it appropriately, and when
hitting the deadline, you can return a failure to your web clients and
tell them to retry. Are you using urlfetch service to retrieve
external resources? If so, sometimes those external resources can be
the culprits. If you entirely rely your app's performance on the
memcache service, which has no SLA, your app might see high latency
when the memcache is flushed.

In this particular case, as far as I know, there was no significant
system issue around that time, so I don't think this was a system wide
issue, and in such cases, please understand that we can not offer
reports like that every time to every customers who experienced high
latency(again, premier customers are different, at least to my
knowledge).

> I understand there is no SLA support channel for non-premium accounts. Does
> that mean that us paying customers that cannot justify the extra $500
> monthly fee cannot depend on any support from Google when experiencing
> problems?

No, not at all. There are still several options. You can use a new
feature for reporting production issues in your admin console. You
should be able to see a link 'Report Production Issue' on the right
top side of your admin console, where you can report a production
issue alongside of a screenshot with some highlights and privacy
masks. That way, now you can report issues to us privately without
revealing your app-id.

Of course, you can also post here, then we offer a best effort
support, like this ;)

-- Takashi

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo | Developer Advocate | tma...@google.com
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/omv0xvTjHnwJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Nathan Skone]

Takashi,

The latency spike has hit again. This time I have appstats turned on, which is revealing some of the large latencies.  For example, here is a screenshot from appstats showing that a db.key on a single key object taking over 7 seconds!

Also attached is a screenshot showing a single memcache.get (on a single key) taking over 1 second.

What is causing this problem and how can I make it go away?

Thanks,

Nathan

>> > To post to this group, send email to google-appengine@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengine+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo | Developer Advocate | tma...@google.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/omv0xvTjHnwJ.
>

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Nathan Skone]

I forgot to attach the second screenshot, here is it.

[Barry Hunter]

Isn't this thing par for the course. They can happen occasionally.

Eg.
https://developers.google.com/appengine/articles/handling_datastore_errors

>> >> > google-a...@googlegroups.com.

>> >> > To unsubscribe from this group, send email to

>> >> > google-appengi...@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/google-appengine?hl=en.
>> >>
>> >>
>> >>
>> >> --
>> >> Takashi Matsuo | Developer Advocate | tma...@google.com
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Google App Engine" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/google-appengine/-/omv0xvTjHnwJ.
>> >

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo | Developer Advocate | tma...@google.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/bqfDa8Un9ewJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Nathan Skone]

The high latencies are still a problem for us today. We have had hours of unacceptably slow datastore and memcache operations. In addition, today we are receiving "Uncaught application errors" randomly that do not show up in our admin console logs. This is making the application totally unusable for us.

Thanks,

Nathan

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

[Andrin von Rechenberg]

+1

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lEmQChO7LRgJ.

[Andrin von Rechenberg]

app-id: miumeet-hr

[Ronoaldo José de Lana Pereira]

Hello Takashi,

Is there a way to set a deadline for memcache? Does the pattern:

get from cache:

  if not there, get from datastore:

    store in cache

.. in a scenario where the memcache is down and the timeout to see that it is down will degrade the app performance instead of make it better?

As we can see on the status dashboard, there was a huge increase in memcache and datastore latency:

   

>> > To post to this group, send email to google-appengine@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengine+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo | Developer Advocate | tma...@google.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/omv0xvTjHnwJ.
>

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Nathan Skone]

For some reason the latency logs in my admin console were cleared, which made proving the latency spike much more difficult... here is a before and after picture.

Thanks,

Nathan

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

[Renan Mobile]

Hi... i now that M/S is depreciated, but it's latency is too high.... like everyone at this post complain.

Thanks,

-----
CEO Renan Franca

2012/5/9 Nathan Skone <nsk...@headsprout.com>

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/BxIoqA4_88MJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Takashi Matsuo]

Hi Ronoaldo,

A good question!

Looking through the Python memcache API, some methods like get_multi_async() or set_multi_async() on the Client class receive rpc object as a keyword argument, so you can pass an rpc object with a particular deadline. For more details, please refer to our doc at:

https://developers.google.com/appengine/docs/python/memcache/clientclass

Thanks,

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bFIsUaUnxScJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Nathan Skone]

I am still waiting for a explanation of this problem.  I have a bug report open here, and ie...@google.com responded six days ago claiming they would look into the issue.  However, we have heard nothing since then.  Meanwhile, we have run into other unexplained reliability issues explained here.

Is anyone else experiencing the same low level of reliability we are? Also, does anyone have any advice on how to get technical support from Google when experiencing connectivity issues?  I am aware of the premium support plan, but the price is excessive for us compared to our regular Google App Engine bill.

Thank you,

Nathan Skone

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

[Nathan Skone]

This problem has reoccurred and is currently in effect on our application.  The graphs clearly show the problem.  The errors per second graph is showing that errors are happening, even though no errors are being displayed in the logs.

The lack of technical support and reoccurrence of this problem is proving the unreliability of the Google App Engine.

On Wednesday, April 25, 2012 10:43:51 AM UTC-7, Nathan Skone wrote:

[Nathan Skone]

This error is reoccurring again this morning. Google's unreliable processing speed bug is clearly not fixed yet.

Thanks,

Nathan

[Cesium]

Nathan,

You called it a "processing speed bug".

That feels like what I am seeing too.

An almost trivial servlet that usually takes 30 milliseconds to run is taking as much as 2000 milliseconds.

This is negatively impacting my capitalistic endeavors.

David