Very high latency since March 13th

[andidol]

Hi,

I have very high latency on my AppEngine since yesterday. It was really fast before < 100ms. Now there are a lot of requests performing with > 200ms without any reason. I did not update or did anything.

I'm using High Replication Datastore.

Thanks

[Ikai Lan (Google)]

We're addressing issues with high replication datastore applications. 

In the future, however, this information is typically not enough. What are you doing? Are you making datastore calls? Are you doing URLFetch? Are you making XMPP or email API calls? Are CPU intensive tasks slower? This is important because there are a lot of operations for which 200ms would seem low latency. If you're trying to brute force SHA1 hashes and it's taking 500ms on average, you're doing okay.

One of the fortunate things about working with software developers instead of normal internet consumers is that I can usually just give the guidance, "report issues that way you want your users to report issues or bugs", and it gets the message across =).

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/UKDaOm5bNJsJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[andidol]

I'm doing a Model.get_by_key_name().. on a model with ~50 million entries (I know this should not effect the query-time). The Model.get_by_key_name() method often takes more than 200ms for finding an entry, not every time but often. Not finding a key fails much faster, IMO this is ok. I think this is not normal, do you have any suggestions?

On Wednesday, March 14, 2012 8:01:26 PM UTC+1, Ikai Lan wrote:

We're addressing issues with high replication datastore applications. 

In the future, however, this information is typically not enough. What are you doing? Are you making datastore calls? Are you doing URLFetch? Are you making XMPP or email API calls? Are CPU intensive tasks slower? This is important because there are a lot of operations for which 200ms would seem low latency. If you're trying to brute force SHA1 hashes and it's taking 500ms on average, you're doing okay.

One of the fortunate things about working with software developers instead of normal internet consumers is that I can usually just give the guidance, "report issues that way you want your users to report issues or bugs", and it gets the message across =).

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

On Wed, Mar 14, 2012 at 5:48 AM, andidol <andreas....@gmail.com> wrote:

Hi,

I have very high latency on my AppEngine since yesterday. It was really fast before < 100ms. Now there are a lot of requests performing with > 200ms without any reason. I did not update or did anything.

I'm using High Replication Datastore.

Thanks

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/UKDaOm5bNJsJ.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Ikai Lan (Google)]

I see. Let's continue watching this to look for trends. Please post if it fluctuates either higher or lower.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/_OFHy-k7QE8J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Thomas]

Hi Ikai,

I've been writing in similar threads for about a week. Our app (id=typescout) lately experienced very high latency but was somehow performing better since Monday. Today again, performance broke down, and we see lots of requests taking over 50s or failing without any reason. 

I can remember, more than half a year ago appengine "stability" often was as bad as it was the last two weeks, but everything worked beautifully since then. Unfortunately, during the last two weeks, appengine performance was really horrible for us and our users.

Do you think this problems will be resolved by the scheduled maintenance on Monday 19th? We're using M/S, because we sometimes have lots of concurrent writes which would just not work well with HRD. At the same time, we tried to run our app with HRD and read/write quota was running up incredibly fast, about 10-50 times faster than M/S, so we decided not to migrate our app.

Thanks for your support!

Kind regards,

Thomas

[toonetown]

Looks like this is a big issue - there is another thread about it at
http://groups.google.com/group/google-appengine/browse_thread/thread/baf6af00b0a46709

I can also verify that this is happening to us. In looking through
the issue list, it appears that *SOMETHING* happened a couple weeks
ago, causing all sorts of people to have Deadline and HardDeadline
exceptions. In addition, since so many instances are starting and
stopping, it appears that performance has been hampered as well.
Also - in looking through the issue list, it seems that this is
mostly
affecting Java AppEngine instances - but similar problems appear to
be
happening in the Python instances as well.

From what I can tell in reading forums and postings, there was a
system update at some point on March 7th, 2012, and problems have
been
happening ever since.

Here is a list (from a quick scan - may not be comprehensive) of all
the bugs I have found that are having this issue:
http://code.google.com/p/googleappengine/issues/detail?id=7133
http://code.google.com/p/googleappengine/issues/detail?id=7130
http://code.google.com/p/googleappengine/issues/detail?id=7135
http://code.google.com/p/googleappengine/issues/detail?id=7138
http://code.google.com/p/googleappengine/issues/detail?id=7143
http://code.google.com/p/googleappengine/issues/detail?id=7146
http://code.google.com/p/googleappengine/issues/detail?id=7153
http://code.google.com/p/googleappengine/issues/detail?id=7093
http://code.google.com/p/googleappengine/issues/detail?id=7097
http://code.google.com/p/googleappengine/issues/detail?id=7101
http://code.google.com/p/googleappengine/issues/detail?id=7102
http://code.google.com/p/googleappengine/issues/detail?id=7103
http://code.google.com/p/googleappengine/issues/detail?id=7108
http://code.google.com/p/googleappengine/issues/detail?id=7113
http://code.google.com/p/googleappengine/issues/detail?id=7121
http://code.google.com/p/googleappengine/issues/detail?id=7122

I would suggest that we all vote for a single one of these - so that
it get get some visibility within Google. I think we should vote for
http://code.google.com/p/googleappengine/issues/detail?id=7133 since
as of today, it has the most stars already, and it is listed as
status
"Accepted" - which is better than all the other ones that are listed
as "New".

We are paying for AppEngine, we expect that there should be some
level
of support. This is ridiculous to have this problem going
on for as long as it has been, with no word on what is going on or
how
to resolve it.

(Yes - this is a cross-post...but *something* needs to be done to have
the visibility of this issue raised)

-Nathan

> > On Thu, Mar 15, 2012 at 4:19 AM, andidol <andreas.dolin...@gmail.com>wrote:
>
> >> I'm doing a Model.get_by_key_name().. on a model with ~50 million entries
> >> (I know this should not effect the query-time). The Model.get_by_key_name()
> >> method often takes more than 200ms for finding an entry, not every time but
> >> often. Not finding a key fails much faster, IMO this is ok. I think this is
> >> not normal, do you have any suggestions?
>
> >> On Wednesday, March 14, 2012 8:01:26 PM UTC+1, Ikai Lan wrote:
>
> >>> We're addressing issues with high replication datastore applications.
>
> >>> In the future, however, this information is typically not enough. What
> >>> are you doing? Are you making datastore calls? Are you doing URLFetch? Are
> >>> you making XMPP or email API calls? Are CPU intensive tasks slower? This is
> >>> important because there are a lot of operations for which 200ms would seem
> >>> low latency. If you're trying to brute force SHA1 hashes and it's taking
> >>> 500ms on average, you're doing okay.
>
> >>> One of the fortunate things about working with software developers
> >>> instead of normal internet consumers is that I can usually just give the
> >>> guidance, "report issues that way you want your users to report issues or
> >>> bugs", and it gets the message across =).
>
> >>> --
> >>> Ikai Lan
> >>> Developer Programs Engineer, Google App Engine
> >>> plus.ikailan.com
>

> >>> On Wed, Mar 14, 2012 at 5:48 AM, andidol <andreas.dolin...@gmail.com>wrote:
>
> >>>> Hi,
>
> >>>> I have very high latency on my AppEngine since yesterday. It was really
> >>>> fast before < 100ms. Now there are a lot of requests performing with >
> >>>> 200ms without any reason. I did not update or did anything.
> >>>> I'm using High Replication Datastore.
>
> >>>> Thanks
>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "Google App Engine" group.

> >>>> To view this discussion on the web visithttps://groups.google.com/d/**
> >>>> msg/google-appengine/-/**UKDaOm5bNJsJ<https://groups.google.com/d/msg/google-appengine/-/UKDaOm5bNJsJ>
> >>>> .
> >>>> To post to this group, send email to google-appengine@googlegroups.**
> >>>> com <google-a...@googlegroups.com>.

> >>>> To unsubscribe from this group, send email to

> >>>> google-appengine+unsubscribe@**googlegroups.com<google-appengine%2Bunsubscr i...@googlegroups.com>
> >>>> .
> >>>> For more options, visit this group athttp://groups.google.com/**
> >>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
> >>>> .

>
> >>>  --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To view this discussion on the web visit
> >>https://groups.google.com/d/msg/google-appengine/-/_OFHy-k7QE8J.
>

> >> To post to this group, send email to google-a...@googlegroups.com.

> >> To unsubscribe from this group, send email to

> >> google-appengi...@googlegroups.com.