Observed: put()'s completion time is dependent on time of day

[Richard]

I don't care it may not be dependent of time of day from an architecture point of view..... picture is observed behavior.

Y axis is number of players per 3 minute game.  X axis is 24 hour with time defined per sever time.  At the end of the round, all scores are gathered by F1's and saved in a lightweight object with a put().  A backend then reaps them and creates a leaderboard and deletes all the original lightweight db objects.  Any time the number of players jumps, it means we missed capturing submitted scores from Android clients.

[Richard]

And, of course.... including an image from Google docs does not actually work.   Grrr!

https://docs.google.com/drawings/pub?id=1nbrpRb7bSYIvS7329fY2jmynKCJOtUtlCMImo0wXnpE&w=1440&h=1080

[Takashi Matsuo]

Hi Richard,

I'm still not 100% sure, but my first guess is that it might be
because of datastore tablets splitting/merging behavior. When you
create or delete lots of entities, the datastore tablets might be
splited/merged into larger/fewer number of tablets. If that happens,
you may experience such high latencies.

This blog post by Ikai explains this issue very well:
http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/

Questions are:
How often those entities are created? How much QPS?
What kind of keys do you use for those small entities?

If you write entities at a somewhat high rate like >100QPS, and you're
using keys which are not distributed well, the culprit might be
splitting/merging behavior mentioned above.

Also, maybe you can consider using pull queues for such tasks.

-- Takashi

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/tGsoCfpFC_kJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.



--
Takashi Matsuo

[Richard]

Thank you for that interesting post.  Some points though:

 - behavior is observed at the same time each day
 - behavior is observed when QPS is lowest
 - behavior is not observed at high QPS

Object datamodel:

class Score(db.Model):
    a = db.StringProperty(indexed=False)
    b = db.ReferenceProperty(indexed=False)
    c = db.StringProperty(indexed=False)
    d = db.StringProperty(indexed=False)
    e = db.StringProperty(indexed=False)
    f = db.IntegerProperty(indexed=False)
    g = db.StringProperty(indexed=False)
    h = db.IntegerProperty(indexed=False)
    i = db.IntegerProperty(indexed=False)
    j = db.TextProperty(indexed=False)

I really don't think this is the problem.  Unless it is happening to another app on the same hardware, and I am seeing issues too ?  Not sure if that is possible ?

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Takashi Matsuo]

On Tue, Jul 31, 2012 at 11:08 PM, Richard <stev...@gmail.com> wrote:
> Thank you for that interesting post. Some points though:
>
> - behavior is observed at the same time each day
> - behavior is observed when QPS is lowest
> - behavior is not observed at high QPS

The traffic patterns seems identical, so isn't it possible that the
issue always happens after deleting bunch of entities?
What happens if you just leave those entities instead of deleting them?

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/Sgdd6z4nzPsJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Richard]

That might be... but sometimes it is just fine with 600+ players in a game.  Then the same time each day when it has only 180 players, it has problems ?  Also, this problem started 4 days ago......

I just tried pull queues in the last hour.  Some entries to the pull queue arrive later than 10 seconds after being added to the queue.  Also, when processing at around 160 players/game, the number of 'late' entries was randomly between 1-10.  Obviously, this is completely unacceptable as it appears to players that we 'lose' their score!

So it seems PULL queues are not for anything that requires performance (except maybe for batch?) and DEFINITELY NOT for timing....

[Takashi Matsuo]

Hi Richard,

How do you generate keys for those entities?
What is your app-id?
What time do you see this issue(It's not clear from the image, which
timezone you're in)?

-- Takashi

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/ZxqrFCQl0YsJ.

[Richard]

Hi Takashi,

App id is sven-wordhero.  I don't specify keys when creating the Score() entities.  Just:
x = Score()
x.member = thing
x.put()

Time is listed on the bottom plot and is actually from a datetime() on the server, so between 8am and 1pm UTC ?

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Takashi Matsuo]

Hi Richard,

Alright, I've been misunderstanding your issue. I thought you were
saying that the datastore put(s) are actually taking more than 10
secs. However, the issue you're tackling with is that, the datastore
put calls themselves succeed, but the subsequent global query doesn't
pick up the entities within 10 secs.

If that's the case, please forget about tablet split/merge for now.
It's a general eventually consistent problem. Let me think about it
again.

-- Takashi

>> > To post to this group, send email to google-a...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengi...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/P1y70hJ2BLIJ.
>
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengi...@googlegroups.com.

[Richard]

Hi Takashi,

This is exactly my problem.  Along with the fact that it seems to be time-of-day dependent.  Luckily/unluckily, this just happens to coincide with the time when I have the least number of users playing.

-R

>> > To post to this group, send email to google-appengine@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > google-appengine+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> Takashi Matsuo
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/P1y70hJ2BLIJ.
>

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Hernan Liendo]

We've also the same problem. We even had to implement a versioning controlling algorithm to prevent ugly dirty reads.

Our appid is 'racetownzc'