Large difference between datastore quota detail and appstat.

[Timofey Koolin]

Datastore quota details show  about 40 thouthands read operations by last 5 hours (see attach).

AppStat show only 62 datastore_v3.Get, 155 datastore_v3.Next and 426 datastore_v3.RunQuery for last 4 hours. I have constant load - handle inboung email and sync it with external system.

--
Blog: www.rekby.ru

[Simon Knott]

How many entities are you pulling back in your queries?  Each query costs 1 read for the query, plus one read for each entity returned (I believe).

[Timofey Koolin]

...

Othen request can read up to 30 entity it execute about 350 times for screenshots in first email.

2011/11/18 Timofey Koolin <tim...@koolin.ru>

In query I always fetch only keys (about 40-60), then I fetch every entity by get_by_id, and almost all fetched from memcache (I use ndb).

My code:

db_keys = EMail.query(EMail.Status != 'Complete').fetch(keys_only=True)

keys = []

for key in db_keys:

    keys.append(key.id())

res = []

for key in keys:

  res.append(EMail.get_by_id(key))

app-stat timeline in attached screenshot.

It sync request execute every one minute. 

Othen request can read up to 30 entity it execute about 350 times for this screenshot. Code:

channels = ChannelInfo.query()

for channel in channels:

  ...

2011/11/18 Simon Knott <knott...@gmail.com>

How many entities are you pulling back in your queries?  Each query costs 1 read for the query, plus one read for each entity returned (I believe).

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4NruFnQsNmAJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Blog: www.rekby.ru

--
Blog: www.rekby.ru

[Timofey Koolin]

In query I always fetch only keys (about 40-60), then I fetch every entity by get_by_id, and almost all fetched from memcache (I use ndb).

My code:

db_keys = EMail.query(EMail.Status != 'Complete').fetch(keys_only=True)

keys = []

for key in db_keys:

    keys.append(key.id())

res = []

for key in keys:

  res.append(EMail.get_by_id(key))

app-stat timeline in attached screenshot.

It sync request execute every one minute. 

Othen request can read up to 30 entity it execute about 350 times for this screenshot. Code:

channels = ChannelInfo.query()

for channel in channels:

  ...

2011/11/18 Simon Knott <knott...@gmail.com>

How many entities are you pulling back in your queries?  Each query costs 1 read for the query, plus one read for each entity returned (I believe).

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/4NruFnQsNmAJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Blog: www.rekby.ru

[djidjadji]

Does ndb support lists in get_by_id()? This could then be executed in parallel.
You should then check with appstats on the GAE hardware if it works in parallel.

db_keys = EMail.query(EMail.Status != 'Complete').fetch(keys_only=True)

res = EMail.get_by_id( [k.id() for k in db_keys] )
list comprehensions are faster then append() calls