datastore timeout every time - on reads, not writes

[tayknight]

I have a problem. I'm getting datastore timeouts when doing reads. The
code finished about 5% of the time. The code looks like:

alerts = Alert.all().filter('expires >= ', datetime.datetime.now())
# ge active alerts
for alert in alerts:
#get the db.Keys from the ListProperty
zones = ZoneMaster.get(alert.zones)
for zone in zones:
if zone:
#get the users for this zone
if zone.siteusers:
us = SiteUser.get(zone.siteusers)
for u in us:
if u:
self.response.out.write(u.name + '<br />')

The Model looks like:
class Alert(db.Model):
effective = db.DateTimeProperty()
expires = db.DateTimeProperty()
zones = db.ListProperty(db.Key)

class ZoneMaster(db.Model):
siteusers = db.ListProperty(db.Key)

class SiteUser(db.Model):
name = db.StringProperty()
zone = db.ReferenceProperty(ZoneMaster)

This code is repeatably timing out with a "Timeout: datastore timeout:
operation took too long." error.
I'm not doing any writes. All the reads are by key (that come from a
ListProperty). Why would this be timing out?

Thanks.

[tayknight]

And, I should add, this works perfecty (and quickly) from the
development server's datastore.

[Liang Zhao]

Each request can only run 30 seconds in server side,

but in development server, there is no limitation on it...

--

Cheers!

Liang Zhao

[tayknight]

Data is returned in development almost instantly. In production, the
error is returned after about 4 seconds. I get the datastore error
long before 30 seconds.

> Liang Zhao- Hide quoted text -
>
> - Show quoted text -

[tayknight]

I figured out that I needed to do a

alerts = Alert.all().filter('expires >= ', datetime.datetime.now

()).fetch(1000)

Apparently it is faster to do a fetch() than iterate over the values.

[Liang Zhao]

I'm not sure, but it seems both

alerts = Alert.all().filter('expires >= ', datetime.datetime.now())

and

alerts = Alert.all().filter('expires >= ', datetime.datetime.now()).fetch(1000)

should be the same, since according to documents, app engine only
return the first 1000 records.

--

Cheers!

Liang Zhao

[Wooble]

On May 1, 12:28 pm, Liang Zhao <alpha....@gmail.com> wrote:
> I'm not sure, but it seems both
>
> alerts = Alert.all().filter('expires >= ', datetime.datetime.now())
>
> and
>
> alerts = Alert.all().filter('expires >= ', datetime.datetime.now()).fetch(1000)
>
> should be the same, since according to documents, app engine only
> return the first 1000 records.

Fetching the records as a batch and then doing stuff with them is
faster than treating a db.Query object as an iterable.

On the other hand, fetch(1000) is fairly likely to get you a datastore
timeout, depending on your data.