strange behavior memcache/taskqueue/datastore

[Felippe Bueno]

Hello friends,

I'm getting a strange task queue/datastore/memcache behavior.

I have something like the following code:

report = memcache.get(id)

if not report: 

  try:

   if not deferred.defer(self.addindex, id=id): 

      logging.error('cannot store task for report index')

      report = False

   else:

      memcache.set(id, 1)

 except:

  logging.error('cannot store task for report index')

  report=False

def addindex(id)

 reportindex=Foo(id=id)

 try:

  reportindex.put()

 except Exception, e:

  logging.error('can not put report index')

When I can not put 'reportindex' I'm getting the logging.error as expected.

But some times, I don't get the error, and don't have related id into the DS.

To prove that was an error, I wrote to run every hour something like:

for x in range(startid, lastid):

  if memcache.get('%s' % x) and not Foo.all().filter('id', x).fetch(1):

    logging.error('there is no register for id %s' %x)

    a=Foo(id=x)

    a.put()

    

The problem is that I need Foo() to "index" memcache.

Is this an expected behavior ?

Should I trust on deferred.defer() ?

Thanks a lot

Felippe Bueno

[Nicholas Verne]

How long after the call to deferred.defer are you running the loop
that checks that datastore has a record for each value in memcache?

Perhaps you're running the loop before a deferred task has had a
chance to run, in some cases?

Nick Verne

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Felippe Bueno]

Hi Nicholas, 

Thanks for your reply.

This was one of my concerns. Even running my loop only one time per hour.

So, to prove again that there is a strange behavior , I changed the loop to:

 for x in range(startid, lastid):

   r=memcache.get('%s' % x)

   if r and r > 50 and not Foo.all().filter('id', x).fetch(1):

     logging.error('there is no register for id %s' %x)

     a=Foo(id=x)

     a.put()

It's ok to assume that I will not have 50 requests (each request do a memcache.incr()) in less than 3 minutes. 

Doing this way, it seems even more, that there is something wrong. I'm not sure if it is me that doing it wrong :-).

thatnks again.

[Nicholas Verne]

What's your app id?

Nick Verne

[Felippe Bueno]

Hi Nick.

I sent it to you in a private message.

Thank you very much.

[Kaan Soral]

my brain melted while trying to understand

[Nicholas Verne]

I suspect your deferred tasks are running, but some datastore writes
are committed but not yet applied when your .all() query is reading.

http://code.google.com/appengine/docs/python/datastore/overview.html#Understanding_Datastore_Writes

Nick Verne

On Thu, Jan 5, 2012 at 8:21 AM, Kaan Soral <kaan...@gmail.com> wrote:
> my brain melted while trying to understand
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/AJ0qu1I9L9AJ.