Inconsistencies in remote api data retrieval

[bsb]

I am trying to cycle through all elements of a table via the remote
API, using a modification of the code snippet from
http://code.google.com/appengine/articles/remote_api.html:

#---------------------------------------------------- SNIP
----------------------------------------------------
query = db.GqlQuery("SELECT __key__ FROM Model_mobileapp ORDER BY
__key__")
apps = query.fetch(500)
total = 0
while apps:
total = total + len(apps)
query2 = db.GqlQuery("SELECT __key__ FROM Model_mobileapp WHERE
__key__ > :1 ORDER BY __key__", apps[-1])
apps = query2.fetch(500)

print "%d" % total
#---------------------------------------------------- SNAP
----------------------------------------------------

However, I'm getting unpredictable behaviour. Sometimes I get the
correct number (104512), sometimes I get 15999, 30999 and all sorts of
other values.

Does anyone have any clue what this could be caused by? Is there some
sort of magic caching going on for the fetch() command, that return
some keys which are not in the correct sorting order? Could the
(automatic) ascending key index be broken? Any ideas?

Thanks,
Ben

[bsb]

No ideas? Noone? I just don't understand why not all keys are
retrieved every time. It doesn't make any sense...

Thanks in advance,
Ben

On Dec 8, 6:47 pm, bsb <b...@pearcomp.com> wrote:
> I am trying to cycle through all elements of a table via the remote

> API, using a modification of the code snippet fromhttp://code.google.com/appengine/articles/remote_api.html:

[Eli Jones]

Maybe one of the fetches times out and it just silently fails.. And
prints out its current count?

> --
>
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
>

--
Sent from my mobile device

[bsb]

I doubt that, I did get some explixcit timeouts from the fetch, which
went away when I reduced the limit from 500 to 300. So, I think if it
was a timeout, I'd get an exception thrown, no?

Best,
ben

[bsb]

I dug a bit deeper, as nobody seems to have a clue either. I printed
all keynames that I retrieved from the datastore in the order they
were retrieved. I repeated this 3 times, each time calling the exact
same code. What turns out to happen is that occasionally, the GqlQuery
returns the keys in the wrong order! Suddenly, I get the LAST key in
the table and all following keys in DESCENDING order. When the
GqlQuery randomly flips back to normal, it then goes in ASCENDING
order but from almost at the end of the table.

The peculiar thing is that this seems to happen at a particular
position in the table each time. The start of the key reversing seems
random, but the last 500 are always the same. I'm wondering if the
index is broken and that causes the problem, cause I seriously can't
think of any other explanation.

Just to say, I have a custom descending index on the key_name for this
class, if that makes any difference.

Can someone from Google please look into this?

Thanks,
Ben

[Eli Jones]

It may be related to you mixing usage of their filter method from the page you first linked to (the apps[-1] method)... and GQLQueries.

Try the method from this page:

http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html

They are casting the last key grabbed to a string before using it to grab the next GQL batch.. like so:

if len(entities) == 21:

    new_last_key_str = str(entities[19].key())

Or, use the filtering method they use on the remote api page you link to.  Mixing them together may be what is causing your problem.

[Eli Jones]

..or.. the fix may be as simple as changing "apps[-1]" to "apps[-1].key()" in your query2 line.

On Sat, Dec 12, 2009 at 6:18 PM, bsb <b...@pearcomp.com> wrote:

[bsb]

I think they are casting to a string because the method is meant for
paging from a website, so the string is passed back and used in the
next request, when it is converted back into a Key object. Overall, it
looks exactly like what I'm doing.

As for your second suggestion: The query already returns Keys, apps
[-1] IS a Key. calling .key() on it will raise an exception. I
should've probably named the variable "keys" and not "apps".

Anyway, thanks for the effort,

Ben

> > > > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > > > --
> > > > Sent from my mobile device
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[Robin Houston]

I'm seeing the same problem. I've filed a bug report at
http://code.google.com/p/googleappengine/issues/detail?id=2577

Robin