query.fetch_page() doesn't always work as expected...

[Wade Holst]

Suppose I have the following method:

 def allElements(query, page_size=300, keys_only=False):

    result = []

    cursor = None

    more = True

    while more:

      models, cursor, more = query.fetch_page(

        page_size, keys_only=keys_only, start_cursor=cursor)

      result.extend(models)

    return result

Suppose further that my datastore contains 3281 instances of a Person class.  The following output would suggest that the validty of fetch_page() is contingent on the page_size passed in:

>>> q = Person.query()

>>> for N in [100, 200, 299, 300, 301, 400, 500, 599, 600, 601, 700, 800, 899, 900, 901, 1000]:

...   print '%4d = %5d' % (N, len(allElements(q, page_size=N, keys_only=True)))

  50 =  3281

 100 =  3281

 150 =  3281

 200 =  3281

 235 =  3281

 290 =  3281

 299 =  3281

 300 =  3281

 301 =  1781

 400 =  2000

 500 =  2081

 599 =  2381

 600 =  3281

 601 =  1803

 700 =  2081

 800 =  2081

 899 =  2081

 900 =  3281

 901 =  1802

1000 =  2000

As long as page_size is less than 300, or an integer multiple of 300, fetch_page returns all of the elements, but for any other value of page_size it returns only a subset.  This is happening in both ndb 0.9.3 and 0.9.4, haven't tested in other versions.

Wade

[Guido van Rossum]

I would like to know if this also fails with old DB or with datastore_rpc.py.

--Guido van Rossum (on Android)

[Guido van Rossum]

Following up after off-line exchanges: this is apparently caused by
the use of remote_api, so NDB users need not be concerned about using
fetch_page() in their apps.

--Guido

--
--Guido van Rossum (python.org/~guido)

[ben...@getcleverpet.com]

I'm using ndb, and I have the same problem with fetch_page. Not only this, but 'more' is returned as false at an even earlier stage in the traversal. I have no idea how to work with this.