Best practices implementing paging

[Abel Rodriguez]

I am currently doing a detailed study on how to optimize my
application to be scalable.

I have thoroughly studied the lectures delivered by the engineers at
google, and my opinion: fantastic

In this case, my question comes from the presentation

Buildding Scalable Web Applications with Google App Engine whose
rapporteur was Brett Slatkin

In it, in the section "Building a Blog: Paging ", using the attribute
index, get to paginate infinite elements, using the index as a guide
for the next entry

Now, suppose you delete an entity, the attribute index would not be
consecutive. taking into account that, any idea howto implement
paging that supports not only next and previous, but also, goto "n"
page number.?

thanks in advance,

Abel

[Adam Crossland]

I am wrestling with the very same problem, Abel. I don't have a good
answer just yet.

[bowman...@gmail.com]

I gave up and just went with paging through 1000 results and figure
people shouldn't have to go back further. That's a reality for the
application I'm working on, but not one for others I'm sure.

Besides the "if something is deleted" problem, I also found it
problematic to try and page for things that rely on taxonomy, and
other indexed features like search results and such.

I'm hoping one day they may just figure out a way to allow offsets to
work the way most of us expect them to in queries.

[Adam Crossland]

So far, I haven't had to deal with problem of paging through more than
a couple dozen items, so I'm just grabbing them all and then slicing
the results to get what I need. It's ugly and doesn't scale at all,
but it is all that I have been able to come up with. I'm glad that
I'm not the only one being stymied by this problem.

I know that it is primarily designed to scale well, but other than
that, the datastore is a giant PITA.


On Nov 19, 3:26 pm, "bowman.jos...@gmail.com"

[Alexander Kojevnikov]

I use Brett Slatkin's technique: in the model I page over, I have a
string property called "order". It's a composite string and it's
guarantied to be unique.

When querying for entities, I ORDER BY this property. If I want to
have 20 entities per page, I fetch 21 entities, show 20 of them and
use 21st's "order" in the "Next >>" link: /mypage?next=order

The "next" parameter is used in the query to skip entities: "WHERE
order >= :next"

This allows to page as far as user wants without much burden on the
datastore. The only downside is that it's not possible to page back,
but alas, that's what the browser's "Back" button is for!

Alex
--
www.muspy.com

[Michael Hart]

You could achieve a back button by using a similar scheme (selecting
pageSize+1) and ordering descending.

eg, Say you had an index field with entries "index1", "index2", etc.
Gaps would be fine (but assume none in this example). Say the page
size is 10.

First page you just select the first ten items, with index1 to
index10. The next button would have "?start=index10&order=asc" (you
could assume the default order to be "asc"). On the next page, you
would select pageSize+1 (11) items >= index10 in ascending order,
remove the first (index10) and display the rest (index11 to index20).
The "Prev" link on this page would be "?start=index11&order=desc" and
the "Next" link would be "?start=index20&order=asc". If you clicked
"Prev", it would (based on order="desc") select 11 items <= index11 in
descending order, remove the first (index11), reverse the rest and
display (index1 to index10).

You still wouldn't be able to show links to each numbered page (only
prev/next) - however, if you selected, say (pageSize*5)+1 items (and
adjust the start offset accordingly), then you could show links to the
prev three and next three pages. If you also kept track of the page
number you were on (for the sake of link names), then you could come
pretty close to achieving standard paging functionality (first and
last pages could be done by selecting without a filter, asc or desc).

Cheers,

Michael

[Gopal Patel]

best way i found is to use any entity and +1 rule. and back link using
HTTP REFERER (not accurate but reliable in most conditions..)

[Jon Watte]

> The only downside is that it's not possible to page back

> HTTP REFERER (not accurate but reliable in most conditions..)

Couldn't you build a paging index incrementally, and put it in a
cookie?

[Tim]

I plan to do this by building an index entity as entries are added.
So, for example, the index entity would contain the order string for
every 10th entry (the first on each page), allowing you to jump
straight to a particular page.

[Alexander Kojevnikov]

> You could achieve a back button by using a similar scheme (selecting
> pageSize+1) and ordering descending.

Thanks Michael, I'll give it a try!

[Alexander Kojevnikov]

> best way i found is to use any entity and +1 rule. and back link using
> HTTP REFERER (not accurate but reliable in most conditions..)

The referrer will point to the wrong page once the user clicks the
back link.

[Adam Crossland]

I just started using this class in my blog software. It requires two
queries, one to get the main page of results plus the index of the
next page and one to get the index of the previous page. Since it
requires two queries per request, you definitely want to use memcache
to save the results. It works well with non-consecutive index
numbers, but it does only provide Next and Previous pages; it doesn't
provide for directly indexing page N.

I'd love to hear any thoughts on this approach and feedback, and
please feel free to use to code in your own app.

#Copyright 2008 Adam A. Crossland
#
#Licensed under the Apache License, Version 2.0 (the "License");
#you may not use this file except in compliance with the License.
#You may obtain a copy of the License at
#
#http://www.apache.org/licenses/LICENSE-2.0
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS,
#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
#See the License for the specific language governing permissions and
#limitations under the License.

from google.appengine.ext import db
import copy

class PaginatedList(list):
def __init__(self, *args, **kw):
list.__init__(self, *args, **kw)
self.prev_index = None
self.next_index = None
self.curr_index = None

class Paginator:
"A class that supports pagination of AppEngine Models"
def __init__(self, page_size, index_field):
self.page_size = page_size
self.index_field = index_field

def get_page(self, query=None, start_index=None, ascending=True):
"Takes a normal AppEngine Query and returns paginated
results."

fetched = None

# I need to make a copy of the query, as once I use it to get
the main
# collection of desired records, I will not be able to re-use
it to get
# the next or prev collection.
query_copy = copy.deepcopy(query)

if ascending:
# First, I will grab the requested page of entities and
determine
# the index for the next page
filter_on = self.index_field + " >="
fetched = PaginatedList(query.filter(filter_on,
start_index).order(self.index_field).fetch(self.page_size + 1))
if len(fetched) > 0:
# The first row that we get back is the real index.
fetched.curr_index = fetched[0].index
if len(fetched) > self.page_size:
fetched.next_index = fetched[-1].index
del(fetched[-1])
# Now, I will try to determine the index of the previous
page
filter_on = self.index_field + " <"
previous_page = query_copy.filter(filter_on,
start_index).order("-" + self.index_field).fetch(self.page_size)
if len(previous_page) > 0:
fetched.prev_index = previous_page[-1].index
else:
# Follow the same logical pattern as for ascending, but
reverse
# the polarity of the neutron flow
filter_on = self.index_field + " <="
fetched = PaginatedList(query.filter(filter_on,
start_index).order("-" + self.index_field).fetch(self.page_size + 1))
if len(fetched) > 0:
# The first row that we get back is the real index.
fetched.curr_index = fetched[0].index
if len(fetched) > self.page_size:
fetched.next_index = fetched[-1].index
del(fetched[-1])
# Determine index of previous page
filter_on = self.index_field + " >"
previous_page = query_copy.filter(filter_on,
start_index).order(self.index_field).fetch(self.page_size)
if len(previous_page) > 0:
fetched.prev_index = previous_page[-1].index

return fetched

[Adam Crossland]

If anyone cares, I just made a big post on my blog about this code and
a particular use case. Also, you can see the code in action, as it
was extracted from my blogging software, which runs on AppEngine:

http://blog.adamcrossland.net/showpost?id=aglhZGFtY2Jsb2dyLQsSCUJsb2dJbmRleCIJYWRhbWNibG9nDAsSBFBvc3QiC2FkYW1jYmxvZzE0DA

[thuan]

It's possible to achieve the same result by maintaining a general
index of all the items of the same class. The index would be a list,
which could be massive and takes times to be loaded but if memcached,
would not be that impressive to use. Using a list would enable you: 1)
to sort items by a specific property, 2) to create slices that would
be returned as pages to the user.