[google-appengine] Pagination

[Kamath Laxminarayana]

HI,
I somehow have the feeling that for pagination, cursors,
memcache and task queues can be used. But I can't get my head around
to get the exact code. Any experts here who can help?

The idea is like this ..
First time the query runs do something like this

q=Something.all()
q.filter
.... and so on ...
total_results=q.count()
q.limit(20) # say,
current_results=q.count() # what if it only returned 15 at the end ?
self.response.out.write(" Showing " + str(current_results) + " of "
+str(total_results))

Add a task with current query as parameter(somehow, may be get some
kind of unique id for it or something) . The task should generate all
(or a set of next N*5) cursors ready in a memcached list(may be comma
seperated row)

The next time the cgi runs it should first check if there is a
memcached list of cursors for the current and next (and may be
previous) N set of pages .. if not , call the task again to generate
it. If it was existing , use the cursors from the memcached list.

the task could do something like this :

q=soemthing.all()
if cursor:
q.with_cursor(cursor)
q.limit(20)
next_cursor=q.cursor()
add the cursor to cache with unique session/query id
rinse, repeat till satisfied.

what say?



if cursor:
q.with_cursor(cursor)

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kamath Laxminarayana]

I know there are a bunch of errors in the code above .. Its just to
give an idea.

[Kamath Laxminarayana]

On Apr 18, 10:33 am, kamathln <kamat...@gmail.com> wrote:


> the task could do something like this :
>
> q=soemthing.all()
> if cursor:
>    q.with_cursor(cursor)
> q.limit(20)
> next_cursor=q.cursor()
> add the cursor to cache with unique session/query id
> rinse, repeat till satisfied.
>

Fix:
-- q.limit(20)
++ temp =q[20]

[Jaroslav Záruba]

On Sun, Apr 18, 2010 at 7:32 AM, kamathln <kama...@gmail.com> wrote:

HI,
   I somehow have the feeling that for pagination,   cursors,
memcache and task queues can be used. But I can't get my head around
to get the exact code. Any experts here who can help?

The next time the cgi runs it should first check if there is a
memcached list of cursors for the current and next (and may be
previous) N set of pages .. if not ,  call the task again to generate
it. If it was existing , use the cursors from the memcached list.

The "set of pages" is a little bit confusing for me. Do you mean "current, next and previous set of records"? (I.e. records to be displayed on current/next/previous page.)

I was thinking about using memcache for paging (of articles) too, I did not think about using cursors though, as I have only very shallow experience with them. I have to look at them, this sounds promising.

[Kamath Laxminarayana]

On Apr 18, 3:57 pm, Jaroslav Záruba <jaroslav.zar...@gmail.com> wrote:

> The "set of pages" is a little bit confusing for me. Do you mean "current,

> next and previous set of *records*"? (I.e. records to be displayed

> on current/next/previous page.)
>
> I was thinking about using memcache for paging (of articles) too, I did not
> think about using cursors though, as I have only very shallow experience
> with them. I have to look at them, this sounds promising.
>

See, with cursors, you can do basic pagination of entries (not the
content of the entries themselves. Like if you have 200 entries in the
datastore for an entity type, you can show 10 at a time and using the
cursor, you can generate a "next" link.

But that in turn poses a problem: you can only generate a "next" link,
and not "previous" link. So my idea for the solution is to go over the
remaining results in a quick way, and cache the cursors that point to
first page, second page and so on.. in a csv in memcache
entry(session managed, of course). Now using this memcache entry, you
can generate a whole array of page links.

The next problem that arises is that if the database has grown too
big, then the csv generation will be slow(as it has to jump a lot of
times). So the next idea is , instead of generating all cursors in the
main cgi , the main cgi generates only first few set of page links,
and then triggers a task queue entry to continue from there. By the
time the user would have read the first page and clicks on "next
page", the task would have generated few more entries. As the set of
cursors for pagination is in memcache, you can use it even in your
first page, if you are using AJAX (guessing, not that sure.)

[Ulrich]

kamathln wrote:
> See, with cursors, you can do basic pagination of entries (not the
> content of the entries themselves. Like if you have 200 entries in the
> datastore for an entity type, you can show 10 at a time and using the
> cursor, you can generate a "next" link.
>
> But that in turn poses a problem: you can only generate a "next" link,
> and not "previous" link. So my idea for the solution is to go over the
> remaining results in a quick way, and cache the cursors that point to
> first page, second page and so on.. in a csv in memcache
> entry(session managed, of course). Now using this memcache entry, you
> can generate a whole array of page links.
>
> The next problem that arises is that if the database has grown too
> big, then the csv generation will be slow(as it has to jump a lot of
> times). So the next idea is , instead of generating all cursors in the
> main cgi , the main cgi generates only first few set of page links,
> and then triggers a task queue entry to continue from there. By the
> time the user would have read the first page and clicks on "next
> page", the task would have generated few more entries. As the set of
> cursors for pagination is in memcache, you can use it even in your
> first page, if you are using AJAX (guessing, not that sure.)
>

Why don't you just generate a few links (something like page 1,2,3, last
page) in the first request and then generate additional links if a user
goes to page 2 or 3 (or last page), so you do not have to generate them
in the background with a task.

-Ulrich

[Kamath Laxminarayana]

On Apr 19, 8:27 pm, Ulrich <mierendo...@googlemail.com> wrote:

> kamathln wrote:

> Why don't you just generate a few links (something like page 1,2,3, last
> page) in the first request and then generate additional links if a user
> goes to page 2 or 3 (or last page), so you do not have to generate them
> in the background with a task.

/me slaps forehead.

Yeah. That should do :-D

May be my mind was thinking in terms of Ajax. I have no exact clue. As
I said in my first post, this is a blurred idea. Or I can say the
gist of the idea. I am not able to get my head around for exact
implementation details, hence I posted here.

[Kamath Laxminarayana]

I realized one thing..

q=Somemodel.all()
q.filter(some cursor compatible filters) # if needed
....
.... # anything here but dont set cursor on the query
q.keys_only=True
temp=q.fetch(1,999)
q.with_cursor(q.cursor)
item=q.fetch(1,999)


Will get you 2000th element.
Using this technique(as long as number of elements per page is a
factor of 1000), you can calculate pagination using normal offsets,
but if it crosses 1000 you split the offsets into 1000s place and the
rest (using integer division and modulo by 1000) . You jump towards
the nth 1000 using the above technique .. then make a final finetuning
fetch .. example will clearly demo what I say:

suppose you want the 226th page with 20 elements per page

######################
page=226
itemsperpage=20

q=SomeModel.all()
q.filters(some cursor compatible filters)
....
.... # anything here but dont set cursor manually on the query
q.keys_only=True

offset=page*itemsperpage # in this case 4520

new_offset=offset % 1000
for i in xrange(0,offset/1000):
q.fetch(1,999) ## we dont assign it because we want to discard
item_keys=q.fetch(itemsperpage,new_offset)

items=db.get(item_keys)
##############

With some smart use of memcache, not so many jumps would be needed
either.
(If someone comes up with a good paginator function or class *with
memcaching*, please post :) )

Pros of this method:
* You can use numbers directly in the urls instead of cursors and
the like :) This makes generating pagination links (First Prev , 1,
2,3 ... 350 Next Last) a breeze while still being able to go beyond
1000 offset limit.
* You can use this method with almost any cursor friendly query
* Cursors come into picture only if you have more than 1000
items.

Cons of this method
* Items per page are limited to factors of 1000 (must be
validated!)
* If users often access items beyond 5000 items, it can still have
an impact on performance????