sorted set Redis feed's pagination issue

[Mukesh yadav]

Hi All,

I am implementing feed(post) like twitter. I have created feed(post) and stored in sorted set with timestamp(created_at) as a score.

I am using Rails and Redis as my technology stack.

Now, I am facing two issues.

Problems: 

          1) Pagination :  How do I implement pagination?

                                      Initially I am using zrevrange(key, min, max). 

                                      Now problem is let say my data(pool) is [10,9,8,7,6,5,4,3,2,1]

                                     parameters  from clients are page=0 and page_size=2 then command will be

                                     zrevrange(key, 0, 2*1 -1 ) and result is [10,9]. 

                                     similarly for next request page=1 and page_size=2 then command will be 

                                     zrevrange(key, 2, 2*2 -1 )   and result is [8,7]

                                    but problem is here, let say feed data has been added by others users then 

                                    pool will contain [14,13,12,11,10,9,8,7,6,5,4,3,2,1] .

                                    now next  page=2 and page_size=2 request result is wrong  

                                    zrevrange(key, 4, 2*3 -1 )  and result is [10,9] which is a wrong.

                 Please, suggest me how do I implement pagination.

          2) score value:

                                  which key(timestamp or post's primary key) should I take as score to store data in redis

                                  timestamp = post's created_at.to_i          => many posts may be save in same timestamp value if numbers of post is large. 

                                  post's id = Post's model's primary key   => post may be deleted then I may not be perform calculation based on id.

              Please, suggest me better way to do this.

Thanx in  advance.

Regards,

Mukesh

                                    

          

                      

[David Czarnecki]

On Friday, November 15, 2013 12:56:35 PM UTC-5, Mukesh yadav wrote:

Hi All,

I am implementing feed(post) like twitter. I have created feed(post) and stored in sorted set with timestamp(created_at) as a score.

I am using Rails and Redis as my technology stack.

Now, I am facing two issues.

Problems: 

          1) Pagination :  How do I implement pagination?

                                      Initially I am using zrevrange(key, min, max). 

                                      Now problem is let say my data(pool) is [10,9,8,7,6,5,4,3,2,1]

                                     parameters  from clients are page=0 and page_size=2 then command will be

                                     zrevrange(key, 0, 2*1 -1 ) and result is [10,9]. 

                                     similarly for next request page=1 and page_size=2 then command will be 

                                     zrevrange(key, 2, 2*2 -1 )   and result is [8,7]

                                    but problem is here, let say feed data has been added by others users then 

                                    pool will contain [14,13,12,11,10,9,8,7,6,5,4,3,2,1] .

                                    now next  page=2 and page_size=2 request result is wrong  

                                    zrevrange(key, 4, 2*3 -1 )  and result is [10,9] which is a wrong.

                 Please, suggest me how do I implement pagination.

I'd have to think some more about how to handle the dynamic feed w/ pagination, but this library may initially help with pagination that I wrote, https://github.com/czarneckid/redis_pagination.
 

          2) score value:

                                  which key(timestamp or post's primary key) should I take as score to store data in redis

                                  timestamp = post's created_at.to_i          => many posts may be save in same timestamp value if numbers of post is large. 

                                  post's id = Post's model's primary key   => post may be deleted then I may not be perform calculation based on id.

              Please, suggest me better way to do this.

I've preferred the timestamp since that also allows for the feed to change if you wanted to say bump a post to the top of a timeline based on recent replies/comments. Another library that may help you here that I wrote, https://github.com/agoragames/activity_feed.

[Emil Vikström]

You can use timestamps instead of page numbers. Show page numbers to the user but use timestamps under the hood.

//Emil

[Mukesh yadav]

Hi Emil,

Which command I should use to fetch data because I have two doubts.

  

1)  zrevrangebyscore(key, timestamp_min, timestamp_max) : I can decide timestamp_min(sent by client's devise) but how I can decide timestamp_max. Because, I may get different number of feeds count based upon the popularity of application.

2)   zrevrangebyscore(key, timestamp_min,  -inf limit offset count ) : Let say I have popular app and I got 1000 request/second then there may be possibility that I may save several feed with same rank. Let say there may be 20 feed with time stamp 98777.

Now I made a request to fetch feed like this.

zrevrangebyscore(key,  98778 -inf limit 1 10) then I will get next ten results with last timestamp 98777 and I will make another request with the same parameters then I will get the same result and I will not be able to fetch new results now onwards.

Thank you,

Mukesh

[Mukesh yadav]

Hi David,

Which command I should use to fetch data because I have two doubts.

  

1)  zrevrangebyscore(key, timestamp_min, timestamp_max) : I can decide timestamp_min(sent by client's devise) but how I can decide timestamp_max. Because, I may get different number of feeds count based upon the popularity of application.

2)   zrevrangebyscore(key, timestamp_min,  -inf limit offset count ) : Let say I have popular app and I got 1000 request/second then there may be possibility that I may save several feed with same rank. Let say there may be 20 feed with time stamp 98777.

Now I made a request to fetch feed like this.

zrevrangebyscore(key,  98778 -inf limit 1 10) then I will get next ten results with last timestamp 98777 and I will make another request with the same parameters then I will get the same result and I will not be able to fetch new results now onwards.

Thank you,

Mukesh

[Dvir Volk]

Since social feeds tend to be rapidly changing and items sliding down, the trick to doing that usually involves not doing real pagination, but simply marking the last item id or timestamp that was visible on the previous view. 

so instead of saying "bring me items 0-10, then 10-20, etc"

you say: "bring me 10 items from the start, bring me ten items from item 4387459834759, bring me ten items from 2459814856", etc etc

this can be done with zrevrangebyscore. 

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

--

Dvir Volk

Chief Architect, Everything.me

http://everything.me 

[Mukesh yadav]

Hi Volk,

Which command I should use to fetch data because I have two doubts.

  

1)  zrevrangebyscore(key, timestamp_min, timestamp_max) : I can decide timestamp_min(sent by client's devise) but how I can decide timestamp_max. Because, I may get different number of feeds count based upon the popularity of application.

2)   zrevrangebyscore(key, timestamp_min,  -inf limit offset count ) : Let say I have popular app and I got 1000 request/second then there may be possibility that I may save several feed with same rank. Let say there may be 20 feed with time stamp 98777.

Now I made a request to fetch feed like this.

zrevrangebyscore(key,  98778 -inf limit 1 10) then I will get next ten results with last timestamp 98777 and I will make another request with the same parameters then I will get the same result and I will not be able to fetch new results now onwards.

Thank you,

Mukesh

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/Bo3xcylLr30/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

[Dvir Volk]

If the ranking itself constantly changes, this method is not ideal. If the problem is that new items are constantly pushed at the top, the the suggested method is perfect, use ZREVRANGEBYSCORE <min timestamp> -inf 0 <num>

BTW your problem sort of reminds the problem that reddit are handling with their algorithm. You can read more about their algorithm and the ingenious yet super simple way it works here: http://amix.dk/blog/post/19588

[Mukesh yadav]

Thank you, This will help me to figure out solution.

Regards,

Mukesh