Implementing infinite scrolling with large data set

[Dal Chand Choudhary]

Hi, 

I am working on implementing an infinite scroll kind of functionally withing meteor template. After doing some research around how to do pagination and what are the suggested patterns for implementing infinite scrolling, I implemented it using cursor.fetch() and then using Array.slice() function to limit the data being rendered in the template. However I have a couple of doubts: 

1. Is it okay to subscribe to entire data set (I'm not talking about the autopublish) belonging to the that template and do only client side pagination which if I am not wrong, will only help in improving rendering performance. Rendering the entire data (data set is around 2000 documents) is certainly not an option.
2. I am also facing performance issue when calling cursor.fetch() function. It's slow and personally it's not acceptable from user point of view as it takes time in rendering data.

The another approach I could think of is making subscription in parts. As user needs more and more data I can open new handle on the data within that (offset, limit) by subscribing on scroll. But I am worried about multiple observe handles on the server + (offset, limit).

I want to know which method is better and why. 

This is how I have implemented it right now (I'm not happy with the performance though :) )

infinite.html

<template name="infinite">

    <div class="container>

        {{#each photos}}

        <div class="thumbnail" style="background-image: {{url}}"></div>

        {{/each}}

    </div>

</template>

infinite.js

var Photos = new Meteor.Collection("photos");

var offset = 0;

var limit = 20;

var photosPerPage = 20;

var offsetDep = new Deps.Dependency;

var cursor = Photos.find();

var fetching = false;

Template.infinite.created = function() {

    this.dataHandle = Meteor.subscribe("photos");

}

Template.infinite.helpers({

     photos: function() {

         offsetDep.depend();

         cursor.rewind();

         var photos = cursor.fetch();

         var ret = photos.slice(offset, limit);                      

         fetching = false;

         return ret;

     }

});

Template.infinite.events({

    'scroll .container': function(event, template) {

          var el = evt.target;

          if(!fetching) {

               if(el.offsetHeight != el.scrollHeight 

                        && el.scrollTop + el.offsetHeight > el.scrollHeight - 100 ) {

                    limit += limit + photosPerPage;             // increasing number of photos being rendered;

                    fetching = true;

                    offsetDep.changed();

               }

          } 

    }

})

[mk_too]

Your helper appears to be fetching all photos from minimongo even though it only needs some.  Setting up the cursor with limit should be faster.  Something like this

   Template.infinite.helpers({

     photos: function() {

         offsetDep.depend();

         var cursor = Photos.find({}, {limit: limit});

         var fetching = false; //not sure this does what you intend since I am not sure execution waits on the line above.

         return cursor.fetch();

     }

  });

[dalchand choudhary]

Even if I change this I don't see any improvement. The problem is fetch() function itself is taking around 2s. Is that normal?

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/bL3Vr00UgUY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Dal Chand Choudhary
Member of Technical Team
LT Research
Bangalore, India.

[dalchand choudhary]

I also found that even if I make limit = 5, it doesn't make any difference in time taken by fetch() function. So how do I solve this thing any idea?

[Gadi Cohen]

Does this help?

http://www.meteorpedia.com/read/Infinite_Scrolling

You definitely want to limit on the server side, otherwise you'll be sending a loooot of data across the wire, most of which will never be rendered.

Also in your case, you're fetching the entire query (!), and then slicing the array afterwards, which as mk_too pointed out is bad.  Furthermore, if you return the cursor itself, as opposed to cursor.fetch(), Meteor does some further performance optimizations.

[dalchand choudhary]

thanks for the link I will try different approach and see how it works... :)

[dalchand choudhary]

Yes it worked. Thanks

[dalchand choudhary]

ok... I see the same problem when I go all the way down to load all the photos. I think there has to be some better design pattern for this problem.

[Gadi Cohen]

Which method did you use in the end?  The package or the code?

In the helper function, are you returning a cursor, or are you calling fetch() and returning an array?  (The former is much faster).

If you just mean the delay in terms of fetching the data from the server... well, I guess you can improve on the method by prefetching the next batch of data when the user gets close to the end, instead of only requesting it at the end.

[Serkan Durusoy]

Large datasets and infinite scrolling tend to bring the browsers down to their knees.

Is there a way we can clean up previous records while loading new ones and perhaps load them back again as we scroll back up?

The intent is to keep as little data as possible on the browser memory, perhaps just a couple of "viewport-heights-of" data.

[Jan Hendrik Mangold]

On 24 Mar 2014, at 23:06, dalchand choudhary <dc4...@gmail.com> wrote:

On Mon, Mar 24, 2014 at 5:04 PM, Gadi Cohen <dra...@wastelands.net> wrote:

Does this help?

http://www.meteorpedia.com/read/Infinite_Scrolling

I implemented a demo of infinite scrolling using blaze on http://blaze-infinite-scroll.meteor.com

Not sure if I used the patterns correctly, but it works except for some small oddities.

Jan

[dalchand choudhary]

https://github.com/dalchand/meteor-infinite-scroll

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/bL3Vr00UgUY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gadi Cohen]

I'm not sure if browser memory is the real problem?  Slowdown is from processing.  That requires thought about the client-side query, what's involved in rendering the templates, and issues that effect both, e.g. importance of using a cursor and not an array/fetch().

In case I'm wrong, or either way to limit memory use, yeah, you could keep the LIMIT small on the server, but don't use SKIP, it's not performant, instead try get the correct range using query parameters (e.g. $gt).  Anything that's not in that published cursor will be automatically removed from the client/browser.

But I don't know if this is the best pattern, it's nice to be able to scroll back up with no delay and without refetching previously fetched data.  How much data are we talking about?  Definitely explicitly specify the fields to be fetched in the query.

From my side, thanks for posting the repos and demo sites... hope I'll have a chance to take a more in-depth look soon.  Feedback from others definitely welcome too :)

[Serkan Durusoy]

Hi Gadi, consider an app like facebook's feed or timeline and how slow the browser becomes after you scroll down for while.

It is either memory or cpu, but what matters in the end is user experience which I find suffering from infinite scrolling with large datasets (total amount of html/css/js the browser needs to process).

I do agree with your "skip" comment, but then it is really hard to find a good query selector to match against since the most basic idea is getting the count of elements that fit within a desired amount (multiplier) of viewport-heights (so that scrolling back up has some buffer within the browser). This feels like it does call for the use of skip.

What do you think?

PS:

I found these two to be along the lines of my thoughts:

1. http://airbnb.github.io/infinity/
   
2. http://engineering.linkedin.com/linkedin-ipad-5-techniques-smooth-infinite-scrolling-html

[Gadi Cohen]

I guess my home PC is too powerful :>  I've never noticed a slow browser on Facebook (although I do have to clear the RAM once every few days).  But sure, if there's a problem, there's a problem, and I'm sure it's something we can solve in the community, although I think so far the tips given in this thread are good.  This is something I'd definitely love to investigate further, but unfortunately have other priorities right now.  I expect I'll look it again when I'm working on the Meteorpedia rewrite.

As for skip, what kind of data are you using?  You can still use limit, just not skip.  So if it's anything with a timestamp, you can just do a $gt query on the timestamp of the last document on the client side (or a few earlier if you want to keep a few documents around).  Alternatively you could also use something like awwx's mongo-counter on a Collection.deny() method, to give every document a unique incrementing ID on insert.  But sure, if you're querying on user provided text for example, it's a no-go.

Thanks for the links, will take a proper look when I'm working on this again properly.

[Serkan Durusoy]

Thanks Gadi for the pointers and the discussion.

[Gadi Cohen]

Btw, how this this infinite scroll perform? :)  http://demo.famo.us/tweetus/

[Shawn Lim]

I think one way I can think of to solve the problem is to give each document in a collection a number corresponding to perhaps their created date (#1, 2, 3).

Afterwards we can implement the method of showing X documents without using limit and skip simply by using $gte and $lte on these number fields.

To solve the user ability problem of waiting for past fetched documents, maybe it will be a good idea to pre-fetch documents before they arrive in the user's view port so he doesn't know documents are being taken out and replaced. Of cuz then one issue is the scrollbars which have to be tricked into thinking there's still content above when they were removed

[James Wilson]

The slowdown with more media is real and is to be expected. I implemented an infinite scroll using meteor on spark.

A good pattern I've seen was to implement both infinite scrolling down as well as up, as so that only the "selected" part of the content stream is in memory and visible. Unload it as it goes out of view.

James Wilson

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.

[Serkan Durusoy]

Hi James, that's exactly my point, and if possible, I'd like to know how you've achieved it.

[dalchand choudhary]

+1 to Serkan... James could you please share how you have achieved it.

On Tue, Apr 1, 2014 at 10:24 PM, Serkan Durusoy <serkan....@dna-tr.com> wrote:

Hi James, that's exactly my point, and if possible, I'd like to know how you've achieved it.

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/bL3Vr00UgUY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jan Hendrik Mangold]

On 30 Mar 2014, at 10:11, James Wilson <osir...@gmail.com> wrote:

> The slowdown with more media is real and is to be expected. I implemented an infinite scroll using meteor on spark.
>
> A good pattern I've seen was to implement both infinite scrolling down as well as up, as so that only the "selected" part of the content stream is in memory and visible. Unload it as it goes out of view.

I have been playing with this a little bit more and so far have been coming up with this http://blaze-infinite-image.meteor.com

[dalchand choudhary]

I also thought of using skip but it has two drawbacks:

1. when you scroll down and fire new query with skip, the user experience is not good.

2. Publishing data with skip clause uses polling observer.

One way what I could think of two solve memory issue is to make src="" for the images which are out of view. I am working on that. 

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/bL3Vr00UgUY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gadi Cohen]

Additionally, you should explicitly specify the size of images.  What's happening now is that it knows the width of the image (100% of the container), but it has to retrieve the image before it can calculate the height, so things jump around a bit.  I'm not so familiar with the flickr API, but assuming you can get the original width/height of the image at the same time as the URL, and could work out the scale and specify width/height of the image explicitly as attributes to the img tag.

[Igal Tivoni]

This is a long thread which left me a bit confused...  Is there github project that shows infinite scrolling the "right way"?

[Jan Hendrik Mangold]

sorry for the confusion. I was simply experimenting with Meteor’s ability to only send a subset of data to the client and using blaze to render the result without special “magic”. Unfortunately this results in some flickering when the images get loaded.

[Shawn Lim]

Just bumping the thread, I see a new package has been updated that supports infinite scrolling and neighbour data fetching. Looks pretty cool - https://github.com/alethes/meteor-pages

[Andrew Mao]

That does look pretty cool - now if they would add some tests and maybe clean up some of the janky inline CSS :D I'd be afraid for packages like this to open up a ton of dangling subscriptions.

Also, that's probably the biggest monolithic coffeescript class I've ever seen!

[Jan Hendrik Mangold]

On 16 Apr 2014, at 23:27, Shawn Lim <shawn.l...@gmail.com> wrote:

> Just bumping the thread, I see a new package has been updated that supports infinite scrolling and neighbour data fetching. Looks pretty cool - https://github.com/alethes/meteor-pages

as you scroll, this package loads more and more into the local collection and adds DOM elements. If you really had an infinite number of items, this would exhaust the browser, no? I was looking into paging through the results and having Meteor’s reactivity remove DOM/Collection items as you scroll.

Jan