Internals of SlickGrid's "virtual canvas"

[Jens Kristian Jensen]

In my search to find examples of implementing a virtual canvas, I
stumpled upon SlickGrid. Awesome plugin, which I might use as well.
But what I really want to do right now is make another jQuery plugin
that implements a virtual canvas, except with a Canvas element rather
than a Div as SlickGrid seem to do. If I get it right, SlickGrid makes
the Div slightly larger than its physical display in the browser, thus
forcing a native scrollbar to appear. Could someone please explain
exactly how this work, particularly:

1) How is the scrollbar fooled into believing that the Div is way, way
larger than its physical display?

2) How does SlickGrid hook onto the scrollbar's events and how is it
determined when to add and remove elements to/from the div, which is
what I assume happens.

Explanations and pointers into the SlickGrid code would be greatly
appreciated. If you have suggestions on how to implement virtual
scrolling and dynamic redrawing of a canvas is also of much interest.

Regards and thanks for SlickGrid, which seems like a swiss army knife.

[Michael Leibman]

On Tue, Nov 1, 2011 at 12:29 PM, Jens Kristian Jensen <je...@migselv.com> wrote:

In my search to find examples of implementing a virtual canvas, I
stumpled upon SlickGrid. Awesome plugin, which I might use as well.
But what I really want to do right now is make another jQuery plugin
that implements a virtual canvas, except with a Canvas element rather
than a Div as SlickGrid seem to do. If I get it right, SlickGrid makes
the Div slightly larger than its physical display in the browser, thus
forcing a native scrollbar to appear. Could someone please explain
exactly how this work, particularly:

1) How is the scrollbar fooled into believing that the Div is way, way
larger than its physical display?

There are two cases here.  I'll start with the simple one, which is when the number of rows is sufficiently low (< ~30'000) that the browser's engine can handle a DIV with a height set to the amount of space required for the entire grid (say 30'000 * 30px).  In this case, we just set the height of the DIV to that and then listen to its onscroll events and dynamically (either synchronously or asynchronously, depending on the timings and scroll distance) add and remove absolutely-positioned rows that happen to be in and around the viewport.  At no point is the scrolling controlled directly, which is one of the reasons it's so fast - the browser still does the scrolling at native speed fully utilizing bitmap caching and whatever hardware acceleration it has.

The scrolling is handled at https://github.com/mleibman/SlickGrid/blob/master/slick.grid.js#L1404 where it keeps track of the direction and distance of the scroll and determines whether to swap rows synchronously or asynchronously.

In the second case, i.e. when the number of rows is more than what the browser can handle, things get tricky.  The canvas is 'virtualized' into overlapping panes.  While scrolling within a single pane, everything works just like described above, except there is an position offset we apply when placing rows.  When scrolling by the scrollbar thumb, we determine which pane the user stopped at, apply that offset and re-render.  When scrolling locally, we detect when the pane jumps occur and do the same.

 

2) How does SlickGrid hook onto the scrollbar's events and how is it
determined when to add and remove elements to/from the div, which is
what I assume happens.

See above.

[Jens Kristian Jensen]

Thanks, Michael. This high level description helps a lot when sifting
through the code.

Assuming that a browser uses memory for all of a Canvas element, even
for the parts not visible, it seems like method two in the above is
what I need.

I've got a slight understanding of the "virtual pages" code now, but I
still cannot find the place in the code where you change the
"proportion" of the native scrollbar in case of multiple "pages", i.e.
the size of the scrollbar's draggable part compared to the one or two
non-draggable parts. The proportions of the scrollbar will need to be
changed depending on the number of pages, right? How do you get a
scrollbar that is not directly tied to the physical vs. visible height
of an element?

[Michael Leibman]

The scrollbar is never adjusted manually.  In the first case, it is controlled by the browser.  The scrolling works just as it would normally, except that the scrollable area doesn't contain all the DOM elements in it - they are added and removed on the fly.

The second case is ALMOST the same.  The scrollbar proportions are actually not accurate in that case, but because the the scrollable area is so large, the "position error" is usually not more than a couple of pixels and is imperceptible.

Take a look at http://jsfiddle.net/SDa2B/4/.  It is an early prototype of the virtual scrolling system with support for unlimited number of rows (second case) I developed for SlickGrid.  Actually, it may be a much better starting point for you since it just implements virtual scrolling without any of the grid stuff mixed in.  One warning though - it has a bunch of bugs that I addressed when I ported the final implementation into SlickGrid but never bothered to fix in the prototype.  Use as a learning material only.

[Michael Leibman]

One more thing - the list in the prototype displays 20 million rows and dividing that up into 100 virtual panes makes the scrollbar a little jumpy.  In SlickGrid I've tuned that to be much smoother.

[Jens Kristian Jensen]

It sounds like I will not be able to use the same clever approach. I
assume this works because the browser does not use memory for the huge
DIV, if no DOM elements are present. In my case, a Canvas large enough
to give the scrollbar any kind of precision, would void the whole idea
of a virtual Canvas, since the browser has to use memory for the
entire Canvas.

I'm thinking if I could use a very tall, but very thin DIV without any
DOM elements for a vertical scrollbar, catch the scroll events for
this and use it to control the Y offset drawing of the Canvas
contents. If you think that is the way to go, I will try going in that
direction. I could also use a separate scrollbar (jquery.ui.slider?)
and use the events from there, but this will not look like a native
slider.

Thanks for your help, Michael.

[Bruce Harris]

Michael, thanks very much for your thorough response. I just starting
playing with SlickGrid (which BTW is awesome) last week and have been
trying to wrap my mind around the code. Your explanation in this
thread has been very helpful.

Bruce

[Jens Kristian Jensen]

For what it's worth, it was something like this I had in mind:

http://jsfiddle.net/jensk/88Hue/

Any suggestions for improvements or simplifications are greatly
appreciated.

[10nidhijain]

Hi,

I want to implement parent child relationship using slick grid means every row of slick grid should have another slick grid in it.

can anyone help on this?