Presenting massive number of discrete events on a timeline? Zoom in/out?

[Victor Hooi]

Hi,

I need to present lots of discrete events in some kind of timeline UI.

There is a very high density of event (hundreds per second), and they're often grouped very tightly (e.g. one event every few micro-seconds).

However, I also need a way to zoom out and look at larger scales (e.g. over a few minutes, or even over the whole day).

I did see the MIT SMILE widget (http://www.simile-widgets.org/timeline/), as well as the Verite library (http://timeline.verite.co/), however, neither of them seem to allow easy zooming in/out, and they don't seem to cope well as you scale the number of events up (both performance wise, and UI wise, I guess). I'm hoping that D3.js will at least solve the performance side - but I'm interested if anybody knows of a neat way of solving the UI side.

Does anybody know of a D3.js example that does this sort of thing?

Cheers,

Victor

[Rich Morin]

On Feb 6, 2013, at 17:31, Victor Hooi wrote:
> I need to present lots of discrete events in some kind of timeline UI.

D3 is not going to work well for any application that needs to show
zillions of points, lines, etc. The problem is that there is quite
a bit of work involved in creating DOM elements, etc. That's why Kai
Chang (and others) drop into Canvas for that sort of detail work.

That said, D3 might work Just Fine if you can redefine your use case
to only show abstractions of events until you get down to a scale
where you can actually discern and interact with single events. Try
to imagine some sort of "event clusters" that you could show...

-r

--
http://www.cfcl.com/rdm Rich Morin
http://www.cfcl.com/rdm/resume r...@cfcl.com
http://www.cfcl.com/rdm/weblog +1 650-873-7841

Software system design, development, and documentation

[Simon Russell]

Yeah, d3 isn't going to provide a magic solution here. (It would, of course, be excellent for building the visualisation.)

Apart from the complication of plotting a zillion points, you've got the problem of getting the data to the ui in the first place. Also, just plotting the points may not be that useful.

At the more zoomed out levels, you may possibly want a different view... A frequency histogram, for example.  Check out WeatherSpark, if you haven't already come across it.

So yes, you're going to have to put some smarts into your client-side data management, and into your data server. Or find a one-stop solution I guess.

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Victor Hooi]

Hi,

Hmm, so would I use D3 for some of the calculations, then use the Canvas element to actually render it to screen?

Are there any good Hello World type examples for this D3 and Canvas combined approach? From what I read, Cubism.js does this - however, I'm fairly new to a lot of this (D3, Canvas, SVG etc)., so a lot of it is going over my head right now.

Also, all my interactivity, that'll still be done in D3.js, right?

And yes, an abstraction would work fine, actually - maybe something similar to how Google Maps does it with market clustering (https://developers.google.com/maps/articles/toomanymarkers)?

If I went down that route, I'd still do everything with D3.js, right? Or am I better off learning about Canvas and using it anyhow.

Finally, are there good examples I could use as a starting point to learn about:

• Plotting my events (with dragging left/right with the mouse to pan through them)
• Creating clusters of events at different zoom levels.
• Zooming/zooming out, with some kind of event list that syncs to the main sliding timeline

I'm hoping that taking apart a few existing examples will help me learn what I need to do for this.

Cheers,

Victor

[Marielle Lange]

No direct answer, only two pointers. 

- timeline demos implemented using swimlanes - http://bl.ocks.org/2416304
- syntagmatic provided an example of how to load large datasets (over 7000 entries) in one of his parallel coordinates examples - http://bl.ocks.org/3135763

[Simon Russell]

You'd be using d3+canvas, if you were going down the path recommended by other posters on this thread.  d3 basically helps you build code to decide what/where/how to display stuff, canvas is just a way of rendering it to the screen.  So you put data into your d3-using code, and line-drawing commands to the canvas come out the other end.  (Or DOM-manipulations, in the case of SVG.)  So yes, d3 could be managing your interactivity as well.

Personally, I'm not sure you couldn't get by using SVG -- the problem is really about what you want to show on the screen.  Don't forget you probably only have on the order of 1000 horizontal pixels to show stuff -- plotting many points on top of each other doesn't add anything to the graph; it probably just makes it less usable.

But yes, start with d3.js on top of SVG -- switch to using a canvas as the backend if it turns out you need it.  (It might be good to learn how canvas acts vs SVG, but it may not be essential to begin with.)

[Rich Morin]

On Feb 6, 2013, at 18:09, Victor Hooi wrote:
> Hmm, so would I use D3 for some of the calculations, then use the Canvas
> element to actually render it to screen?

Erm, it depends?

Whenever I hear someone asking how to display zillions of items, my take
is that they're asking the wrong question. There's no point in showing
things that the screen can't resolve or putting event handlers on things
that the mouse can't select accurately.

And, in order to display all those things, you're making the server, the
Intertubes, and the user's browser deal with a lot of useless data. So,
do your refinement on the server and ship over only what is needed.

As an example, consider Google Maps. It certainly doesn't send a fully
detailed map of the world to your browser when you first access it...

[Chris Viau]

I totally agree that too many points is not an implementation challenge but a UX challenge. If it were not, I would say just go with WebGL, like D3+X3DOM: http://bl.ocks.org/1291672 I used OpenGL extensively to prototype big graph interaction. But then it all boiled down to selecting the right level of abstraction and the right collaboration between front and back-end. 

[Simon Russell]

The original poster (Victor) did actually link to a couple of timeline projects, so I assume they're at least thinking about UX.  There are obviously many things to consider.

But yes, I guess to summarise what I and the others in the thread have been saying: this is not a trivial exercise.  But d3 is more than ready to use for the part that it actually does.  But it's mostly about the stuff outside d3 -- the right server, the right API, and the right client-side data management.  Google Maps is actually a good example; notice how the rendering detail changes at each level -- it's not just a zoom.

[Victor Hooi]

Heya,

Thanks to everybody for their comments so far.

You're all right, it doesn't make sense to plot every single event at the zoomed out scale. 

(Although there are an awful number of pixels on the giant screens we're looking at projecting these onto...lol.)

The Google Maps clustering I mentioned before is nice, but I'd need to find one that works in this context. Somebody mentioned frequency histograms. Or I was thinking maybe heat maps? I'm not sure, open to any other thoughts...

I still need to learn how to handle mouse panning, and have the macro/micro view move in sync. If I could just get a D3 version of the Verite or SIMILE timeline, that would be a good start, lol, but that's probably asking a lot.

At the backend, I'm looking at using Twitter's Storm to process all of our logfiles in real-time, extract correlated events, and store them in some kind of back end.

I'll probably also need some kind of service layer to serve up the data in a format that D3 likes.

One thing though is, if my graph is streaming in real time (e.g. like Cubism), do I need to delay it somehow?

The reason is that it takes a finite time to process each event in the logfiles, as well as grab all the associated bits.

Can I have the graph scrolling along and backfill any delayed events on the screen as they get processed by the backend? Or am I better forcing the graph to render a bit behind real-time? How can I achieve that?

Cheers,
Victor

[Simon Russell]

You're certainly biting off everything at once :)

As for how to get d3 to do panning, macro/micro view in sync -- should be pretty straightforward, but I'd say just start from a little thing and build it up yourself -- the learning will be quite valuable when it comes time to do the rest of the stuff you're aiming at.  Just create an array of random events, make a visualisation of it, and use the d3 brush stuff to make it zoom/pan.  While you're building that, you can think about how the rest of it will work :)

For realtime graphs, you need to think about how you're actually going to stream them to the browser (websockets, server-sent events, polling) -- this'll have a big effect on whether/how you delay the update batches.  (As for that, I would just batch them if processing is reasonably quick -- you say there can be a few a second; there's not much point in updating the graph super-frequently, although I guess it would look cool.)

Once you've done the bit I was talking about above (making a timeline with mock data), experiment with adding events to it, and see how usable it is.  Then worry about putting real data in it (make your mock data somewhat realistic).

All of the answers to your questions will become apparent, or at least you'll have more specific questions.  Which sounds mysterious and useless when I write it out, but it's true.

[Chris Viau]

Here is the closest I know of a brushable timeline: http://bl.ocks.org/1962173 and others by Bill Scheidel (Bunkat): http://bl.ocks.org/bunkat

[Ian Johnson]

what you want might be found in this thread:

https://groups.google.com/d/topic/d3-js/-Pla93_PZcE/discussion

ziggy made a version thats a slippy timeline, but his block points to d3.v2 so it doesn't load: http://bl.ocks.org/4260533

Ian Johnson - 周彦

http://enja.org

[Chris Viau]

Let's fix it: http://bl.ocks.org/4733216

[Ian Johnson]

easier done than said! haha

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Benjamin West]

I've been wanting something similar.

Ziggie helped me add pan/zoom and I updated my demos to point to d3 v3.

http://bl.ocks.org/4292338 - experiment with the same chart, varying the time ranges many times.

http://bl.ocks.org/4265100 - basic pan/zoom of glucose results, Ziggie simply applied the pan/zoom technique to my static chart.

Eventually, what I'd like is google maps or openlayers style pan/zoom: where I can arrange layers, just like the maps, so that as I zoom in and out of different layers, I get a chance to "reset" what the layers are doing.  This would allow me to kind of intuitively zoom from individual data points through different kinds of trailing averages and other metrics, whatever is appropriate for the range of time able to be shown in the display.

Eg, I'd like to smoothly zoom from the heatmaps and averages shown towards the bottom here: http://jebeck.github.com/blog/2012/10/12/lessons-learned-from-100/ to the individual data points as I've shown above.

-bewest

[Adam Kennedy]

These timeline plots are great. Can anyone point to en example that uses multiple "swimmers" per lane? Dose that make sense?

[Victor Hooi]

Hi,

NB: Not sure what happened here - but my last post seemed to get mangled? Anyhow, reposting here. If my earlier post re-appears, can the mods please delete one? Thanks.

@bewest: Those two examples are pretty awesome, and very close to what we're hoping to use as well =).

One thing I noticed - it doesn't seem to cope well at the extremes of the zoom range.

For example, if you try to zoom in too much, the labels get mashed on top of each other.

Conversely, if you try to zoom out too much, the points all jump to the left axis - I'm not quite sure why:

Also - I do not mean this as a criticism in any way- but the overal feel and animation doesn't have quite the same "polish" as some of the wiki ones:

http://bl.ocks.org/3885705

http://mbostock.github.com/d3/talk/20111116/pack-hierarchy.html

http://mbostock.github.com/d3/talk/20111018/tree.html

http://bl.ocks.org/4063423

http://bl.ocks.org/3943967

It's a bit hard to describe - do you think maybe it's the timing, or the easing or something? Is this something that can be tweaked easily?

I noticed the Timeline.js has a different feel:

http://timeline.verite.co/

However, I think they've overddone it a bit, and it doesn't feel as responsive.

Cheers,

Victor