Best way to incrementally load data with d3

[Dan]

I'm using d3 to visualise a large dataset that's being read from a MySQL DB.

At any point in the visualisation there's a very manageable amount of data needed on screen, but the problem I'm facing is that depending on user navigation, there's a vast amount of data that could be needed from the DB (i.e. it's not feasible to load all the possibility needed data upfront like pretty much all the d3 examples I've seen so far).

Does anyone know if there's any recommendations/examples how to handle a situation like this specifically in d3? For example, I'm thinking about things like...

• is it wise to use node "on click" events to trigger requests to the DB for more information (that feels the natural place to do it, but I'm worried it could cause lags in updating the visualisation).

• or do I perhaps need to stay at least 1-node-click ahead of the game (i.e. load all of the data required by any possible single node-click, and keep doing that asynchronously every time a node is clicked).

• should I anticipate performance problems with updating my definition of "data" (and it's parent-child hierarchy), every time a node is clicked.

Appreciate I can learn this through trial-and-error, but I'm hoping there's some people out there who've learnt the hard-way and are prepared to share their experience! Thanks for any thoughts!

[farrelld...@gmail.com]

I don't know if it's a direct answer to your question, but you might try: http://oboejs.com/ to stream your JSON response. You might take a look at this too. It's referring to DC.js, a library built on top of D3, but the principles should be the same.

Dean

[farrelld...@gmail.com]

Sorry left out the actual link to the optimization post: https://groups.google.com/forum/#!msg/dc-js-user-group/idtDcIBup6o/ifI6QMXaTHcJ

Dean

[nick]

Does anyone know if there's any recommendations/examples how to handle a situation like this specifically in d3? For example, I'm thinking about things like...

• is it wise to use node "on click" events to trigger requests to the DB for more information (that feels the natural place to do it, but I'm worried it could cause lags in updating the visualisation).

If it is (and will stay) quick (less than a few hundred milliseconds) to get that data back, and it feels right, I think on click is fine. You can use a little animation to update the visualization with the information you already have about the new state of navigation, then fully populate it when the new data arrives, and the delay might not be perceptible.

• or do I perhaps need to stay at least 1-node-click ahead of the game (i.e. load all of the data required by any possible single node-click, and keep doing that asynchronously every time a node is clicked).

Sounds like it could be wasteful if you pop open something that has 1000 links for some reason, especially if you can't bundle all of them up into one request.

• should I anticipate performance problems with updating my definition of "data" (and it's parent-child hierarchy), every time a node is clicked.

As long as you aren't ever doing something that uses all of the data, you should be fine.

Appreciate I can learn this through trial-and-error, but I'm hoping there's some people out there who've learnt the hard-way and are prepared to share their experience! Thanks for any thoughts!

Naked d3 with a deeply nested POJO might eventually make this pretty hard to think about, especially if you want to update just some of the data. You may wish to investigate something for the data management: if the nodes naturally have one or two types of structure, something like Backbone.Model and .Collection might be good enough, and the relationships can be handled with something like backbone-relational.

There are other approaches:

• pouchdb which may not be a good match if your data is really tabular.
• baobab which is really intriguing
  

However you do it, the pattern you'll likely want is:

var focusedThing,
  model = Whatever();

model.on("change", render);

function render(){
  d3.selectAll(".container)
    .datum(model.get(focusedThing))
  .selectAll(".thing)
    .data(function(d){return d.get("children"); })
    // do enter/update stuff
  .select("thing").on("click", thingClicked);
}

function thingClicked(d){
  model.fetch(d.id);
}

[Dan]

Thanks Nick and Dean!