D3 and the Single Page Architecture / Application / Interface

[Thug]

Hi all,

I see a number of forum posts converging on, but not quite naming by name, the topic of single page architectures. With so much interest and effort being put into these elsewhere, I would welcome some guidance from within the D3.js community.

Imagine, for example, the case where a given data set is used to generate a main d3 visualisation, this in turn being used to drive one or more dependent but highly configurable orthogonal visualisations. In this case, potentially large configuration parameter sets could be accessed through menus interacting with discrete (SOA) services. Both the central d3 representation, the range of dependent visualisations on offer (or menus governing their appearance) would then occupy different screen regions.

Clearly this goes beyond "mere" synchronisation of changes across multiple objects on a page. Neither are we limited to the loading of all markup in one go (a orthogonal visualisation may conceivably be swapped for another at any time).

D3's event bindings seem a possible starting point for such an approach, but I haven't the depth of experience to say how easy this would be in practice. 

What I have in mind are clear architectural guidelines, based on what those users with experience have found to work well.

Thanks and regards

Thug

[Zack Maril]

Maybe try reading this? 

http://zacharymaril.com/blog/2012/07/10/ideas-about-future-of-d3/

I feel like I recommend this to people all the time it feels like; sorry if I sound like a broken record. In short, I walk through some ideas about how to use backbone and d3 together which might be useful to someone who is thinking about how to structure their web app.

-Zack

[Kai Chang]

I'm also a broken record in this regard. I used Backbone a few times,
such as getting bidirectional linking between parallel coordinates,
SlickGrid, and some pie charts:

http://exposedata.com/parallel/ -- this code is old and not designed to be read

At the time I was using my own parallel coordinates, then the d3
version came out, and it only took 15 minutes to swap the d3 version
in. Backbone made it easy to do that, because the charts weren't
hardcoded together-- they were coordinated with a Backbone Model.

On the other hand, check out this page:

http://bl.ocks.org/d/3150059/

It's similar with bidirectional linking, rendering , a search, group
filtering, brushing, etc. Parallel coordinates is a good visualization
to work with, because it keeps you from assuming the number of
dimensions involved in the app.

The kicker? It's all hardcoded together! It's also much simpler.
Instead of bringing in 19 javascript files, some CSS dependencies
(SlickGrid) and tying together a bunch of libraries, it's just d3.js
and Underscore with a ~600 line javascript file. I think for most
people, especially beginners, just hardcoding stuff till it works is a
good first step, since you can start from any d3 example and work from
there.

My latest effort is factoring *just* parallel coordinates into a
reusable chart that has that use case in mind: bidirectional linking
with other visualizations, possibly through Backbone, Dance.js, Miso
Dataset, or other library. Need to create examples of this, I'm still
on the hunt for a good data table that's simpler than SlickGrid:

http://syntagmatic.github.com/parallel-coordinates/

I use d3.dispatch to emit events for linking:

https://github.com/syntagmatic/parallel-coordinates#parcoords_on

One downside to d3.dispatch is you can only attach one event handler.
With Backbone, you can have several functions listening to the same
event. This is a crucial difference when you have more then 2
visualizations on a page. When I go to build applications with my own
library, I will probably still have a Backbone model listening to
parallel coordinates and broadcasting those events.

Another feature Backbone has is serializing the model loading it with
JSON. When I get some time, I was going to look at adding this to
d3.svg.brush. That way I could save the brushes in parallel
coordinates, and re-initialize them easily. It's an important feature
to turn the visualization from a toy into a tool.

See these discussions on that issue:

https://groups.google.com/forum/?fromgroups#!searchin/d3-js/json$20serialize
https://groups.google.com/forum/?fromgroups#!searchin/d3-js/json$20serialization

[Ian Johnson]

you can use namespacing to get around this, which has recently saved me a lot of headache

e.g.

chart.on("change.internal", function() {})

chart.on("change.foo", function() {})

chart.on("change.bar", function() {})

etc.

--
Ian Johnson

http://enja.org

[Simon Russell]

I'm developing a large single page app, using AngularJS and d3, and a
model layer we're developing in-house. The d3 charts are designed as
AngularJS directives, which makes them pretty easily reusable. It
also means that AngularJS is driving the redrawing of the charts when
the underlying data is changed. (The model underneath is synced with
the backend server using Server-Sent Events, so everybody using the
app sees exactly the same data.) AngularJS has its issues, but it
certainly does some stuff pretty well.

All the synchronisation of data between charts obviously happens via
the model layer. We're probably getting some benefit from not using
particularly complex or massively interactive d3 visualisations
though, although I can't (at this stage) see why that would be a
problem in this design.

In relation to the original query, I think you need to separate your
model from your views. Come up with an internal representation that
suits your data, then think of each d3 element as a way of looking at
that data -- so there's no "primary" view that owns the data; the page
owns the data. Luckily browsers are single-threaded, so even if your
visualisations are changing the data, you're not going to run into too
many issues. Also, make your d3 views completely independent of each
other; if they listen for "data updated" events (and that's probably
as complex as the events should get), bind them to the internal model,
not other views. Also, don't worry about performance initially, just
get it working; the bits that are slow won't be the bits you think
will be slow before building it.

Simon.

[Kai Chang]

I just want to mention that while MVC is a nice paradigm that works
well, it is in some ways directly at odds with d3's data-binding which
places data directly in the DOM.

For instance, here:

http://bost.ocks.org/mike/nations/

You can grab all the current chart's data by entering this into a console:

d3.selectAll(".dot").data()

Now that's cool. Not sure you can structure a complex app this way...
or can you?

[Marc Fawzi]

But that doesn't mean that you can't have a model that is separate from the DOM.

having your own model is key to separating the state from the animation

Sent from my iPhone

[Simon Russell]

I think I'd have to disagree; d3's design doesn't really have anything
to say about MVC. The data you put in the DOM doesn't have to be your
main dataset; it's just whatever you need to get the thing working.

[Marc Fawzi]

Yup agree with Simon

I think Kai was talking at the ideology level when he said "at odds" which they are... But doesn't mean that the approaches can't compliment each other in a real setting

Sent from my iPhone

[Mike G]

Has anyone looked at Knockout to provide a framework for single page applications? http://knockoutjs.com/   It is a small JavaScript library that can be used to create rich single page applications with a clean underlying data model. It is designed to handle dynamically changing sections of a UI based on user actions or data source changes by using views and declarative bindings, and managing the data using view models and observables. It sounds like it would be a nice compliment to D3 for this purpose?

[pax roman]

Looks like a nice library Mike G. However you have to make the difference between whatever works and production level work....

I would stick to d3, jQuery, underscore, Backbone, Node and some other libraries that have a large user's base for production code.

Whatever works - is something good for developing prototypes, research projects, sketches, proof of concepts - you rarely think about maintenance and scalability in this pahse

Production code is something you will have running for months / years / decades sometimes so you have to be sure that the library you are using has a large following

and if your developers move to another place or die you will be able to pick up others fast to resume work ASAP...

I guess that's why I would wait before using something like knockout.js in a production environment. If it's just a fun project - proof of concept or stuff like this I guess it's worth it....

Best regards!

[Peter Rust]

Derick Bailey is a blogger who has done a lot of work with Backbone (and written some Backbone Plugins). He also done significant development work with Knockout and comes from a .Net background, so he's familiar with some of the philosophy behind Knockout. He's written a couple of well-informed, detailed blog posts about Knockout: http://lostechies.com/derickbailey/2011/11/22/backbone-vs-knockout/ and http://lostechies.com/derickbailey/2011/07/29/knockout-style-data-bind-attributes-for-backbone/.

That said, Backbone is the most popular MVC/MVVM framework at the moment -- almost 5x the forks and 4x the stars on github. Of course, that can always change (look at the fate of PrototypeJS).

-- peter

[Simon Russell]

I looked at Knockout, Backbone and Angular before deciding on Angular
for the app I'm building. They all seem reasonable, Angular seemed to
"get it" a bit more; although perhaps it's got a steep(ish) learning
curve.

In terms of getting developers, I'd be more concerned about getting
developers who understand d3 properly than getting developers who
understand the app framework properly...

[Thug]

To everyone who has contributed advice, my thanks: I'm fully in the fact-finding stage and have pressing reasons to get on top of this topic. :-)

While at it, I thought I might issue a challenge beyond my own capabilities, but hopefully of inspiration to others.

The ToDoMVC website allows users to experiment with a variety of MV* frameworks in the context of a simple web application.

While not an ideal showcase for it's capabilities, it would be useful to be able to directly compare d3.js (a "vanilla JS" library) with other libraries, and with MV* frameworks.

My hope is that this might provoke wider discussion on, and practical demonstration of d3.js usage in the contexts of:

• web application framework
  
• a single-page architecture
  

I for one would be fascinated to see where this leads..

Thanks and regards
Thug

[Thug]

Minor clarifier: the "dependent, orthogonal views" I have in mind would essentially be driven by transitions in the main visualisation.

[Thug]

I see a post or two elsewhere in the forum (and earlier this year) mentioning ember.js, and in particular a link to this tantalising blog. Anyone care to add some up-to-the minute opinion?

Given the convention over configuration commonalities between Ember and Ruby on Rails (and having been through much of edx.org's SaaS course material, which uses RoR as an opinionated SaaS teaching vehicle) I'm curious to know more here too.

Thug

[Thug]

Thanks, Zack, lots of interesting ideas here. A little strange that a search for "ember" (mentioned further below) omits your post.

[Thug]

Thanks, Simon, for your suggestions, all of which make sense. There may, however, be a couple of differences in our outlines, in particular synchronisation of data between charts. Where you achieve this via (in?) a (client?/server?) model layer, I envisage a strict separation of core and configuration data (for selection by the user) into discrete services, any data transfer between these taking place only in the browser, the goal being simplicity and modularity:

• one model for data feeding the core visualisation
• for dependent visualisations, "driver" data derived from an in-browser transition operating on the core visualisation, and
• discrete configuration data models (services) for possibly multiple, dependent visualisations.
  

Regards

Thug

[Simon Russell]

Hi,

The data in our design is in a client layer which is synchronised with
a server layer. That layer contains "the data" -- it's not aimed at a
particular visualisation particularly, it's the domain model. The
visualisations look at this and take what they need, and update their
views accordingly. Some of them do a reasonable amount of processing
to convert the data into a form that suits them.

I'm not sure I really fully understand why you have a "core"
visualisation and dependent ones; it sounds like the visualisation and
the data are very bundled up; are you absolutely sure you couldn't
separate this into two layers? As for configuration/visualisation
options, keeping them separate seems like a good idea.

Simon.

[Thug]

Hi Zack,

Small point concerning the section on the eidetic router (which has encouraged me to think more about state and data lifecycles in my architecture), I have a -possibly naive- question.

Presumably because D3 provides much the same array functionality, Mike Bostock seems on the whole to have discouraged jQuery dependencies in d3 code.

A backbone views tutorial claims Backbone's official documentation endorses jQuery, going as far as to suggest that Backbone.View events may not work with other libraries other than jQuery.

It's important for me that any SPA toolset I put together harmonises well with D3. Doesn't this dependency effectively lead to quite some redundany? Could D3 be used in place of jQuery, or is a jQuery dependencies outweighed by the benefits?

Thanks
Thug

[Kai Chang]

Neat project visualizing a Backbone app with d3:

http://latentflip.github.com/violin/