d3 vs Processing

[Thug]

Hi Mike

Though (once I've got my head around it) convinced of the value of d3 to my particular goals, I'm curious as to the differences between it and Processing. Could you briefly compare their :

- paradigm differences

- technical strengths and weaknesses

.. and highlight application areas to which they are particularly suited

Are there other frameworks you feel are a good fit in the data-driven visualisation space?

Perhaps another d3 aspirant feels tempted to apply the results in a "visualisation of visualisation frameworks" :-)

Thanks

Thug

[sspboyd]

Hi Thug,
I have used Processing and Processingjs quite a bit and from my limited exposure to D3 I certainly recognize a significant difference between the two.

The biggest difference I am noticing so far is that D3 provides you with a range of functionality for transforming data using the capabilities already available to the browser (eg svg, or css, or html or whatever else browsers will be able to support). The functions/methods are native to the browser context. Quoting the D3 site "...there is no new vocabulary of marks to learn. Instead, you build directly on standards such as CSS3, HTML5 and SVG".

Processingjs by contrast seems to be more general purpose. It recreates the methods and functionality (mostly) in the original java version of Processing. This is great since there is a large community and example code to pull from, but you are abstracted from the browser a bit by the translation layer of Processing.

D3 requires a much better understanding of javascript than Processingjs.
D3 looks like it requires you to write less code to create charts, graphs, etc.. than if you were to code in Processingjs (maybe says more about my coding than the libs).

Hope that helps.
Stephen

[Rick Otten]

FWIW, here is my list of bookmarks for basic data-charting javascripty
libraries (other than D3 and Processing) that I've collected over the past
couple of years. Of course there are many specialized libraries for
things like graph-network visualizations, and geo-mapping, and wavelets,
and finite element analysis and that sort of thing. And I'm guessing my
bookmark list is not yet complete.

Protovis: http://vis.stanford.edu/protovis/
DyGraphs: http://dygraphs.com/
Google charts: http://code.google.com/apis/chart/
Infovis: http://thejit.org/
Flot: https://code.google.com/p/flot/
HighCharts: http://www.highcharts.com/
Dejavis: http://dejavis.org/
Graphael: http://g.raphaeljs.com/
JSXGraph: http://jsxgraph.uni-bayreuth.de/wiki/index.php/Main_Page

Of course, lots of people talk about the flash based tool �flare�:
http://flare.prefuse.org/

This is a good reference for real time graphing tools:
http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

D3 is the fastest and most flexible of the browser based solutions of
tried so far.

--
Rick Otten
rot...@windfish.net
O=='=+

[Mike Bostock]

Sure, there are quite a few differences.

1. Pixel vs. vector.

Processing, like HMTL5 Canvas, is a pixel-based graphics library. D3,
assuming you are using SVG, is vector-based. Depending on the type of
application you are building, one may be more appropriate.

For example, certain graphical effects such as motion blur and glow
are easier to do with pixel-based graphics, because you can composite
multiple frames or apply convolutions. It's possible to do this in SVG
using filters, but pixel-based access offers a lot of flexibility. On
the other hand, vector-based graphics offer advantages as well, such
as the ability to scale the graphic up to arbitrary sizes. You can
print SVG to PDF to embed it in other documents, or simply increase
the size of the SVG and the graphic will resize automatically.

2. Immediate vs. retained.

Related to #1, Processing draws directly to a pixel buffer. After
you've drawn things, all you have left is a pixel buffer. In contrast,
a retained-mode system maintains a scenegraph of visible elements.
This offers some significant advantages in terms of ease-of-use: you
can manipulate the scenegraph and the display will automatically
update, rather than needing to maintain dirty regions and redraw
explicitly. This also means you may get better performance, because
the browser can optimize the redraw; although, the scenegraph can
introduce overhead on its own, so it depends.

And, as with animation, it's easier to add interaction to a
scenegraph. With HTML5 Canvas, you have to do hit testing manually
using pointInPath; with SVG you merely need to bind event listeners as
you do with HTML. You can even use CSS :hover, anchor tags, and other
techniques for simple interaction.

3. Proprietary vs. standard.

Processing is a beautiful and simple API. But it's not as prevalent as
HTML. D3 builds on standard HTML, which means you can inspect that
output using browser developer tools. It also integrates with other
technologies—for example, you can style D3 (either HTML or SVG) using
external stylesheets. You can even use CSS3 animations. Likewise, you
can generate some of your HTML on the server, and then use D3 to
transform it on the client. Processing, in contrast, is a closed
system.

4. Imperative vs. declarative.

Processing is, in essence, a C (or C++) -style programming language
using functions and for loops. This is generally called "imperative"
programming because each statement has side-effects. D3 isn't strictly
declarative like CSS, but it eliminates most of the control flow you
typically need. For example, you can bind data to a set of elements
and transform them without ever writing a for loop or mapping data to
elements.

5. All the extras…

Both libraries offer an assortment of add-ons that you may find
useful! For example, I am a big fan of toxiclibs:

http://toxiclibs.org/

Likewise, D3 has various shapes, scales, layouts and charts that can
simplify your visualizations.

Mike

[Thug]

First off, thanks for the comments and explanations above. This is very helpful.

I've no experience of Processing or the other frameworks mentioned, but may be able to contribute some impressions of my own on D3.

Though not a programmer, I've been carrying an application idea around in my head for some 10 years for which the available technologies had never really seemed suited. Following the recent loss of my job, I revisited the idea, in particular in the context of recent developments in the area of in-browser applications. Having played around with HTML5, CSS3, SVG, Canvas, jQuery, jQuery UI, RoR, XMPP and Jingle for some 3 and a half months, I was convinced not only that we were on the threshold to a new in-browser application wave, but that I had at last found the kind of components I had been looking for.

Despite a light euphoria, I remained constantly on the lookout for some means of holding everything together. I couldn't believe my luck when I came across D3. It's ability to orchestrate and synchronise is particularly seductive. In this sense I feel it's potential goes far beyond visualisation. I anticipate it becoming a possibly key component in a new breed of interactive, event-driven, peer-to-peer, browser-based applications, and it's exactly here that my goals and interests lie.

The only real difficulties encountered so far are -as mentioned above- a lack of experience with JavaScript, combined with D3's as yet sparse API documentation. The examples are excellent, but leave me guessing as to behaviours under other conditions. This is borne out in practice, with code spikes frequently producing results that diverge from expectations - or simply fail.

Though my own idea needs only a proof of feasibility to move forward, time is not really on my side. All things considered, though, I've made significant progress - much of which can be attributed to D3. I'm certain the difficulties will decrease as my experience grows, but would be glad if priority could be given to API documentation. This seems to me the only weak link in an otherwise pretty awesome chain. :-)

Thug

[Mike Bostock]

Thanks for the thoughtful note, Thug.

I should have some time this weekend to work on API reference
documentation. I agree it's sorely needed!

Mike

[neotrex]

hello Mike,

by using d3 + jquery you can get some amazing effects in the force directed layouts: nodes, links and levels that appear and disappear, scaling, tooltips, etc.

while d3 seems easier to use than protovis, please do work more on the API documentation during next days. we need it if we are to use it in a financed project.

perhaps you or somebody from your team can add a tutorial on how to use polymaps with D3?

and maybe add more persons like Jan Wilhelm Tulip to the D3 project (even if they work several hours per week only). 

an open source project with 5 people has more chances on the long term than one with 1-2 people.

having more persons would definitely help. also a book would be nice in the future. you could write it with one of the persons you will involve.

I think libraries like D3 will play a major part in the interaction design in the coming mobile applications war. 

Right now D3 is alone in several areas (you know this very well) and it would be nice to see it developed further so that we don't see the Protovis discontinued story repeated.

with sustained effort and a good team you will go beyond processing.js in the next months without doubts.

looking forward to the API documentation.

best regards,

neotrex

[neotrex]

hello Mike,

also please add a space where people will provide links to their d3 projects and improve the gallery for the future.

this way d3 will probably really take off.

best regards,

neotrex

[Mike Bostock]

> and maybe add more persons like Jan Wilhelm Tulip to the D3 project (even if
> they work several hours per week only). an open source project with 5 people
> has more chances on the long term than one with 1-2 people.

I welcome contributions and pull requests! Jason Davies in particular
has been instrumental in contributing a ton of awesome new features,
including some that we haven't yet pushed to the website, such as
histograms, q-q plots, and polylinear scales (one of the most-loved
features from Protovis). My Protovis compatriots Jeff Heer and Vadim
Ogievetsky are also active "behind the scenes", such as the recent
addition of cubic monotone interpolation. And I expect we'll see
contributions from my coworkers at Square in the coming months.

> Right now D3 is alone in several areas (you know this very well) and it
> would be nice to see it developed further so that we don't see the Protovis
> discontinued story repeated.

Protovis was only "discontinued" in the sense that it grew into D3. If
D3 grows into something even more awesome in the future, then the
future will be very bright indeed. ;)

> also please add a space where people will provide links to their d3 projects
> and improve the gallery for the future.

Good idea. We have this already:

http://mbostock.github.com/d3/ex/#user_gallery

And I just enabled the wiki on GitHub, so you can anyone can add links here:

https://github.com/mbostock/d3/wiki

Mike

[Tim Triche, Jr.]

oh nice!  GitHub wikis and Gists are great features, I was just highlighting them the other day to a person to whom I thought they could be of use (John D. Cook).  Even really smart people can stand to have other folks pitch in, especially for drudgery-type work.

It's hard not to be excited about the d3 project.  More people know how to use a web browser than (choose any or all of the following:) R, Excel, IGV, [your favorite program]. Having a well-thought-out and thoroughly collaborative library like d3 reach maturity right around the beginning of a sustained and very public data explosion is a wonderful thing. 

Not that there's anything wrong with the many libraries already out there (including protovis), but rather, there's so much about d3 which feels "right".  Thanks for writing it.

--

If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.

John von Neumann