Parallel Coordinates

[Kai Chang]

I've just finished rough version of Parallel Coordinates in Canvas/SlickGrid, inspired by the Protovis/d3.js versions:

http://exposedata.com/parallel/veggie/

A Backbone model coordinates the filters to link the chart and grid.

The next step is refactoring + column reordering. The code is highly unstable right now, but I welcome suggestions.

It's possible to listen to the filter model with other visualizations. On this page I've named the filter model 'dimensions'. This code will listen to changes in the filtered data:

dimensions.bind('change:filtered', function() {

var data = dimensions.get('filtered');
// your code here

});

I've loaded 100k rows into this and had it render, but interactive performance gets unbearable at ~8k rows. Coloring on hover seems a bit skittish and costs a lot.

[Mike Bostock]

Nice; I like the coordinated table and Backbone integration!

I'd recommend using linear rather than Bézier interpolation. I
experimented with the same (a "diagonal-x" interpolation mode), and
decided the standard linear interpolation is better at showing slope
between adjacent dimensions. This slope is meaningful information for
a parallel coordinates visualization: it encodes correlation.

I gotta ask, though… why use Canvas when you have D3? For example, if
you used D3, you could resize your brushes by grabbing an edge, and
preserve the center by option-dragging.

Mike

[Kai Chang]

I've just been learning canvas recently and this grew out of that. Also I have an Android phone. The new d3 version Jason Davies is working on looks spectacular:

http://bl.ocks.org/1341281

So I might just swap it out for that. SlickGrid has worked out really well though. I'd like to add events on hovering over the grid, so you can see which lines correspond to which rows.

[Mike Bostock]

> I've just been learning canvas recently and this grew out of that. Also I
> have an Android phone. The new d3 version Jason Davies is working on looks
> spectacular:

Thanks! We've been working on that for a while and just released 2.5.0
this weekend with the new brush component. I believe that Android
recently enabled SVG support, but Android manufacturers are pretty bad
about keeping up-to-date:

http://theunderstatement.com/post/11982112928/android-orphans-visualizing-a-sad-history-of-support

> So I might just swap it out for that. SlickGrid has worked out really well
> though. I'd like to add events on hovering over the grid, so you can see
> which lines correspond to which rows.

Yep; two-way linking would be nice.

Mike

[Kai Chang]

Here's the d3 version linked to the grid.

http://exposedata.com/parallel/veggie/vegetables-d3.html

[Kai Chang]

Added an upload button which uses d3.csv to parse the file, then display it in parallel coordinates and the grid:

http://exposedata.com/parallel/veggie/upload.html

The parallel coordinates chart only accepts numeric columns plus one "name" column that can contain strings.

Any ideas on making this more robust? Some quick-and-dirty CSV readers I've seen inspect the first row of values and assume that is the type for the whole column.

[Johan Sundström]

On Wed, Nov 9, 2011 at 18:17, Kai Chang <kai.s...@gmail.com> wrote:

Added an upload button which uses d3.csv to parse the file, then display it in parallel coordinates and the grid:

http://exposedata.com/parallel/veggie/upload.html

The parallel coordinates chart only accepts numeric columns plus one "name" column that can contain strings.

Any ideas on making this more robust? Some quick-and-dirty CSV readers I've seen inspect the first row of values and assume that is the type for the whole column.

Not a perfect fit for your implementation, but maybe useful input nonetheless:

When I hacked up http://bl.ocks.org/1343714 I needed a csv reader that was cross origin and let me rename the columns to toss away a leading "# Columns: " on the header line (see: http://earthquake.usgs.gov/earthquakes/dyfi/events/nc/71676251/us/cdi_zip.txt), and hacked up a load_csv that took separate header and data callbacks to clean up the data to whatever I wanted the result to come out as.

It's got the same call signature as d3.json but accepts a third parameter if you want to do any kind of manual cleanup of the results: https://gist.github.com/1343714#file_y_pipes.js

Yahoo! Pipes does some csv parsing of its own first, so the callbacks get tidied-up strings, surrounding quotation marks being removed already. My example calls it like this:

load_csv(usgs_url, show, { header_cb: parse_header, line_cb: parse_line });


function parse_header(name, n) {
  return ({ '# Columns: ZIP/Location': 'zip'
          , 'CDI':                     'cdi'
          , 'No. of responses':        'responses'
          , 'Epicentral distance':     'dist'
          , 'Latitude':                'lat'
          , 'Longitude':               'lng'
          , 'Suspect?':                'suspect'
          , 'City':                    'city'
          , 'State':                   'state'
          })[name];
}


function parse_line(val, n) {
  return ([ String // zip
          , String // cdi
          , Number // responses
          , Number // dist
          , Number // lat
          , Number // lng
          , Number // suspect
          , String // city
          , String // state
          ])[n](val);
}

... so the "show" function gets to see data like this:

[ { zip: "93109"

  , cdi: "2.0"

  , responses: 1

  , dist: 443

  , lat: 34.4064

  , lng: -119.7276

  , suspect: 0

  , city: "Santa Barbara"

  , state: "CA"

  }

, { ... }

...

]

--
 / Johan Sundström, http://ecmanaut.blogspot.com/

[Kai Chang]

Column names with quotes, spaces, and # ? symbols are valid in d3.csv, so parse_header()-type functionality isn't necessary yet.

Having a list of types, as in your parse_line(), would be valuable.

Here's a file I'm working with, M1+ earthquakes in pass 7 days:

https://gist.github.com/1356889

The challenging columns are:

Src, Eqid, Version, Region:
Possibly entities displayed equidistant in the column, alphabetical order. I'll experiment with the default rendering of strings.

Datetime:
Ideally, convert to Unix timestamp for positioning on column. Requires knowing how to parse the date. Automagically converting Datetime may be too complex. 

2011/11/10 Johan Sundström <oya...@gmail.com>

[Mike Bostock]

> http://exposedata.com/parallel/veggie/vegetables-d3.html

This is lovely! Thank you for sharing with the group.

You've inherited a tiny bug of mine, which is that you can't select
the topmost pixel. If you switch to inclusive bounds (<=) then you'll
be able to select a data point that interests the axis upper bound.

Mike

[Kai Chang]

Added all the foodgroups from the USDA nutrition database:

http://exposedata.com/parallel

There's a new "Remove Selected" button that will remove data you've selected.

Try removing all the high-sodium soups (greater then 3000 mg) from the sodium column. The axis rescales (must... tween), giving a closer view of how the rest of the data is distributed.

[Alex Simoes]

This is really great! Will certainly add it as a bookmark. 

One thing that would be nice, would be the ability to add/remove whole foodgroups by toggling them (maybe as buttons) since you have this data now. The reason I say that is because when searching I keep getting "Spices and Herbs" as the top group which is not really something I care about when trying to find a food high in fiber for instance.

- alex

[Johan Sundström]

One thing that would be nice, would be the ability to add/remove whole foodgroups by toggling them (maybe as buttons) since you have this data now. The reason I say that is because when searching I keep getting "Spices and Herbs" as the top group which is not really something I care about when trying to find a food high in fiber for instance.

Another powerful improvement is to make the visualization bi-directional – in other words, not just having the parallel coordinates on filtering the table below by property range, but to add mouseover/mouseout handlers to that table which highlights the hovered food's line in the graph.

An example of the same idea, letting you explore a data set from many different aspects with very little effort:

http://johan.github.com/d3/ex/recettear.html

That one is a view of items in a computer game, that you can sift by whom can use what (top), or where to come by what (bottom), and as you hover each item in the page, the corresponding game characters and locations light up, so you can browse by whichever aspect interests you.

The idea is better than the code, as that was my first hack to get to know d3, but in case someone wants to play with the data set (I've had a lot of fun with it myself, as it is inter-linked in so many ways :-), a more accessible version is here:

https://gist.github.com/1306472

and an example use of it (hot-linking the data from there rather than copying it) here:

http://bl.ocks.org/1338778

But, to get back on topic: it's delightful to see your new iterations on this visualization; thanks for posting each new version on the list! :-)

-- 

[Kai Chang]

Alex,

Category toggles would be very nice. I need to refactor my code a bit to use filters that aren't numeric ranges.

For now I've removed the Spices & Herbs and Baby Foods column, since they were the most in my way trying to find food for adults.

[Kai Chang]

Johan,

Bi-directional linking is definitely on my mind. I'm using SlickGrid for the table below, and haven't figure out it's event system entirely. It's slightly complex for my taste, especially with d3.js and backbone.js, but it performs so well and has so much functionality for tabular data that I'd rather just learn its quirks.

I'll take another crack at this feature tonight, especially since it will aid comprehension for new users.

2011/11/16 Johan Sundström <oya...@gmail.com>

[Kai Chang]

New updates at: http://exposedata.com/parallel/

You may need to refresh the page due to caching issues with github pages.

There are some new toggles in the top right.

Hide Ticks makes the ticks/labels go away. With 10 columns, these ticks create a lot of visual noise. Seeing "shapes" and patterns in the data is much more relaxing without them, once you're acquainted with the scales.

Dark inverts the color scheme. Easier to distinguish colors with a dark background.

Full Width stretches the graph horizontally to fill the screen. Try Chrome's full screen mode on a wide-screen monitor.

Hide Background hides the unselected "background" lines in the chart. Turn this on if you drop the opacity on the slider next to it. Lower opacity is helpful to get a rough idea of how data is distributed, since there is a lot of occlusion with 1500 lines.

[Kai Chang]

Added bi-directional linking at http://exposedata.com/parallel/

Hover over the grid to see the corresponding line in parallel coordinates.

[Johan Sundström]

On Tue, Nov 22, 2011 at 04:22, Kai Chang <kai.s...@gmail.com> wrote:

Added bi-directional linking at http://exposedata.com/parallel/

Hover over the grid to see the corresponding line in parallel coordinates.

Beautiful! :-D

Thanks; splendidly educational view of the dataset.

[Kai Chang]

Added two new linked views.

http://exposedata.com/parallel/

The first is a pie chart displaying the ratio of selected foods in each group. Prior to this widget, I was severely underestimating the number of Vegatable and Vegetable Products, since they are mostly occluded and a quiet lavender color. Next update a click on a wedge will filter by category, as Alex described. I was a bit hesitant to do a list of buttons, but a pie chart is the perfect space-constrained solution.

The second is a ratio of selected vs filtered out data points. Pie chart may not be the best representation. I'm considering attaching the Keep/Remove functionality to clicks on these wedges.

2011/11/22 Johan Sundström <oya...@gmail.com>

[Kai Chang]

These are notes on user difficulties from watching newcomers use the chart. The toughest interface problems were:

* Noncomprehension of the visualization

* Accidentally destroying filters

* Accidentally grabbing ticks/title, but never completing a reordering action

* Small target areas for filter creation/resizing

* Not realizing filters were draggable

* Letting go of filter drag too early. Some attempt to place with two clicks. Others try to "throw" filters like lassos, perhaps used to inertia in touch devices. The effect is placing is a very small filter

* Placing filters that removed everything. The filter outlines get lost in the ticks, and users had no way of understanding why the data had disappeared

* Clicking on rows in the table, with no effect

* Click pie charts, with no effect

* Not scrolling down far enough initially to see the data table, even while interacting with the chart

I'm thinking icons for all the actions one can take on a column. Delete and a dropdown for more advanced actions like colorings, scales, reordering and flipping.

A neat feature would be programmatic brushes which explore the parameter space without user input, similar to GGobi's Grand Tour.

Currently working on data schema and csv uploader. If anyone has interesting data with many numeric fields, I would appreciate some real data to build test cases.

[Johan Sundström]

Not new data sets (but you can find tons of it easily on freebase.net), but some more input on your user test results:

 

These are notes on user difficulties from watching newcomers use the chart. The toughest interface problems were:

* Click pie charts, with no effect

I assume you noted that this is because you didn't implement the two intuitive filtering functions for the those charts?

The left pie chart would be a "filter by group" control (in addition to the range filters by the draggable constraints), which limits the other views to a the clicked colour. (This too suggests the pie chart is not the ideal intuitive control for that purpose, as you might want to filter to show or hide more than one group.)

The right pie chart would presumably invert the view to show everything currently hidden, and hide everything currently shown, and, in the process, invert all the constraints – requiring the drag controls to handle discontinuous ranges like a union of ]800] and [1200[, if the user had previously selected [800, 1200].

* Not scrolling down far enough initially to see the data table, even while interacting with the chart

And this is an effect of putting the large block of howto text front and center, instead of the data. I think you would get more understanding by hiding all of but the first sentence behind the "show more!" first, assume that the graph and controls speak for themselves, and when not, that users then click to see more, once their curiosity sets in.

I think that text block is too long, and that most people skip right to the graph, and only note as much of it as fits on the page.

[Kai Chang]

Johan, thanks for the design suggestions. I made a several changes as a result of your advice:

• A legend, eventually to be used for filter by group instead of the pie chart
• Reduced the opacity of pie chart to make it less prominent, and removed grey backgrounds
• Removed instructions for now. There should be a separate tutorial page and video, maybe by January once things are more polished for new users.

Other changes:

Curved polylines using cardinal interpolation. The tension is subtle at 0.85, but it helps a lot with the continuity of lines. There is some motivation for curves in this paper, especially for lines with shared value pairs on adjacent columns:

http://www.soc.napier.ac.uk/~marting/parCoord/GrahamKennedyParallelCurvesIV03.pdf

The page is full-width by default now. Perhaps will add auto-adjusting height to fill the window.

Added auto-scaling opacity based on number of foods selected (more foods, more transparency).

Lastly, I've included much more of the USDA data (~80% compared to ~25% before), including foods with null values for some nutrients. Nulls are shown as 0 on the parallel chart. This totals up to almost 7000 rows with 14 numeric columns!

The page slows to a crawl even in Chrome. Worse in FF. Seems to be the weight of the SVG elements on the DOM. Even CSS changes like hiding ticks becomes much slower. The number of tick marks doesn't change with more data, but maybe there's some unnecessary rendering going on... I'll take a closer look at these issues and report back.

To get better performance, use Keep/Remove to subset the data. Filter down to 1200 data points and click Keep- things will speed back up.

2011/12/4 Johan Sundström <oya...@gmail.com>

[Nenad]

Kai, this looks great. Can't wait for the tutorials being that I'm a
newbie with D3 and js.
You had asked for some data and use cases, I have a tom in SAS format
but I have a feeling that the response would be very slow as the
amount of data increases to over 10K?

Impressive work, much appreciated data exposure.

On Dec 8, 3:08 am, Kai Chang <kai.s.ch...@gmail.com> wrote:
> Johan, thanks for the design suggestions. I made a several changes as a
> result of your advice:
>

>    - A legend, eventually to be used for filter by group instead of the pie
>    chart
>    - Reduced the opacity of pie chart to make it less prominent, and
>    removed grey backgrounds
>    - Removed instructions for now. There should be a separate tutorial page

>    and video, maybe by January once things are more polished for new users.
>
> Other changes:
>
> Curved polylines using cardinal interpolation. The tension is subtle at
> 0.85, but it helps a lot with the continuity of lines. There is some
> motivation for curves in this paper, especially for lines with shared value
> pairs on adjacent columns:
>

> http://www.soc.napier.ac.uk/~marting/parCoord/GrahamKennedyParallelCu...

>
> The page is full-width by default now. Perhaps will add auto-adjusting
> height to fill the window.
>
> Added auto-scaling opacity based on number of foods selected (more foods,
> more transparency).
>
> Lastly, I've included much more of the USDA data (~80% compared to ~25%
> before), including foods with null values for some nutrients. Nulls are
> shown as 0 on the parallel chart. This totals up to almost 7000 rows with
> 14 numeric columns!
>
> The page slows to a crawl even in Chrome. Worse in FF. Seems to be the
> weight of the SVG elements on the DOM. Even CSS changes like hiding ticks
> becomes much slower. The number of tick marks doesn't change with more
> data, but maybe there's some unnecessary rendering going on... I'll take a
> closer look at these issues and report back.
>
> To get better performance, use Keep/Remove to subset the data. Filter down
> to 1200 data points and click Keep- things will speed back up.
>

> 2011/12/4 Johan Sundström <oyas...@gmail.com>

[Johan Sundström]

On Thu, Dec 8, 2011 at 00:08, Kai Chang <kai.s...@gmail.com> wrote:

Lastly, I've included much more of the USDA data (~80% compared to ~25% before), including foods with null values for some nutrients. Nulls are shown as 0 on the parallel chart. This totals up to almost 7000 rows with 14 numeric columns!

The page slows to a crawl even in Chrome. Worse in FF.

I'd suggest looking at it in Google Speed Tracer to see if you can find hot spots and/or strength reduction opportunities, and if not, going back to a smaller set, maybe with the option of pulling in separate csv:s into the visualization by ajax.

(E g: grouping foods into separate csvs by category, and listing categories that can be loaded / unloaded, much like your keep / remove.)

Unusable by default removes pretty much all the sexiness of the visualization.

[Florian Gysin]

Hey Guys

I am just starting to learn d3.js and am trying to implement parallel coordinates for a project of mine. I wondered if I might ask a question...

The problem is that my data contains many ordinal variables. So I read online about the d3.scale.ordinal - is this already implemented in parcoord?

If yes how would I go about using it? (Any examples of that online?)

If no - is it still possible to somehow work with ordinal values in the current parcoord version? Parallel coordinates would be the perfect representation of my data if I could get this to work...

Any help would be greatly appreciated... I am totally new to d3.js and also fairly new to JavaScript in general, so I am kind of lacking the confidence to go about implementing new features in the parcoord plugin...

Thanks,

Florian

[Kai Chang]

Florian,

I think there are 3-4 other people who are also new to JavaScript
trying to add ordinal axes to parallel coordinates this month. I was
going to implement it myself, but it's a great opportunity for you
guys to discuss your attempts and figure out a solution.

I think YoDude is probably farthest along. If you want to participate,
reply to this thread:

https://groups.google.com/forum/#!topic/d3-js/SurT5gXHQEM

He's working from the example here:

http://bl.ocks.org/d/3150059/

There are a lot of threads on parallel coordinates. I'll reiterate
here that the version in active development now is a reusable chart. I
recommend using it for most applications, but to hack in ordinal axes
feel free to work from any past example. Once you all find a solution,
I'll add it to the library.

http://syntagmatic.github.com/parallel-coordinates/