How dc.js chart grouping works

[ankit agrawal]

Hi folks, 

I am not able to understand how dc groups chart. So that the change in one filter reflects in all others. 

I have a simple code with two series charts. When I draw brush on one, it does not filter the other. Not sure why ? Can someone please have a quick look at the small code and suggest. 

d3.csv("data/compareData.txt", function(data) {

  ndx = crossfilter(data);
  runDimension = ndx.dimension(function(d) {return [+d3.time.format.iso.parse(d.timestamp), +d.meterid]; });
  frequencyGroup = runDimension.group().reduceSum(function(d) { return +d.frequency; });

  magnitudeGroup = runDimension.group().reduceSum(function(d) { return +d.magnitude; });


 frequencyChart
    .width(768)
    .height(480)
    .chart(function(c) { return dc.lineChart(c).interpolate('basis'); })
    .x(d3.time.scale().domain([1366621166000, 1366621179983]))
    .y(d3.scale.linear().domain([90, 100]))
    .brushOn(true)
    .yAxisLabel("Measured Speed km/s")
    .xAxisLabel("Run")
    .elasticY(true)
    .dimension(runDimension)
    .group(frequencyGroup)
    .mouseZoomable(false)
    .seriesAccessor(function(d) {return +d.key[1];})
    .keyAccessor(function(d) {return +d.key[0];})
    .valueAccessor(function(d) {return +d.value;})
    .legend(dc.legend().x(350).y(350).itemHeight(13).gap(5).horizontal(1).legendWidth(140).itemWidth(70));
  frequencyChart.yAxis().tickFormat(function(d) {return d3.format(',d')(d);});
  frequencyChart.margins().left += 40;

 magnitudeChart
    .width(768)
    .height(480)
    .chart(function(c) { return dc.lineChart(c).interpolate('basis'); })
    .x(d3.time.scale().domain([1366621166000, 1366621179983]))
    .y(d3.scale.linear().domain([90, 100]))
    .brushOn(true)
    .yAxisLabel("Measured Speed km/s")
    .xAxisLabel("Run")
    .elasticY(true)
    .dimension(runDimension)
    .group(magnitudeGroup)
    .mouseZoomable(false)
    .seriesAccessor(function(d) {return  +d.key[1];})
    .keyAccessor(function(d) {return +d.key[0];})
    .valueAccessor(function(d) {return +d.value;})
    .legend(dc.legend().x(350).y(350).itemHeight(13).gap(5).horizontal(1).legendWidth(140).itemWidth(70));
  magnitudeChart.yAxis().tickFormat(function(d) {return d3.format(',d')(d);});
  magnitudeChart.margins().left += 40;
  dc.renderAll();

});

[Matt Traynham]

This is not too clear from the documentation, but there are two places that you'll have to remember one chart filtering the other.

The first is dc.js.  dc.js respects charting groups.  When you create a chart (i.e. dc.barChart("#barChart", "group1"), you can specify the chart group ("group1").  If you do not assign a group, it uses the default group, which you have used in your own code.  

^ This is not your issue.

The second place filtering works across chart is through the use of crossfilter, the underlying library that does multi-dimensional filtering.  It's a very intricate library to say the least, but it has a single workflow.

1. Create an index (ndx = crossfilter(data))

2. Create a dimension (var dim1 = ndx.dimension(accessor)

3. Create a group (var group1 = dim1.group().reduceCount())

In you're charts, you assign the same dimension to both charts.  This will not work, as when you filter a crossfilter dimension, it filters all other dimensions.  So to fix the bug, duplicate the dimension in one of your charts.

  runDimension1 = ndx.dimension(function(d) {return [+d3.time.format.iso.parse(d.timestamp), +d.meterid]; });
  runDimension2 = ndx.dimension(function(d) {return [+d3.time.format.iso.parse(d.timestamp), +d.meterid]; });
  frequencyGroup = runDimension1.group().reduceSum(function(d) { return +d.frequency; });
  magnitudeGroup = runDimension2.group().reduceSum(function(d) { return +d.magnitude; });

[ankit agrawal]

Thanks a ton, Matt. For much needed reply. Tried creating separate dimensions, does not seem to resolve the problem. Still the same. Here is a screen capture. I was just thinking if it has anything to do with series chart. As you can see in the capture, drawing brush does not even fades away the unselected part. 

[Matt Traynham]

I think that is expected, atleast I think it is from previous work done with series charts.  By chance, could you load your code in jsfiddle?  Maybe I could help you debug it.

[ankit agrawal]

Really appreciate another quick reply. I am struggling with creating jsfiddle with external resources. Still working on it. Here it is so far.  Not seems to pick up external data file and other resources. 

http://jsfiddle.net/JPLGF/

[Matt Traynham]

Ok, I'm going to give you an update as I'm not completely finished with it, but I have some progress for you.

The problem with the way you are doing filtering, every dimension you create is an array of two values ([time, meterId]).  This is not going to work for the chart, because your key accessor (which triggers the filter), is just returning the time.  Thus, when it goes to crossfilter, just the time, will not match any dimension key of [time, meterid].

That being said, the work around is to instead make meterId part of the crossfilter grouping functionality:

1. The dimension is just time (as is your x-axis, and keyAccessor)

2. You'll need to implement custom reducer functions for crossfilter that can maintain the sum for each meterId separately.

I got you're jsfiddle working, and I have it working the way you need it to be, but the filtering doesn't work as it should yet.  Effectively this is the correct route, just needs some more debugging love.

http://jsfiddle.net/H3kh5/

Some of the things I've done thus far:

- I put all of your CSV into the HTML ( so I could get it work)

- I created accessor functions for all properties, just so they can be reused

- I created custom reducer functions, that can take 2 accessors, one being the dimension accessor, the other is you're standard sum reducer accessor.  This functionality create's an associative array for each dimension key.  In the end, the data will look like:

  [{ key: time1, value: { meterId1: frequencySum1, meterId2: frequencySum2 }} ....]

var reduceSumInit = function() {

    return function() {

        return {};   

    };

};

var reduceSumAdd = function(dimensionAccessor, reducerAccessor) {

    return function(p, v) {

        var dimValue = dimensionAccessor(v);

        p[dimValue] = (p[dimValue] || 0) + reducerAccessor(v);

        return p;

    };

};

var reduceSumRemove = function(dimensionAccessor, reducerAccessor) {

    return function(p, v) {

        var dimValue = dimensionAccessor(v);

        p[dimValue] = (p[dimValue] || 0) - reducerAccessor(v);

        return p;

    };

};

- dc.js can't handle list accessors, so we provide a custom data function to each chart to blow out the group values.  All this does is take the data from each reduce iteration and blows it out:

i.e. 
  [{ key: time1, value: { meterId1: frequencySum1, meterId2: frequencySum2 }} ....]

becomes:

 [ {key:time1, value:[ meterId1, frequencySum1]},

   {key:time1, value:[ meterId2, frequencySum2]}

]

// Custom data function explodes the data

var dataFunction = function(group) {

    var all = group.all();

    return all.reduce(function(previous, current) {

        for(var k in current.value) {

            previous.push({

                key: current.key,

                value: [+k, +current.value[k]]

            });

        }

        return previous;

    }, []);

};

I know this is complicated, but to get around the array dimension problem, you'll have to push the meterId into the grouping.

Now when you filter, it should just be filtering the time.  Thus crossfilter should be correctly filtering the keys, but it didn't seem to work, so maybe I'm missing something.

[ankit agrawal]

Even before I start reading this, I must say I am overwhelmed. Have no words to thank you enough.  :)

I am currently working with IBM Research labs. But quitting the job tomorrow. :) 

Will go through the mail and get back to you. 

Thanks a TON !! You are THE MAN .... let me know if there is anything I can help you with anytime .. :)

Thanks again!

Ankit

[Matt Traynham]

Heh, my fault if it was overwhelming at all.  I've been using dc.js/crossfilter for ~3 months now, so I've had my fair share of hacking things to get them to work, even if they are awesome libraries.

Hope everything works out in your future endeavors!  I'll probably still post a solution, maybe they can be of use.

[ankit agrawal]

Hey Matt, 

I meant I am overwhelmed (moved emotionally :) ) by the kind of efforts you have put in this.  Did not mean to say that I am overwhelmed with the length of answer. :\

I mean it in all good way.  :)  

[Matt Traynham]

Alrighty, fixed.

http://jsfiddle.net/H3kh5/2/

Added to the top:

/****************************************************************

* DC COMPOSITE CHART OVERRIDES

****************************************************************/

var compositeChart = dc.compositeChart;

dc.compositeChart = function(parent, chartGroup) {

    var _chart = compositeChart(parent, chartGroup);

    

    _chart._brushing = function () {

        var extent = _chart.extendBrush();

        var empty = _chart.brushIsEmpty(extent);

        

        dc.events.trigger(function () {

            if (empty) {

                _chart.filter(null);

            } else {

                _chart.replaceFilter(dc.filters.RangedFilter(extent[0], extent[1]));

            }

            _chart.redrawGroup();

        }, dc.constants.EVENT_DELAY);

    };

    

    return _chart;

};

There is actually a bug for this, which I had attempted, failed and then never came back to. :(   https://github.com/dc-js/dc.js/issues/390

Series (Composite charts) have a weird flow when it comes to filtering, which I don't find correct at all.  The one that is currently in dc.js, hands the filter to each child chart and let's the child filter the dimension.  That is unnecessary, since the composite chart has a reference to the dimension ( so it can filter ), and also provides the implementation of how to filter ( the brushing ).

That being said, I just override the Composite Chart brushing function globally.  After a brushing, the composite chart (abstract parent of child charts), just filters the dimension and notifies to redraw all children.

Hope this helped!

On a side note, something is weird with the elasticY not correctly stretching your charts, maybe setting the .y domain explicitly and remove the elasticY(true).

[ankit agrawal]

That is amazing. I really need to learn a lot form you. 

Did you try that in local as well ?? When I try to get this working in local, it is giving unexpected 

Uncaught Mandatory attribute chart.seriesAccessor is missing on chart[#magnitude-chart]

But should be able to sort that out, I guess.

Thanks a lot, Matt.

[ankit agrawal]

Hi Matt, 

First of all thank you again. Just one quick thing, I just noticed that filter is not actually the way it is in original brush examples. 

If we draw a brush on any chart ..it just shows those part if other charts .. while keeping the x axis domain same. 

Ideally it should have also reset the X-axis in other two chart to match the selection in brush.. only that it will look like zooming in.  Isn't it ?

Ankit

[Matt Traynham]

I think you can set .elasticX(true) on the charts to get the zooming functionality.