is anyone else trying to get dc to talk to something other then crossfilter?

[Blair Nilsson]

I'm trying to move the data processing server side (since, we have a lot of data, and a tablet won't wear having it all loaded). I don't really want to give up DC.js since, it is awesome.

So I'm trying to either build a library to act like crossfilter, but actually move the "query" to the server where we can process it, or to get dc itself to have a pluggable system for handling the aggregation of the data.

I'd prefer the second, since I think it would help the most people, but, it is taking me a lot longer then I'd like to get my head around what I need to pull out to its own system.

Anyone else tried this?  

[Blair Nilsson]

Ok, I have found where filters are created and added to crossfilter.

Now to see where the data comes out, and how to signal when that has happened. 

If I put in some indirection in those 2 places, we should be able to move the interface for crossfilter out to its own area?

Anyone know of a reason this shouldn't work?

[Adam Reynolds]

Can I ask why you are trying to avoid using it?

The only issue I've had with crossfilter is where the group returns values that I don't want in a graph (which I discussed here in another thread yesterday).

[Blair Nilsson]

(sorry, answer went to his mail box rather then back to the group - I'm giving a more complete answer here now)

Bigger datasets, delivered to tablets. The end devices can't hold the dataset in memory.
We have a lot of processing power / memory we can throw at the problem, just not where the visualization is happening.
An example is all the warrant of fitness reports of every vehicle our county has had, or the results of every court case in the country. We have a lot of data!
We have built a back end that aggregate all the reports in sub 100ms, but we want the data to be directly accessible to the ministers / decision makers.

We have a proof on concept for a builder for dc charts, a drag and drop thing. It doesn't work all that well right now, I'll post the code when it does, but it proves it is possible.

Think http://raw.densitydesign.org/ with a grid drag and drop target. Build your charts, lay them out, use the results.

This has a side effect of us not knowing which charts or data a ministry will be using - so we can't preprocess the data down by hand (well, we could, do a first pass on the server, and cross filter the rest on the client, also an option, but it looks like we still end up with too much data).

So, we are at a crossroad here. Do we try to factor out dc.js's dependency on crossfilter, opening up the way for anyone to plug in whatever aggregation engine they like, be it client or server - or... do we build our own visualization components that can talk to our backend directly?

Well.... 
Nick Zhu said in an earlier post to the group (mid last year) "My original thinking was to introduce a concept called "drivers". Drivers can be used to hook back-end data analysis framework (crossfilter/mongo db) into dc.js library, and they can also be used to filter group and dimensions as what is needed in this case. However never got around to implement them; hopefully in the future I or other contributor will have some extra bandwidth to introduce this flexibility."

Well... I have an itch to scratch, so it looks like I get to be that contributor.
But, while I have used dc.js a whole lot and my javascript skills are not too horrible, it is taking me a while to get to grips with the codebase - and any chance of handling functions being passed back to the crossfilter layer is likely to fail - since we can't emulate that server side.

[Blair Nilsson]

Ok, I think we have our answer :)

I was worried because when we tried to push "the big board of everything" to the tablets, it failed. The tablet can't wear that kind of dataset.

but....

When we are using our "big board of everything" we have a big chunky computer with it and a heap of different charts at once - but we will never actually end up with that many charts at once on a tablet at all - it would be unusable, from a UI point of view as well as a data point of view.

less charts -> less dimensions -> more pre aggregation on the server -> less data at the client.

I'll have to go back and try this out.

[Adam Reynolds]

Sounds good. Remember to destroy the SVG elements when rendered charts are not needed.

DC.renderAll can take a chart group id string so you can be very selective which charts get shown.

The group id is the second param after the element id when creating a chart.

[Gordon Woodhull]

Well, it looks like you found an answer, or found you didn't need an answer.

But I wanted to chime in that the number of crossfilter functions actually called from dc.js is very small - filter on the dimension, all and top on the group, not much else. So this should be possible.

It's actually just the onClick (or the brushing) that triggers a redrawAll for the chart group after setting the filter, if you can believe it. Then each chart pulls in new data through group.all().

Nothing more automatic than that. Kind of surprising, because it feels like the charts are pushing each other.

I don't think there would be a problem with just marshaling group.all() and dimension.filter() to another thread, process, or machine. But... of course I haven't tried it... yet.

[Blair Nilsson]

Yeah, I suspected as much. I found that putting some logging on the filter function, and the functions that create range filters, 2d filters and ordinal filters gave a lot of insight to what was going on under the hood.

DC.js is VERY well written. You can see the quality of the code pretty quickly.
I will end up writing a driver layer for it, since I think there is some goodness to be had there. But the things I need to do now are more around resizing of components, I made changes to do that in 1.6, I'll see if they port nicely to 2.0, (mostly, it was around flushing out the cached values around bar sizes for the bar chart and the like.) I can see there is an invalidate cache call in 2.0, I'll see if all I have to do is call that then a component gets resized on the fly. If not, then hey - you will have some pull requests, since I got it working once, it should be easy to do it again :)

I do have some questions though... With a stacked chart, wouldn't have more sense to have the stacks be generated by another crossfilter group?
Take a thing for tracking vehicle warrant of fitness records... you may want to track how many vehicle where checked as a stack of vehicle types. 

I CAN run a loop and add each vehicle type as a separate filter, and build the stack like that, (in fact, with the backend, we get the list, so it is easy) but it doesn't seem all that natural. Is a "stackBy" function possible? I'll look into it.

Either way, thanks so much for looking after this project :) I use it almost every day (seriously, in the last week, I have built 8 of them, roughly the same the week before - it is rapidly turning into my full time job now - not that I have ANY complaints.)

[Matt Traynham]

Our implementation uses dc.js without Crossfilter.  We still include Crossfilter into source, although I likely will remove it.

The only thing that was absolutely necessary when creating charts that do not hook to Crossfilter:

Set dimension and group to no-ops, provide a data function, implement you're own filter handler

        chart.
            .dimension({ filter: function() {} })
            .group({ all: function() {} })
            .data(function () {
                   // return server data (cache per redraw, since this get's called more than once sometimes.)
            })
            .filterHandler(function (dimension, filters) {
                  // tell server what to do with filter values?
            })

[Gordon Woodhull]

Thanks for the report, Matt.

Are you using .data with all the different charts? I've found that this doesn't always work because dc.js is using .data internally, so you can end up replacing the internal .data hook or getting replaced by it.

[Matt Traynham]

You are correct Gordon.  As I mentioned, those were the necessary changes for every chart type.

We did have to compensate for every where that dc.js set's a different data function (i.e. Stack Mixin, Cap Mixin and Bubble).  The stack mixin patch was a bit more drastic because we wanted it by a stack accessor rather than groups.  Cap mixin we ignored, since we don't use it.  Bubble was almost a no-op, still seems to work. 

[John Griffin]

I'm currently looking at integrating dc.js into our data visualisation tool - http://getdataseed.com.  We already have a back-end so we would want to replace crossfilter with something that talks to our API.

@Blair, @Matt - are your implementations online anywhere?  If not could you produce a patch and attach here perhaps?  I'm interested in the prior art here before diving in and duplicating effort.  All our front-end code is available on Github btw:  https://github.com/dataseed/dataseed-visualisation

Thanks,

Johhn

[Matt Traynham]

I can't post it online, but I did send an email to someone else earlier which had basically a pseudo-implementation and points out all the places I needed to override:

So let’s name a few things that we will probably need to replace.
1. chart.data()
2. chart.dimension()
3. chart.group()
4. chart.filterHandler()
5. chart.redrawGroup()

 

And how to do that:
2 & 3.  Since we no longer need dimensions/groups from crossfilter, our dimension and group only need to be objects that don’t break anything…
    var noop = function() {},
          chart = dc.barChart(‘#chart1’, ‘chartGroup1').
            .dimension({ filter: noop })
            .group({ all: noop });
1.  Our data function is going to be just be a function that returns the data from the server.
    var serverData = {};
         chart.data(function() { return serverData[chart.chartId()] || []; }); // Returns the array of data or empty array
4 & 5.  Here is where the magic happens with the server.  Typically in every chart, any time a filter happens:
It calls chart.filter(<someFilter>), which calls chart.applyFilters(), which calls chart.filterHandler(dimension, filters)
https://github.com/dc-js/dc.js/blob/master/src/coordinate-grid-mixin.js#L770
https://github.com/dc-js/dc.js/blob/master/src/base-mixin.js#L519
https://github.com/dc-js/dc.js/blob/master/src/base-mixin.js#L496
And after that has finished processing, it calls chart.redrawGroup();
https://github.com/dc-js/dc.js/blob/master/src/coordinate-grid-mixin.js#L771
So, what you need to do is override chart.filterHandler.  The ultimate goal of our own filterHandler is figure out what to do with the filters.  What I did:
     // Create a map to maintain all filters of our chartGroup
     var chartGroupFilters = d3.map();
     // Convenience function to return all filters for a chart (these are filters provided by other charts being filtered)
     function getChartFilters (chartId) {
        var tmp = new d3.map(chartGroupFilters);
        tmp.remove(chartId);
        return d3.merge(tmp.values());
    }
    // Override the charts filter handler
    chart.filterHandler(function(dimension, filters) {
        // Here you will provide the way to map the array filters to a server friendly filter array
        // Then you will push that object into the filters d3.map()
        var mappedFilters = …;  // Array of filters (could be only 1 filter)
        chartGroupFilters.set(chart.chartId(), mappedFilters);
    });
    // Update the redrawGroup function of the chart to instead execute new queries for each chart and redraw them in callbacks.
    chart.redrawGroup = function() {
        // Loop any other charts and ajax call to the server for data on that chart.
        var charts = dc.chartRegistry.list(‘chartGroup1’);
        for ( var i = 0; i < charts.length; i++ ) {
                var chartId = charts[i].chartId();
                // The filters for those other charts can be returned by calling
                var filters = getChartFilters(chartId);
                // Perform ajax call to update that chart, with a callback of
                var callback = function(serverData) {
                      serverData[‘chartId’] = serverData;
                      charts[i].redraw();
                }
        }
        // One thing to note, you do not need to ajax call to the server for the chart that added a new filter.  To keep similarity with Crossfilter, all you need to do is call
        chart.redraw();
    };


This is a lot to take in, but the overall flow is:
1. FilterHandler should locally update a maintained map of charts -> filters (filters are something your server understands)
2. RedrawGroup should execute the ajax call for each chart and update the data object referenced in the charts.data function
My code is a bit different so I haven’t really tested any of what I just wrote, but this is the general idea.  If you have any more questions, feel free to ask!

[John Griffin]

Thanks Matt, that's really helpful.  Looking forward to trying this out on the weekend!

[Matt Traynham]

No problem John.  As I said previously, there are some places that just don't work and you'll have to work around them.  I've made quite a few PR's to get the ball going and thankfully Gordon has been kind enough to accept most of them :)

One thing to point out, how you interpret filters in the `filterHandler` function and translate them to your server side is completely up to your implementation, but I would suggest that the server side be able to handle some kind of data aggregation and boolean logic that can be parsed from json or query strings (which any good database usually supports).

[Blair Nilsson]

I've ended up approaching it a very different way.

I've built a fake crossfilter, which pushes the requests to the back end (in this case, ElasticSearch)

the dimension object can have these things called.

    filter

    filterExact

    filterRange

    filterFunction

    filterAll

though, I think DC only calls filter, filterFunction and maybe filterAll.

On the group object, only 2 things are called - All and Top.

Any of the calls to the dimension object, just let the new library know that we will need to fetch new data next time all or top is called.

The calls to All and Top, well.... they return the old data unless we have to fetch new data, in which case - they then iterate though DC chart registry list, (as in the code earlier in the thread) to get the current filters. then calls the back end to get the data, and returns the result.

DC can't tell the difference, since they are the only things called.

Hopefully I will be able to open source the entire thing - it really kicks arse.

[Gordon Woodhull]

Thanks for explaining this, Blair.  I have always thought that the coupling between dc.js and crossfilter is very light, but it's nice to have confirmation that you only have to emulate about half a dozen methods to make dc.js think it's talking to crossfilter.

In the long run, it will be great to define the interface formally, but this should be very helpful for anyone who needs a solution now.

[Blair Nilsson]

One of the things that would have to be done is to stop calling filterFunction for everything in crossfilter. 

You can't just emulate crossfilter and understand what filterFunction is being called with. When I receive a call to filterFunction, I have to grab the dc's chart registry and walk through it to find what filters are in place, because I can't parse a function :). 

getting DC to call filterExact and filterRange would make a huge difference there.

Also 'top' gets called a lot, right after 'all' is called. I suspect it is in the part of DC that is dealing with caps, I think we should only call 'all', and do the rest of the processing ourselves. 

I'm trying to work on a spec for a "generic" crossfilter, and I don't really know if I should include stuff that crossfilter can't do itself in it (but we could polyfill in for it). Stuff like getting trees back as a result (so you can trivially make treemaps / sunburst diagrams)

I have crossfilter / dc working with the force layout engine in d3. I'll put up a demo on gist today.

By the way - I've said it before but I will say it again - thanks Sooooo much for looking after DC. I use it every single day.

[Wladimir Coka]

Hi Blair,

Do you have some working demo of the implementation for a fake dimension and group?

[Blair Nilsson]

I'm working on getting the sunburst chart ready to be taken out of my fork and into DC's main one. I'll put it up online after I've got that done.

Mainly because I have to make it so it can build tree structures on the server side to support sunburst / tree map / chord layout / force layout.

[Blair Nilsson]

ok, I've built a minimum version of "fakecrossfilter" so people can base their own code off it.

I've attached it to this post :)

it is pretty stripped down, but it should give you an idea of how to build one, and how to attach it to dc.js

dc does make other calls to the crossfilter library, so you can't just throw it away :/, but this contains the entire lifecycle, how to get filters out, etc.

--- Blair

[Renato Almeida]

Excellent discussion, thank you Matt, Blair and everyone else; I've managed to move the filtering to the server side through your hints.

Another question for those who are currently working with dc.js + crossfilter.js on the server side: how are you handling multiple session filtering? Create N crossfilters for N users would not scale well for me because loading a crossfilter is expensive.

At first I tried to use a single crossfilter and reset its filters (calling filterAll on the dimensions) every time an user requests filtering on a chart. Since I have around 15 dimensions, this solution works fine only for small~medium size datasets. In my case, it starts to slow down for datasets with 70k or more records.

I was wondering if it would be possible (and how difficult would it be) to create a 'multisession' crossfilter. I would keep a single dataset and save the default indexes to replicate for each new session created. Have you guys tried something like that? Am I going in the wrong direction here, perhaps replacing crossfilter with another library would be more appropriate?

PS: I know this is a crossfilter-related question, but I couldn't find anyone else discussing it except for here.

[Blair Nilsson]

On Monday, October 13, 2014 11:13:35 AM UTC+13, Renato Almeida wrote:

Excellent discussion, thank you Matt, Blair and everyone else; I've managed to move the filtering to the server side through your hints.

Another question for those who are currently working with dc.js + crossfilter.js on the server side: how are you handling multiple session filtering? Create N crossfilters for N users would not scale well for me because loading a crossfilter is expensive.

At first I tried to use a single crossfilter and reset its filters (calling filterAll on the dimensions) every time an user requests filtering on a chart. Since I have around 15 dimensions, this solution works fine only for small~medium size datasets. In my case, it starts to slow down for datasets with 70k or more records.

I was wondering if it would be possible (and how difficult would it be) to create a 'multisession' crossfilter. I would keep a single dataset and save the default indexes to replicate for each new session created. Have you guys tried something like that? Am I going in the wrong direction here, perhaps replacing crossfilter with another library would be more appropriate?

 When we did it (which never hit prod, so ymmv) - we had more then one crossfilter on more then one node, firing up nodes as users logged in, and threw them away when they left.

I feel that this would not scale well :)

[Adam Reynolds]

I hate to say it but it really does sound like you are trying to fit a square peg into a round hole.

I would suggest a two tier approach. One where you know the client is using a PC and can handle the complete dataset and another that abstracts the data to a high enough level to be able to cope with it on a tablet but in effect gives a reduced set of charts.

The only server side solution I could see working is to have a crossfilter held in server memory and dimensions held inside each session.

[Gordon Woodhull]

Renato, you may be able to raise some interest on the d3 mailing list; that's where crossfilter has been discussed in the past. And both Davies and Bostock spend some of their "copious free time" on that list. 

But this list is also good; obviously there are a lot of us interested in these problems.

As long as you are talking about sharing the same groups and dimensions across users, I think you are describing the "multiple filter sets" feature, which is discussed in this issue:

https://github.com/square/crossfilter/issues/109

I don't think it can be done without modifying crossfilter.  It seems like a pretty fundamental change, and probably a lot of work to get it right. But I think it's a natural evolution.

Let me also put in a plug for my AT&T colleagues' project nanocubes, which does solve this problem nicely. It's not the same model as crossfilter and doesn't do incremental range filtering, but it optimizes the same kinds of queries:

https://github.com/laurolins/nanocube

It's written in C++ but it has a json query language.

We plan to integrate it with dc.js, time willing. If anyone wants to help, you are very welcome.

[Diogo Dias]

Hi,

Can you explain me how do you match the chartID with server side? in the Blair's example, we need do send the chartID with filters to server side, but how can we apply these filter in the right dimensions/groups?

Thanks

[Blair Nilsson]

Wow, this is the thread that will not die ;)

OK...... how we build the systems to talk to the server....

we send to the server something that looks like....

[

{ chart_id:'gender_chart', filters:['M'], type:'pie', .... }
{ chart_id:'location_chart', filters:[], type:'row', cap:10, ...}
...

]

The server does the setting the filters to filter everything except the chart the filter is associated with etc, and sends back the same chart definitions... but with a data field.
{ chart_id:'gender_chart', filters:['M'], type:'pie', data:[{key:'M', value:25},{key:'F', value:30},{key:'U', value:5},{key:'X', value:1}].... }

Which the client then calls the setData method on the fake crossfilter group for each chart, by....

for (let result of results) {
  charts[result.chart_id].group().setData(result.data)
}

then call dc.redraw() and you are golden.

Setting caps on the charts will sometimes mess you up....

this is all off the top of my head, and I wrote that code quite some time ago. 

now.... the code for querying the underlaying database? that can become super messy...

[Diogo Dias]

Thank you.

But when you are fetching the filters on client side with dc.chartRegistry.list();

the chart does not contain the "custom" chart id, or can you assign the chartID when initializing the chart?

I'm quite new to this and i appreciate the help.

As for the server, im using a server-side nodejs crossfilter processing, i will make tests after to have db query based filter in order to see if i can get any perfomance boost or not.

[Blair Nilsson]

ah... for chart_id we use the anchor...

chart.anchorName() will give you a useful chart name to use :)

For setData we have a setData method on our fake crossfilter component. 

Mmmm... maybe it would be easier if I just moved my fake crossfilter on to github for people to use. I'll need to give it a bit of a clean (but I have taken a week off work to build a charting library, and do a bunch of dc.js related work (like write tests for sunburst, and a dc.js compatability layer for sunshine.)...)

[Diogo Dias]

Yes that would be great, thanks. I am starting a project in this basis, so i could contribute to it if a make noticeable progress.

[Diogo Dias]

Another question, how do you turned the "sync" call 

all:function() {

      fakeCrossfilter._fetchData();

      return this._currentData;

}

in a async one? So we need to fetch the data from the server, and after it returns we update the chart.

[Blair Nilsson]

After you have fetched back all the data from the server and pushed the data into the fake groups, you call dc.redrawAll();

Which in turn, then triggers the charts to call group.all() to get their data, which you have already pushed in.

I've been spending the last week building a version which I am open sourcing. Not quite sure then it will be ready to push out (it uses the mongodb aggregation pipeline, and a node.js service)

--
You received this message because you are subscribed to a topic in the Google Groups "dc-js user group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dc-js-user-group/fkRoFHuqg4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dc-js-user-gro...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/07666afc-743a-443c-963b-6efba3253749%40googlegroups.com.

[Diogo Dias]

Seems reasonable, but when you filter a chart, DC expects you to return the array of values when it calls the group.all(); so it means that we will have some time frame when the data displayed is the "old" one, and only when we get the data back we call redrawAll which will return the new data-set. Is this right?

It would be good to start looking at your codebase, even if its not completely working/ready. Can you push it to github? 

Thanks

[Blair Nilsson]

I'm working on it tonight, and it should be finished this weekend.

I have an old version (which talks to Elasticsearch), which I can email you if you like, but the version I push to github should - you know.... work better.

Ah... the time for the server to return is actually smaller then the time for crossfilter to return once the data size is above 60k rows (there about)

So yes, dc will sit there for a while when the server is processing, but it is faster then using crossfilter - so for the user, it just seems faster overall.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/87d77723-4d7f-4b83-9cb3-ccc1f32c1c80%40googlegroups.com.

[Diogo Dias]

Hi Blair,

Any news on this? Thanks

[sla...@gmail.com]

I'm interested too! Any news?
Thanks!

[Pattabi M]

For those like me, looking for an update on this post:

The sample file attached by Blair Nisson on 9/9/14 worked well for me. Thanks Blair for the test.html

Earlier I was initializing the crossfilter with a CSV data.

Now, using the code shared by Blair, I got it working with data on the server.

I made the following changes to the code shared

//original code

    function _fetchDataFor(filters) {

      results = {};

      console.log('in theory we go get data for ', filters)

      for (chartId in filters) {

        // but for the example, we will only get random data - replace this method with real 'get our data from something that isn't crossfilter' logic.

        results[chartId] =  [{key:'a', value:Math.random()*5},{key:'b', value:Math.random()*10}];

      }

      return results;

    }

//modified

    function _fetchDataFor(filters) {

        return deferredPromise(filters);

    }

I also replaced the following lines in '_fetchData()'

        // ok we have the data we will send to the server....

        results = fakeCrossfilter._fetchDataFor(filters);

with 

    var aPromise  = s.fakeCrossfilter._fetchDataFor(filters);

    aPromise.then(function(data){ // replacement for d3.csv()

        //initialize charts

        //finally

        dc.renderAll();

[twigb...@gmail.com]

Has any of the foregoing code been put on github anywhere?

We are also interested in hooking up dc.js/crossfilter to filter data through ajax requests to a controller on our server.  

[Ganesh Iyer]

I've been on a version of the demo on the DC.js website that uses crossfilter on the server and uses AJAX to communicate the results. Crossfilter could in theory be replaced by any other filtering mechanism. Anyways here is the halfbaked code https://github.com/lastlegion/dc.js-server-side-crossfilter 

I'm hoping to wrap it up over the long weekend! You guys can chip in as well :)

[Vazgen Ayrapetyan]

I started with your code and it helped me to come up with a simpler solution to remove dc.js's dependency on crossfilter 

and allow the backend to return pre-aggregated data for each chart individually.

It's made possible by the recently added commitHandler api which fires for a chart when the filter changes.
Best of all it takes a callback as a parameter to support async data fetching.

The solution looks like this:  

  

        function commitHandler(redraw, drawCallback) {

            

            var promise = Chart.query($scope.filters).$promise;

            

            promise.then(function(data) {

                var charts = dc.chartRegistry.list();

                var groups = _.object(_.map(charts, function(chart) {

                    return [chart.anchorName(), chart.group()];

                }));

                

                //the magic: trick dc.js into thinking it's dealing with a crossfilter group but actually

                //the backend has full control over each chart's group data so it can now render aggregates directly.

                _.forEach(groups, function(group, chartAnchor) {

                    group.all = function() { return data[chartAnchor]; };

                    group.top = function() { return data[chartAnchor]; };

                });

                

                drawCallback && drawCallback();  

            });

    

            return promise;

        }

        

         

        charts.someChart 

             //just give an empty object for dimension and group, 

             //therefore dependency on crossfilter removed

            .dimension({}) 

            .group({})

            .chartGroup(...)

            .commitHandler(commitHandler)

            .on('filtered', function(chart, filter) {

                //update your filter object

                //this fires before commitHandler

            });

          

         //first load

        commitHandler().then(function() {

              dc.renderAll(); 

        });