Webworkers in dc (crossfilter)

[Adrián de la Rosa Martín]

I am using dc for an internal project where I handle too much data, so I thought that moving some of the processing to a webworker would help to mantain the UI responsive. I started investigating how to do this and I found that crossfilter is now mantained by the community, with people from dc among others. I would like to help to make crossfilter from web worker possible, but I haven't been able to know about the actual status of the issue or where to start.

What can I do to make crossfilter/dc work from a web worker? What is the status of this topic?

Thanks a lot,
Adrián.

[Ethan Jewett]

There is a community fork of Crossfilter, which is probably what you are talking about :-) It is under the Crossfilter organization at https://github.com/crossfilter I want to be clear that the Square people are still doing some bug-fix maintenance on their repository and the npm module under Crossfilter is the Square one. They do good work, but some of us wanted to see a little more movement. Therefore the fork.

I’ve done some work on getting Crossfilter running in a WebWorker. That work is here: https://github.com/crossfilter/crossfilter-async  The promise-based API there should be working well, but it is not compatible with dc.js. There is partly done work in that repository developing a wrapper that is compatible with dc.js.

It’s worth noting that if your data won’t load in Crossfilter in the main thread, then it won’t load in a WebWorker either. They both have the same memory constraints, to the best of my knowledge. If that’s the case, then you’ll want to look at a server-side solution like nanocubes or the mongoDB-based solution that was just posted to the list. If, on the other hand, you are just trying to improve main thread responsiveness, then the WebWorker approach is the way to go and I’d welcome help or testing on the crossfilter-async work if you’re willing to do that.

Thanks,

Ethan

--
You received this message because you are subscribed to the Google Groups "dc-js user group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dc-js-user-gro...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/5c79102d-3c63-43ce-8e79-a9b606c40c35%40googlegroups.com.

[Wendy Mak]

I am just trying crossfilter-async out, but I can't seem to resolve any of the promises? e.g. if I tried 

data = crossfilterAsync(dataArr);

dimension = data.dimension(function(d) {return d.quantity})

console.log(dimension.top(1))

it returns a Promise object.

but if I tried dimension.top(1).then(function(d){console.log(d)})  it's still a Promise...

And same if I tried Promise.resolve(dimension.top(1)).then(function(d){console.log(d)}) ...

Is it something I am doing wrong? 

Thanks!

[Wendy Mak]

Ah, actually I think it's fine now. I  was using crossfilter v1.3.14 rather than the v2 alpha

[Ethan Jewett]

Good to hear! Please keep in mind crossfilter-async is very immature. If you run into problems, the best option would probably be to create an issue on the Github repository at https://github.com/crossfilter/crossfilter-async I would love to develop it further but haven’t had the time. If you need fixes, support for newer features, etc, just send a pull request and I should be able to get it merged pretty quickly.

Regarding Crossfilter alphas, the 1.4.0-alpha.06 is the latest alpha release. The 2.0.0-alpha.* releases are old.  We decided to go with 1.4 instead of 2.0 a while ago but left the 2.0 alpha releases up there. All they seem to do is cause confusion, so we should probably just delete them, but that could cause some breakage :-(

Thanks!

Ethan

To unsubscribe from this group and stop receiving emails from it, send an email to dc-js-user-group+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/2eb3fb59-3325-49e2-8239-6d9810e57e84%40googlegroups.com.