Performance

[usha pothiraj]

Hi,

I am using Crossfilter and dc to draw the charts(6) and a dataTable. (barChart(1), rowChart(3) and compositeChart(2) with different dimensions)

I was expecting that the js should be capable of processing huge number of records(in my case 3 Lac).

But I see that the performance is degrading as we increase the records.

Can you please say me in what time how much of data have someone processed using this js combination effectively, so that i could check if I am lagging wrt to using these js?

Thanks,

Usha

[Ethan Jewett]

Hi Usha,

300,000 records is possible (the example page at http://crossfilter.github.io/crossfilter/ has 230,000 records), but you need to structure your data and Crossfilter dimensions/groups efficiently. If you can share a working example of your code with sample data, we can probably help.

Ethan

--
You received this message because you are subscribed to the Google Groups "dc-js user group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dc-js-user-gro...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/ef1908f2-96c3-439f-8817-de3705ea7492%40googlegroups.com.

[Gordon Woodhull]

Hi Usha.

To add to what Ethan said.

Crossfilter usually performs well up to hundreds of thousands of records - I have seen a million work when the records were small and simple.

Reducing the number of columns can help. In particular Date objects seem to be especially inefficient. As well as complex data structures (objects of objects).

--

[Muhammad Ali]

On a Lenovo P50

I tested with 1 Million records 1000000 and it took roughly 1.5 second to filter the records. The data has like 15+ rows.

but Loading those records into the Crossfilter i.e. actually getting the data from source such as a database or file from server into the client interface takes  a shit ton of time. 

[Gordon Woodhull]

I agree, by the time you end up with crossfilter performance problems, you are probably downloading enough data to cause ridiculous load times.

-- 
You received this message because you are subscribed to the Google Groups "dc-js user group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dc-js-user-gro...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/2a667d0e-7a20-4d94-b3e0-e236eac95089o%40googlegroups.com.

[Muhammad Ali]

Would you like to recommend a possible solution to dealing with massive datasets? My records are soon going to cross a million and it is already troublesome.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/87FB7B60-930A-4D32-9EB8-027D9D504486%40woodhull.com.

[Gordon Woodhull]

If you can't reduce the number of columns or avoid the use of Dates, you probably need to either pre-aggregate or go server-side. 

It isn't too hard to use mock crossfilter objects that go to a server. It's a little out of date on the nanocube side, but xfilter is one example of how to do this.

https://github.com/att/xfilter

I've also heard of people having success with elasticsearch for the backend, but I don't have any examples.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/CABgtGJD%2Bxy6f-g-osr5CvztABmQ5QNGZq2THndo_7WQEFCmsig%40mail.gmail.com.

[stutr...@gmail.com]

Hi all, Did anyone have any luck with this implementation?

Currently i'm working on a view that returns c. 5m records (It's nasty but necessary) and of course the Browser just can't handle that kind of payload.

Would be great to have a pluggable SS solution.

Cheers, Stu

[Gordon Woodhull]

Pre-aggregation often works, if you can change a reduceCount to a reduceSum somewhere. (I.e. there is a granularity of aggregation below which individual records are never seen.)

I continue to recommend Nanocubes. There are a few other solutions out there, but I haven't tried them.

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/99cde90c-3981-46ce-805e-4292426a52f7n%40googlegroups.com.

[stutr...@gmail.com]

Good shout Gordon.

I'm very interested in giving Nanocubes a whirl but can't use Docker. Stuck on Windows Server 2012

[Gordon Woodhull]

Good point. It is probably possible to compile Nanocubes under WSL or other Posix emulations for Windows, but they don't support compilation with native Windows tools.

On Dec 9, 2020, at 10:21 AM, stutr...@gmail.com <stutr...@gmail.com> wrote:

Good shout Gordon.

I'm very interested in giving Nanocubes a whirl but can't use Docker. Stuck on Windows Server 2010

On Wednesday, 9 December 2020 at 13:51:34 UTC gor...@woodhull.com wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/dc-js-user-group/5e7899fa-2972-4e16-86ea-48f748f668b9n%40googlegroups.com.

[Gordon Woodhull]

Hi Stu,

Happy New Year!

If you’re still looking for a server side crossfilter solution, Deepak has released a demo package here, written in Ruby and based on ElasticSearch:

https://github.com/kum-deepak/elastic-dc

I am not sure if it is general purpose yet, but given his careful work on the core library, I am sure this is going to be really good.... especially if people test it out and give feedback!

I am bogged down and can’t test / review it yet, so you would literally be first in the door.

If not, I’ll follow up on this thread in a couple of weeks when I have tried it myself.

Cheers,

Gordon

On Dec 9, 2020, at 10:29 AM, Gordon Woodhull <gor...@woodhull.com> wrote:

Good point. It is probably possible to compile Nanocubes under WSL or other Posix emulations for Windows, but they don't support compilation with native Windows tools.

[stutr...@gmail.com]

Hi Gordon,

Happy New Year to you too!

Hopefully our servers will soon get a much needed upgrade and I can start giving Nanocubes a whirl.

elastic-dc does look very interesting. It would be a bit of a journey to get up and running on my dev machine but i'll see if I can spin it up.

Thanks for the early notice and to Deepak for his continued great work.

I'll report back here when I have some feedback.

Chees, Stu