Best practices for visualizing a large dataset in the browser (preferably using d3)

[Fabian Zimmermann]

Hello,

I am facing the challenge of visualizing a "relatively large" data set in the browser and wondering if there are best practices available as orientation. The data set looks like this:

- 100k records with around 50 characteristics each

- most of the characteristics have a nominal scale with with only a couple of possible values, a few have an ordinal scale but also only few possible values (up to five)

- a handful of characteristics are text inputs with, on average, 400 characters

- another handful of characteristics have a nominal scale with numerous possible values that are organized in tree structures (e.g. Region => Sub-region => Country => Area OR Industry => Product Group => Product)

The user should be presented with very simple statistics (bar or pie charts showing the count or percentage of items by characteristic). However, he should be able to select parts of the data using filters (e.g. by region, by product group, by product, ...). My first idea was to push the complete data set to the browser and apply filters there (like I do in Excel with pivot tables). I have the feeling, though, that this might cause performance issues and server-side processing might be a better option.

What do you think? And is there a best practice/book/instruction available somewhere that I could use as a starting point?

Thank you upfront for your expertise.

Regards,

Fabian

[Nathan Harris]

with that much data you are not going to be able to filter through that whole data set  a good idea is to pre filter the data and when the user makes their selection call the pre filter data to the client.

  

this can be cumbersome depending on how many choices a user has as choices can grow as they are a permutation, but even splitting the data in half will cut your filter algorithm by time by 50%