working with large data set

[Arup Sarkar]

Hi,

How can I create a search capability framework using angular having a large data set of a million records.

Scenario: I have a SQL which returns all accounts in the firm, a million records. There are 8 columns being retrieved. I should be able to do dynamic filter of records on all 8 columns and the data list will show records based on the user criteria.

I have already built something very similar, a java class is returning a list of records to the controller which is doing filter by and ng-repeat properties to do it.

How do I do optimize pagination of such a large data set, secondly how do I bring in chunks of data set, thirdly most important how will the user experience be if someone is searching for something which is in the last few records.

Any thought is highly appreciated.

[Sander Elias]

Hi Arup.

Don't send such a dataset to a client, ever! Not to angular, not to any other client side technology! 

you should do server side paging. And let the server do the filtering/searching. That's whats SQL is used for :-)

Let me do a bit off calculation for you:

  1.000.000 records x 8fields x  low average of 15 bytes a field = 120.000.000 bytes = 114Mb worth of data. 

that's way too much data to transfer down a wire, or to cope with client-side.

Regards

Sander