Size limits in filtered lists

[Richard Scannell]

I have a Iist of 80,000 records of people  (these are all generated test data ) stored in a JSON file,  a sample of the names is shown below:

EmpNo   Firstname  Lastname

2949044	Bom	Ackbar
5926741	Sayar	Ackbar
6880068	Knol	Ackbar
9642550	Senni	Ackbar
8919163	Canderous	Ackbar
3969278	Sio	Ackbar
5850995	Ariq	Ackbar
6510943	Rune	Ackbar

When I point my Angular App ( currently set to V1.3.15 - but this seems to be consistent behaviour ersions) at the file with 80,000 records the default sort order behaves correctly according to what I set in my controller : Ascending order of lastname. as per the list above

Here's the relevant HTML ( filtered is my filtered list) 

<tr dir-paginate="item in filtered = ( rows | filter:search:strict|limitTo:500 ) | orderBy:predicate:reverse | itemsPerPage: pageSize |limitTo:500 track by item.Employee_Number" current-page="currentPage">

& in my controller 

  $scope.predicate = 'Last_Name';

  $scope.reverse = 'false';

When I point this at a file containing 90,000 or more records, it all goes wrong. Sort order becomes descending & only a few people with each name are shown:  ( In the test data, I have over 500 records with lastname = Zapalo, but the default behaviour now seems to only show 3)

3368554	Delva	Zapalo
7212131	Karoly	Zapalo
2845659	Pello	Zapalo
4151843	Shayl	Windu
9042355	Tas	Windu

How can I make the larger lists display as intended? 

[Sander Elias]

Hi Richard,

I think there is an issue with the limitTo. You have it in twice. I get why, but I suspect still an issue.

Also, JS is really fast, but resorting 80.000+ records might cause some slow-downs. This is something that's better solved server-side.

Regards

Sander

[Sander Elias]

Hi Richard,

Ir just popped into my mind, if you need datasets this large in your browser, you are better off using IndexDB. in there you can use indexes and so on to handle sorting. (and even to cater for filters)

Regards

Sander

[Richard Scannell]

Hi Sander, 

Thanks for the replies.  I was trying to keep this as a JSON data source if possible ( So I could run JSON exports from a variety of data sources ) & keep it properly flexible.  & its performance with 80k is impressive.

The 90K file behaves nicely when I actually start to filter the data.  Its just the initial load that is problematic-  Is there a way to tell that to it to give itself more time before continuing, I already have a loading spinner on the page ?

[Richard Scannell]

I am now running version AngularJS v1.5.8 & have experimented with the limitTo parm in different places, using my 90K record table : 

The syntax below shows a more complete list, but still in descending order  - I have  all my 'Zapalos' !, 

 <tr dir-paginate="item in filtered = ( rows | filter:search:strict) | orderBy:predicate:reverse | itemsPerPage: pageSize |limitTo:500 track by item.Employee_Number" current-page="currentPage">

The following syntax keeps the original problem 

 <tr dir-paginate="item in filtered = ( rows | filter:search:strict |limitTo:500) | orderBy:predicate:reverse | itemsPerPage: pageSize  track by item.Employee_Number" current-page="currentPage">

[Sander Elias]

Hi Richard,

Are you sure `reverse` is false? Also, why did you put in the round brackets in there? They don't seem to serve any purpose. And the itemsPerPage probably limit's the number of rows too, why a double limiter? 

If I had a list this large, I probably would make a custom filter that combines all of the above functionality in 1. I would write it myself completely to get a maximized throughput, but if you are less confident with JS, Lo-Dash will cover what you need pretty complete.

Regards

Sander

[Richard Scannell]

Hi Sander

Thanks again for the reply.

I am confident reverse is false:  I stepped through in the debugger , AND I added it as a variable to display on the page: it changes when I click to change the sort order. 

 

itemsPerPage is for pagination - so if set to 10, page 1 shows rows 1 - 10, page 2 has rows 11-20 etc....Whereas limitTo lets me restrict the processing in the UI to a maximum of 500 records / 50 pages  out of a total  figure which could have tens of thousands.   

The round brackets are a hangover from some previous code where it had helped _ I have removed them & lists of 120K are processing in the correct order in quite a reasonable amount of time.  Even a list of 300K is rendering OK, albeit slowly :)    ( I wont be using it lists that big - I was just wanting to push the envelope!!! ) 

Thanks for the pointer for LoDash & for taking the time to reply.