Filters - speed and size of result set

[Tom Leadbetter]

Hi all

We have a large data set, potentially several thousand results, though just currently a maximum of about 500. We have paging, ordering and filtering (powered by checkboxes). We have noticed that there is an ever so slight delay between selecting the checkbox and/or checkbox label and the dataset being updated. It is a minor delay, very minor and noticed it after looking at some large sites that use clientside filtering, such as SkyScanner:

http://www.expedia.co.uk/Flights-Search?trip=roundtrip&leg1=from%3AManchester%2C+England%2C+United+Kingdom+%28MAN%29%2Cto%3ANew+York%2C+NY%2C+United+States+%28NYC-All+Airports%29%2Cdeparture%3A14%2F03%2F2014TANYT&leg2=from%3ANew+York%2C+NY%2C+United+States+%28NYC-All+Airports%29%2Cto%3AManchester%2C+England%2C+United+Kingdom+%28MAN%29%2Cdeparture%3A29%2F03%2F2014TANYT&passengers=children%3A0%2Cadults%3A2%2Cseniors%3A0%2Cinfantinlap%3AY&options=cabinclass%3Aeconomy%2Cnopenalty%3AN%2Csortby%3Aprice&mode=search#departingModuleIndex=0&leg=1

Their checkboxes are rapid. Ours have no more than a second delay.

Unfortunately I can't share our site (though anyone wishing to take a closer look, please get in touch) but I have mocked up a demo

http://plnkr.co/edit/HWDtfmUzWcrqux762Fmu?p=preview

Typically this works very quickly (even locally with 2000 results) so it has me thinking that other JS in our app is causing the slow down. So that's something to look at.

However the main reason I come here is to get a sanity check on our filtering approach. Is it correct? Is it scalable? Any issues you see? In the plunker demo, in script.js, there is a commented-out console.log on line 33. If you enable that it will run the console 600 times, exactly three times the amount of results in the dataset. 

SkyScanner uses Backbone and therefore I wonder whether it then uses Underscore. Is Underscore going to be a benefit to our project?

Generally, any thoughts, tips, solutions you have, please say :)

Cheers

Tom

[Tom Leadbetter]

Sorry, the plunker has reset itself. 

I'll sort it again...

[Tom Leadbetter]

ok, it's back now

http://plnkr.co/edit/HWDtfmUzWcrqux762Fmu?p=preview

On Tuesday, 21 January 2014 17:28:40 UTC, Tom Leadbetter wrote:

[Sander Elias]

Hi Tom,

The way you do this isn't the most efficient way, 
Your result data get's piped trough 5 filters. A couple of them need to traverse your entire data-set. This is ok(ish) for a small data-set, but if it grows this becomes a serious lag.

For myself I would replace all those filters with just 1 custom made filter, that can extract the data you need in 1 go. Also I would optimize it in such a way, that if it has enough data to populate the view, you can stop, and return the result.

Also ad an 'track by' to your ng-repeat, then angular can make more efficient use of the data.

Regards

Sander

[Tom Leadbetter]

Hi Sander

Thank you for your feedback.

You suggest a custom filter - do you know of any examples of this? Or does it go by another type of name/function?

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom,

Yes I do know a particular sample you might find interesting:

  app.filter('startFrom', [function() {
    return function(input, start) {
      start = +start; //parse to int
      return input.slice(start);
    };
  }]);

Perhaps it looks familiar to you ;)

Regards
Sander

[Sander Elias]

Hi Tom,

On a more serious note, filters are not that hard to create, and the documentation is ok on this!

Regards

Sander

[Tom Leadbetter]

:) thanks Sander

So you suggest that one .filter can control all the different groups of checkboxes? Each checkbox group needs to pass a different value (sometimes it's a text match, sometimes a boolean)

--

You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom,

 

So you suggest that one .filter can control all the different groups of checkboxes? Each checkbox group needs to pass a different value (sometimes it's a text match, sometimes a boolean)

Yes indeed. You can pass a much parameters to your filter as you need. In your case I would use just 1, an array that holds all the selections you need. A while back I posted an 'or filter' example, perhaps you can take a look at that.

It is a bit more complex as a normal 'and filter', but it might point you into the right direction.

If you need some further help, don't hesitate to ask!

Regards

Sander

[Tom Leadbetter]

Hi Sander, just going through the example you made.

So this flattens each result into a text string? 

With say thousands of results, is that going to be good/bad for performance?

[Sander Elias]

Tom,

It depends, if you have a really large data-set, that might not be the right way to go. You need to do the math! Measure your dataset, and then decide. 

Need to run on mobile? there memory is a scarce resource. Running on desktop systems, go ahead and use whatever you need! But be aware of the memory usage. If your browser runs out of memory, your entire application will drop dead without any notification. Scripts will just stop running. 

I referenced it as an sample on how to make a custom filter. And it is of course just sample code, I wrote is as an answer on a different question. They had a 'messy' data-set and using the flattening trick, I didn't have to traverse all the sub-sub-sub objects.

If you have an simpler data-set, you don't need to flatten at all!

Regards

Sander

[Tom Leadbetter]

Thanks Sander.

Yes it needs to work on mobile. I like your example code, I'm just wondering if it's the 'best' way. Though I am aware there are many ways to skin a cat!

[Sander Elias]

Hi Tom,

On mobile? Nope, not the best way. It will eat up too much memory. What you need can be done without modifying your data. Don't put in extra things in memory unless you really must!

Regards

Sander

[Tom Leadbetter]

Ok. I think I'm going to need some hand-holding then if you, or anyone else, has the time and willingness :)

I think this could be useful to all though as a good example on how to use multiple filters on a large dataset with a nested/complicated JSON file.

--

You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom,

Go ahead, create a new plunk and see how far you get. If needed, I will hold your hand ;)

Regards

Sander

[Tom Leadbetter]

Well not getting very far I will admit!

I've seen many filter examples, a lot of which are very similar to this:

http://plnkr.co/edit/7PhNEADHW099w9kaqS7M?p=preview

or this

http://plnkr.co/edit/b0LMJzUiJy0NO9Hk0oWt?p=preview

But neither seems , to me anyway, to allow for multiple types of filters like my original example.

What do you make of them?

[Tom Leadbetter]

and there is this one that uses Underscore

http://jsfiddle.net/TahmidTanzim/N9Vqk/

again though the filter is specific to one thing (selectedCompany in this case)

[Sander Elias]

Tom,

You can combine a couple of those filters into 1 object. then send this object to your custom filter. In there you can handle the different things in 1 go!

The samples you provided are a starting point indeed. Combine this with the techniques I used in my or filter, and presto!

[Tom Leadbetter]

Ok, so I'm passing various things (checkboxData) to the filter (checkboxFilter)

http://plnkr.co/edit/G6k9MfEyox3vBaO53yQ9?p=preview

Am I on the right track?

I notice that when I tick/untick a checkbox it console logs things twice. Why?

[Sander Elias]

Allmost!

small mistake:

function(results, checkboxData){
      console.log(checkboxData);

    var result = [];

    return result;
    }

should be:

function(input, checkboxData){
      console.log(checkboxData);

    var result = [];

    input.forEach(function (item) {
        if (item does pass) {
            result.push(item)
        }
    })    
    return result;
    }

Regards
Sander

[Tom Leadbetter]

Thank you

Is the way I have constructed the object correct

{"bool=false":true,"details.category=Colorado":true,"details.category=Texas":true}

what I need query for each result is

result.bool

result.details.category

[Sander Elias]

It is indeed.

You need some vanilla JS to create the function that does the actual checking.

Regards

Sander

[Tom Leadbetter]

ok, struggling with this one.

For each result I need to go through each property in the object, so for example

does the result.details.category = Colorado.

Some other requirements, before I get too far down this road.

If no filters are checked (ie: on page load) then all results should show. If you filter and there are no matching results, it should say.

I will also have a budget range slider which returns two values (min, max) so this would need be incorporated also.

[Sander Elias]

Sure, 

you can do this just before returning:

```
if (result.length <1) { //no results!
   results = input // return everything!
}

```

Regards

Sander

[Sander Elias]

Oh and the slider isn't a problem also. 
Your check_if_this_item_needs_inclusion(item) function will get slightly complex though ;)

Regards,

Sander

[Tom Leadbetter]

thanks.

for the actually checking though, how do I go through the object and do the check?

something like?

    for (var prop in checkboxData) {

      if(checkboxData[prop] === true){

[Tom Leadbetter]

this is where I'm up to (not very far! :P )

http://plnkr.co/edit/G6k9MfEyox3vBaO53yQ9?p=preview

[Sander Elias]

Tom, 

I did tell you the  check_if_this_item_needs_inclusion(item) function will get slightly complex didn't I?

Are you just checking for True? then yes, that's a way to handle it. That's the part that decides that you need to check even deeper into your data.

You might even need a switch statement (Handle with care label attached here!)

Regards

Sander

[Sander Elias]

Hi Tom,

Not completely functioning, but it should you going for a while again: http://plnkr.co/edit/boVzYDP8bX5NG1aDNpTV?p=preview

Regards

Sander.

[Sander Elias]

Tom,

I updated the plunk a bit: http://plnkr.co/edit/boVzYDP8bX5NG1aDNpTV?p=preview

[Tom Leadbetter]

wow thank you. I'll dive in to this later. Though two concerns:
1) is getOwnPropertyNames supported by IE8?

2) there does seem a delay between checking a checkbox and the results being updated.

Must stress that I am not being ungrateful :)

On 22 January 2014 17:07, Sander Elias <sande...@gmail.com> wrote:

Tom,

I updated the plunk a bit: http://plnkr.co/edit/boVzYDP8bX5NG1aDNpTV?p=preview

--

You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom 

wow thank you. I'll dive in to this later. Though two concerns:
1) is getOwnPropertyNames supported by IE8?

Probably not. there is a lot of ES5 in here. I did only test chrome. 

 

2) there does seem a delay between checking a checkbox and the results being updated.

Yeah, it needs some more optimization! (still ;) )

I left some stuff for to yourself! for example, there are a lot of functions created that can be reduced at large!

Regards

Sander

[Tom Leadbetter]

bit overwhelmed to be honest, I wouldn't know where to start.

[Sander Elias]

Tom,

Well, just take a stab at it. Go ahead and try! You have a working starting point now. Worst case scenario, it gets slower ;)

If you don't get it going, drop me a note!

Regards

Sander

[Sander Elias]

Hi Tom,

I kicked it up a notch, and made it into a complete sample.

Have a look at it and let me know what you think of it!

Regards

Sander

[Tom Leadbetter]

This is great Sander thank you.

The checkboxes need to work slightly different though.

If I tick two Categories then I should be able to see results. A result can only have one category so if I tick Type 2 and Type 3, then I want to see results that are Type 2 or Type 3.

--

You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom,

That would make much more sense indeed! I have updated the plunk.


Regards
Sander

[Tom Leadbetter]

Hi Sander. This looks great.

Two requests if I may be so cheeky:

1) could you provide some in-depth code commenting so I can show the rest of the team? That way we can figure out what is going on :)

2) Can't get it working in IE8. Looks like an issue with Object.keys. Looks like there is a fallback https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/keys but still got issues. Do you have IE8?

--

You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/-TIYs_vfs74/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Leadbetter

[Sander Elias]

Hi Tom,

Well, you may be as cheeky as you want :-P.

1. Hmm, if time permits I will give it a go, but it won’t be very soon.
2. I don’t have an IE <11. if you need old browser support, include the shims you need. This will patch up non compliant browsers so they can run my ES5 code. I really don’t believe in keeping legacy browsers alive trough keeping all my stuff supported on those! If you want a modern web app, get a evergreen browser!

Oh, btw, I did some additional work on another version of the plunk, I think you will like that even better ;) this is a part off the html:

   <div class="colleft">
      <ul class="list-unstyled">
         <li ng-repeat='category in checkboxData' ng-init='showAll=false;catCount=category.data.length'>
            <h2 ng-click='showAll=!showAll' class='pointer'>{{category.title}}
               <small ng-hide="showAll || catCount<=10"> top 10</small>
               <small ng-show="showAll"> All {{catCount}}</small>   
            </h2>
            <ul class="list-unstyled">
               <li ng-repeat='data in category.data| orderBy:"count":true | limitTo:(showAll ? catCount : 10)'>
                  <label for='label{{data.name}}'>
                  <input type="checkbox" data-ng-model="data.filterUpon" id='label{{data.name}}'>
                  {{data.prefix}}{{data.name}} ({{data.count}}x)</label>
                  </input>
               </li>
            </ul>
         </li>
      </ul>
   </div>

Resulting screen:

How is that as a teaser? ;)

Regards
Sander

[Tim Deighton]

Hi Tom and Sander

I've been following your post with interest and decided to have a go at adapting your code  to fit in with my own project if you dont mind.

The problem I found is, where you have 'thing.details.category', I have 'Products_Categories.CategoryID'. But for some reason the filter cannot see CategoryID! It maybe something to do with the square brackets in results.json. 

I'm using Breeze, MVC to generate the json file. 

I'm sure a lot of folk may suffer the same problem.

Plunker = http://plnkr.co/edit/lWG9M5rdP5vVIlQR4fNM?p=preview

Any ideas for a workaround? I'm stumped.

Thanks a lot

On Tuesday, 21 January 2014 17:28:40 UTC, Tom Leadbetter wrote:

Hi all

We have a large data set, potentially several thousand results, though just currently a maximum of about 500. We have paging, ordering and filtering (powered by checkboxes). We have noticed that there is an ever so slight delay between selecting the checkbox and/or checkbox label and the dataset being updated. It is a minor delay, very minor and noticed it after looking at some large sites that use clientside filtering, such as SkyScanner:

http://www.expedia.co.uk/Flights-Search?trip=roundtrip&leg1=from%3AManchester%2C+England%2C+United+Kingdom+%28MAN%29%2Cto%3ANew+York%2C+NY%2C+United+States+%28NYC-All+Airports%29%2Cdeparture%3A14%2F03%2F2014TANYT&leg2=from%3ANew+York%2C+NY%2C+United+States+%28NYC-All+Airports%29%2Cto%3AManchester%2C+England%2C+United+Kingdom+%28MAN%29%2Cdeparture%3A29%2F03%2F2014TANYT&passengers=children%3A0%2Cadults%3A2%2Cseniors%3A0%2Cinfantinlap%3AY&options=cabinclass%3Aeconomy%2Cnopenalty%3AN%2Csortby%3Aprice&mode=search#departingModuleIndex=0&leg=1

Their checkboxes are rapid. Ours have no more than a second delay.

Unfortunately I can't share our site (though anyone wishing to take a closer look, please get in touch) but I have mocked up a demo

http://plnkr.co/edit/HWDtfmUzWcrqux762Fmu?p=preview

Typically this works very quickly (even locally with 2000 results) so it has me thinking that other JS in our app is causing the slow down. So that's something to look at.

However the main reason I come here is to get a sanity check on our filtering approach. Is it correct? Is it scalable? Any issues you see? In the plunker demo, in script.js, there is a commented-out console.log on line 33. If you enable that it will run the console 600 times, exactly three times the amount of results in the dataset. 

SkyScanner uses Backbone and therefore I wonder whether it then uses Underscore. Is Underscore going to be a benefit to our project?

Generally, any thoughts, tips, solutions you have, please say :)

Cheers

Tom

[Sander Elias]

Hi Tim.

The problem is in the lay-out of your data. This is from your data.

      "Products_Categories": [
         {
            "$id": "2",
            ...
            }
         },
         {
            "$id": "3",
            ...
         },
         {
            "$id": "4",
            ...
             }
         }
      ]
   },

Do you don’t have

Products_Categories.CategoryID

but

Products_Categories.[{CategoryID1...},{CategoryID2..}...]

That is more complex as the filter currently allows. Also, this is way more complex then a data structure should be. If you want I can make it to work, but this is not something I will do in my spare time!
Do you have somebody in your surroundings with knowledge on database design? If so, let him fix your database before going further with this.

Regards
Sander

[Tim Deighton]

Thanks a lot Sander,

my json is...

"Products_Categories": [

         {

            "$id": "2",

            "$type": "MvcApplication12.Areas.Admin.Models.Product_Category, Blitz",

            "ID": 176,

            "ProductID": 49,

            "CategoryID": 167,

            "RowVersion": 0,

            "Categories": null,

            "Products": {

               "$ref": "1"

            }

         },

         {

            "$id": "3",

            "$type": "MvcApplication12.Areas.Admin.Models.Product_Category, Blitz",

            "ID": 177,

            "ProductID": 49,

            "CategoryID": 163,

            "RowVersion": 0,

            "Categories": null,

            "Products": {

               "$ref": "1"

            }

         },

         {

            "$id": "4",

            "$type": "MvcApplication12.Areas.Admin.Models.Product_Category, Blitz",

            "ID": 178,

            "ProductID": 49,

            "CategoryID": 153,

            "RowVersion": 0,

            "Categories": null,

            "Products": {

               "$ref": "1"

            }

         }

      ]

you can see 'CategoryID' its also picked up by Angular in the HTML:-

  <div ng-repeat="catID in product.Products_Categories">

<div>

Cat ID: {{catID.CategoryID}}

</div>

</div>

I'm fairly sure the DB design is OK. Perhaps its the way Breeze generates the json file.

Anyway Thanks a lot 

I really appreciate your help. How much would you charge to fix it?

[Tim Deighton]

this is interesting...

I changed the json file (new plunk http://plnkr.co/edit/qo5OoSJRjYVbknSc2NJY?p=preview ) to a randomly generated one using http://www.json-generator.com/ (thanks)

and I still have the same problem with friends.id

now I'm really stumped.

[Ricardson Albuquerque]

Hi Tim,

Im a newbie on angular, but in your checkbox scope you have a list of friends, so u cant use friends.id, if you change for "friends[0].id" you see a change in filter, but your filter continue not working because only the frist line [0] is used on filter in this example.

Ricardson Albuquerque

--
You received this message because you are subscribed to the Google Groups "AngularJS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to angular+u...@googlegroups.com.

[Sander Elias]

Hi Tim,

The filter can handle array's. It can handle subarrays. However it can't handle sub arrays that exist out of objects. There is no way to know what you want to filter!
It is the same issue as with your previous sample, and the same thing Richardson pointed out.

You can contact me, if you really need this resolved, but it is at least 4 to 8 hours work. 

Regards

Sander

[Tim Deighton]

Aaaahhhhhhh,

I see! 

Thanks a lot Ricardson and Sander now I understand. []  = object?? not array??

Anything inside [] is an object? 

You can tell I'm new

Cheers guys.

Whats your hourly rate Sander?

[Sander Elias]

Tim,

Mail me privately do discuss things like my availability and ratings.

If you really want to make good use of angular, you should really pick up some JavaScript.

[] = an array. An array can be traversed using an index number

{} = an object. can be traversed, but not using an index number.

[{},{},{}] is an array of objects. There is no way of knowing if those objects are similar!

Regards

Sander