My attempt at a "chunk" filter doesn't work, for no obvious reason

[Ole Friis Østergaard]

Hi there!

I'm trying to come up with a good solution to this StackOverflow question regarding Angular and chunking up values into rows for use in Twitter Bootstrap layouts: http://stackoverflow.com/questions/11056819/how-would-i-use-angularjs-ng-repeat-with-twitter-bootstraps-scaffolding

There must be a better way to solve this problem than the current two replies, and I thought of making a filter called "chunk" that, given an array, would output an array of which contains chunks of the original array.

I've put my attempt up as a jsfiddle: http://jsfiddle.net/6vNcX/3/

The problem is that in practice the filter works, but if you look at the JavaScript console, you'll see that Angular has performed 10 $digest runs, and then given up stabilizing the view. For bigger data sets, this results in a great lag in the UI. The problem suggests that my filter's outputs on each $digest are not equal in the Angular sense, but I have a unit test which shows that it ought to work just fine (the last test):

'use strict';

describe('filters', function() {

beforeEach(module('chunkExampleApp'));

describe('chunk', function() {

it('should let non-Array input pass through', inject(function(chunkFilter) {

var input = 'this is not an array';

expect(chunkFilter(input)).toBe(input);

}));

it('should let empty arrays pass through', inject(function(chunkFilter) {

expect(chunkFilter([], 3)).toEqual([]);

}));

it('should chunk up arrays', inject(function(chunkFilter) {

expect(chunkFilter([1, 2, 3, 4, 5, 6, 7], 3)).toEqual([[1, 2, 3], [4, 5, 6], [7]]);

}));

it('should create equal outputs given equal inputs', inject(function(chunkFilter) {

var output1 = chunkFilter([1, 2, 3, 4], 3);

var output2 = chunkFilter([1, 2, 3, 4], 3);

expect(angular.equals(output1, output2)).toBeTruthy();

}));

});

});

This test runs with no errors. And as far as I can see from the angular.equals method implementation, there should be no reason for it not to give "true" on two outputs of my filter (given the same input), since it handles nested arrays just fine.

It would be really nice to get the "chunk" filter working, as it would make using Twitter Bootstrap grids with Angular a breeze. So if some of you clever guys can tell me what I'm missing, I'd be really grateful.

Thanks in advance!

[Ole Friis Østergaard]

Oh, I found out that it's been attempted before, but with same result: https://groups.google.com/forum/#!searchin/angular/chunk/angular/F0kNH69YqL0/mmENSj8dR1MJ

And since the old thread is dead, maybe there's no hope for this one... :-(

[Pawel Kozlowski]

Hi Ole!

On Fri, Aug 24, 2012 at 7:54 PM, Ole Friis Østergaard
<olef...@gmail.com> wrote:
> Oh, I found out that it's been attempted before, but with same result:
> https://groups.google.com/forum/#!searchin/angular/chunk/angular/F0kNH69YqL0/mmENSj8dR1MJ

What you (and others) are facing is a bit tricky combination of how
angular dirty checking and ngRepeat works.
Don't want to go into too much details here as we are really touching
inner workings of AngularJS and ngRepeat but in short ngRepeat will
keep references to elements from a previous run to change DOM elements
only for new objects in subsequent runs [1].
What is happening in your filter (and other filter examples that were
posted here) is that it modifies original object structure (creates a
new array, but more importantly, creates new object in an array for
each and every run!).

Now, AngularJS figures out what to repaint based on dirty checking [2]
and it will run objects comparison waiting for the model to
"stabilize". It keep continue dirty checking for some time (10
iterations) and if it still see changes it will throw the error you
see. So basically the issue is that for AngularJS the model changes
all the time.

The issue you are facing is really the combination of dirty checking
mechanism with ngRepeat. Please observe that you won't see the issue
if you don't use ngRepeat:
http://jsfiddle.net/pkozlowski_opensource/ZADB7/1/ (and this is why
your test is passing)

For now I don't see how this could be solved in a filter (the only
approach that would probably work would be to keep references to the
filtered arrays so we are not systematically re-creating new objects
but this might not be good memory-wise). It would be great to have
some insight from the AngularJS core team.

What I can suggest for now is to move the filter logic to a
controller. This way transformed array will be changed only in
response to the source array changes and this won't disturb ngRepeat.
Here is the working jsFiddle:
http://jsfiddle.net/pkozlowski_opensource/MBpRd/3/

Hope this helps but if you've got further questions, don't hesitate to
ask. I will probable continue looking a bit into inner workings of
ngRepeat to see if there is any hope of putting this logic into a
filter.

Cheers,
Pawel


[1] https://github.com/angular/angular.js/blob/master/src/ng/directive/ngRepeat.js#L83
[2] http://stackoverflow.com/questions/9682092/databinding-in-angularjs

[Ole Friis Østergaard]

Hi Pawel

Thanks for your reply!

So basically you're saying that ng-repeat doesn't use "angular.equal" as normal watch expressions do, but its own heuristics for checking changes? I'll have to look at the Angular code, I guess. To me it sounds like either a bug or some sort of performance hack?

I also thought of caching the earlier results from my filter, but it's just too ugly, and I'll never be able to get it right. It'll result in lots of quirks.

Letting my controller chunk up the array is what I'm doing now, but it's really a view detail which is leaking into my controller. I don't like it. Besides, it means I cannot do nice filter chaining in my view and e.g. do sorting, filtering, etc. in combination with the chunking, which is a real shame.

[Ole Friis Østergaard]

Looking at the ng-repeat code, it looks like defining a $$hashKey function on my arrays should do the trick, as ng-repeat uses a hash queue for detecting changes. (Which I believe should be considered a bug, but YMMV.) I'll try it out and get back.

[Godmar Back]

On Sat, Aug 25, 2012 at 6:20 AM, Pawel Kozlowski <pkozlowski...@gmail.com> wrote:

For now I don't see how this could be solved in a filter (the only
approach that would probably work would be to keep references to the
filtered arrays so we are not systematically re-creating new objects
but this might not be good memory-wise). It would be great to have
some insight from the AngularJS core team.

I had completely put you on the core team, Pawel, with your helpful and insightful comments.

Who is on the core team, though?  Does the core team read this list?  In the short time I've joined, I've seen numerous questions repeated over and over, which arise mainly due to the low quality (or total lack) of documentation. Perhaps a core team member could subscribe to the list, and whenever a frequent question arises, update the documentation?

 - Godmar

[Pawel Kozlowski]

Hi!

On Sat, Aug 25, 2012 at 3:30 PM, Godmar Back <god...@gmail.com> wrote:
>
> I had completely put you on the core team, Pawel, with your helpful and
> insightful comments.

He, he, this is flattering but I'm not part of the angular team :-)
I'm just active here since it is for me a way to learn more about
angular. But glad to hear that you find my posts helpful!

> Who is on the core team, though?

You can see most of them being part of the angular organization:
https://github.com/angular

> Does the core team read this list? In the
> short time I've joined, I've seen numerous questions repeated over and over,
> which arise mainly due to the low quality (or total lack) of documentation.

Yes, there are responses from the core team from time to time. But I
guess this list is a bit a victim of its own success as most of the
questions are dealt with by the community.

Regarding the documentation I really kind of like it. I mean, I've
seen far worst things for frameworks. There is ongoing
documentation-improvement effort so things should be getting better.

> Perhaps a core team member could subscribe to the list, and whenever a
> frequent question arises, update the documentation?

Yeh, this is the part I'm missing as well, having those guys stepping
in and dealing with tricky questions. Fortunately the code base of
AngularJS is super-clean (IMHO), well structured and thoroughly tested
so there is always an option of getting hands dirty with the source
code (good learning experience!).

Cheers,
Pawel

[Ole Friis Østergaard]

Oh well, I tried defining a $$hashKey function, but it has to be defined in terms of hashKey(...) of the objects in the arrays. This function is defined in apis.js in the Angular source, but apparently wrapped in a way so that my code cannot access it. My thought was something like this:

(...for each of the chunks in the resulting array...)

chunk.$$hashKey = function() {

  hash = 0;

  for (int i=0; i<this.length; i++) {

    hash = hash*31 + hashKey(this[i]);

  }

  return hash;

}

But my filter is not able to call "hashKey". Except if it's available through some object somewhere...? However, I guess I'd better give up now. It's a shame, because Angular is really missing something like a "chunk" filter, IMHO.

[Pawel Kozlowski]

Hi again!

So, I've spent some time digging into what is going on with this
repeater / filter stuff and made some good progress so will share
here.

First of all, here is the minimal reproduce scenario with some log
statements to illustrate what is going on:
http://jsfiddle.net/pkozlowski_opensource/nRGTX/1/

Checking the console we can see that after all the children from the
inner ngRepeat are printed we do see another $digest cycle. This will
result in a filter being re-evaluated, new objects created etc. etc.
till 10 $digest cycles are reached and the exception thrown.

The thing about ngRepeat is that this directive maintains an internal
list of objects=>DOM mappings (it is more complex than this, but it
doesn't matter for this particular issue). I believe that this is
maintained as a perf optimization to minimize DOM elements creation /
removal. But since we need a mapping from an object to a DOM element
here means that angular needs to decide what to use as object
identity. It could be its reference or some hash and AngularJS uses
hashes. The funny thing is that those hashes are calculated in an
interesting way: it is simply unique identifier. Once calculated the
hash is stored as the .$$hashKey object property.

Now, as for the solution to the problem you were very close when you
started to look into .$$hashKey. The only missing piece was that
.$$hashKey can be either a function (so each object can decide how to
calculate a hash) or simply a value.

Equipped with the above knowledge we could very easily fix the minimal
reproduce scenario by simply coping over .$$hashKey to indicate that
even if we are creating new objects those have the same identity.

Here is the working jsFiddle:

http://jsfiddle.net/pkozlowski_opensource/zvpVg/2/ (it uses nexuid
function from angular)

and here is your original jsFiddle solved using the same approach:
http://jsfiddle.net/pkozlowski_opensource/6vNcX/6/
Please note that the above fiddle uses very naive method of the hash
calculation that works for primitive numbers. In more general scenario
we would need a more robust solution (probably using combination on
nextuid and another technique for primitives).

To summarize: angular does some "magic" to optimize DOM manipulations
in ngRepeat. Those optimizations don't play nice with filters that
re-create new objects. Fortunately there is a way to inform angular
that 2 objects should be regarded as "the same" even if references are
different ($$hashKey).

When you know what is going on it kind of makes sense and probably
those guys have measured things and figured out that all those
optimizations are necessary. At the same time it is true that if you
don't know it might be hard to figure out what is happening....

Hope this removes a bit mystery from this whole ngRepeat + filter
story. Probably should write a blog post about it...

Cheers,
Pawel

[Ole Friis Østergaard]

Hi Pawel

Thanks for your solution! Meanwhile I found my own working solution, in which I simply set $$hashKey of a chunk to the index of that chunk. That works just as well, though it has a potential problem, which I believe your solution has as well: ng-repeat will consider two chunks the same even if they contain different values. In my solution this happens all the time, since the first chunk always gets $$hashKey 0, etc. In your solution, it happens when the string representation of different chunks are incidentally the same - or for the chunks [12, 3, 4] and [1, 23, 4], for example. However, this doesn't really matter in practice, since the output of the "chunk" filter will typically be used for a later ng-repeat, and the differences will be caught there.

I guess your implementation is the most elegant. I'll insert mine at the end of this post. I've refactored out some functions to make it more readable, which of course makes it more verbose.

Both solutions have the major problem that they are bound to the internal workings of Angular... but I guess it's the best we can do.

Thank you so much for your time, Pawel. Everybody now has a working "chunk" filter, so that using Twitter Bootstrap with Angular is not a dark art anymore. I hope the Angular developers will create an official, clean "chunk" filter sometime in the future.

myApp.filter('chunk', function() {

  function chunkArray(array, chunkSize) {

    var result = [];

    var currentChunk = [];

    for (var i=0; i<array.length; i++) {

      currentChunk.push(array[i]);

      if (currentChunk.length == chunkSize) {

        result.push(currentChunk);

        currentChunk = [];

      }

    }

    if (currentChunk.length > 0) {

      result.push(currentChunk);

    }

    return result;

  }

  function defineHashKeys(array) {

    for (var i=0; i<array.length; i++) {

      array[i].$$hashKey = i;

    }

  }

  return function(array, chunkSize) {

    if (!(array instanceof Array)) return array;

    if (!chunkSize) return array;

    var result = chunkArray(array, chunkSize);

    defineHashKeys(result);

    return result;

  }

});

[Jason Kuhrt]

It appears that an upcoming feature to angular will solve the problems presented in this thread?

https://github.com/angular/angular.js/commit/61f2767ce65562257599649d9eaf9da08f321655

Thanks to both of you for spending the time you have on this issue. It has been helpful to readers like myself,

Jason

[Jason Kuhrt]

Actually on a more careful read I don't think chunking could be resolved by a `track by` feature. Instead the chunking filter would have to supply its own tracking mechanism as you two have presented.

[Jason Kuhrt]

I've created a gh issue following the topic of this thread:

https://github.com/angular/angular.js/issues/2302

If you guys could chime in with clear insight than I can that would be lovely.

Jason

[z...@caudate.me]

I asked this question on stackoverflow:

http://stackoverflow.com/questions/17562312/digest-iteration-error-for-filters-written-for-ng-repeat-in-angular-1-1-x

but wanted to post here as well.

What is happening with this issue? Its very annoying

[Dave Barker]

Well I've figured out a hack that gets it working, you serialise each chunk and then unserialise again as you're repeating through it. Probably terribly slow and I'm not proud of myself but here you guys go http://plnkr.co/edit/XT3tJMlBKvIodYUkoVkO?p=preview

Cheers, Dave.

[Szymon Nowak]

I've got almost exactly the same filter and the way I've did it for now (it still sucks) is to use _.memoize function from underscore library with custom hash function that iterates over input array and joins all $$hashKey values.

group = (array, num) ->

  # return grouped data

_.memoize group, (array, num) ->

  return array if not angular.isArray(array)

 

  hash = ""

  hash += item.$$hashKey for item in array

  hash

[Szymon Nowak]

Small update. It looks like Angular generates $$hashKey later than group filter was executed and thus item.$$hashKey returned undefined for all items.

Here's an updated version (in CoffeeScript):

angular.module('filters')

.filter 'group', ->

  group = (array, num) ->

    return array if not angular.isArray(array)

    grouped = []

    grouped[i] = [] for i in [0...num]

    for item, index in array

      grouped[index++ % num].push(item)

    grouped

  _.memoize group, (array, num) ->

    return array if not angular.isArray(array)

    hash = ""

    for item in array

      item.$$hashKey = _.uniqueId() unless item.$$hashKey?

      hash += item.$$hashKey

    hash

No idea why nextUid() function isn't public.

[Chris Zheng]

I had all but given up on this...

You've inspired me to grit my teeth and try again :) 

--
You received this message because you are subscribed to a topic in the Google Groups "AngularJS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/angular/IEIQok-YkpU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to angular+u...@googlegroups.com.
To post to this group, send email to ang...@googlegroups.com.
Visit this group at http://groups.google.com/group/angular.
For more options, visit https://groups.google.com/groups/opt_out.

[Алексей Грабов]

if you want return new collection, you just need to save javascript links.
I wrote factory for this case:

app.factory('linker', function () {
  var links = {}
  return function (arr, key) {
    var link = links[key] || []

    arr.forEach(function (newItem, index) {
      var oldItem = link[index]
      if (!angular.equals(oldItem, newItem))
        link[index] = newItem
    })

    link.length = arr.length

    return links[key] = link
  }
})

you can see here how it works: http://plnkr.co/edit/2Uc5zsFgVnK3ltHOUUQx?p=preview