Clarification on .present function in PriorityQueueToListAggregator

14 views

Skip to first unread message

Piotr Sokólski

unread,

Aug 19, 2015, 6:29:48 PM8/19/15

to algebird

Hi,

I'd like to ask why is the function returning a sorted list in the aggregator grabbing the unsorted PriorityQueue iterator and then sorting it, instead of using PriorityQueue.tell(). I suspect it has to do with mutable PriorityQueue object, but 1) there is some mutable state already (in plus/reduce) 2) there seem to be other options, like copying the PriorityQueue prior to tell()ing available.

Oscar Boykin

unread,

Aug 19, 2015, 8:32:50 PM8/19/15

to Piotr Sokólski, algebird

The reason is because the "middle" type being mutable is often not such a big deal because it is hard to misuse that since users don't generally get their hands on it.

The result is passed back to the user, so we have a strong bias towards immutable objects in those cases.

For instance, in scalding with an Aggregator, you will never see the middle type (PriorityQueue in this case). similarly if you use .apply/applyOption method of Aggregator, or with the new spark code - same deal.

On Wed, Aug 19, 2015 at 12:29 PM, Piotr Sokólski <pietra...@gmail.com> wrote:

Hi,
I'd like to ask why is the function returning a sorted list in the aggregator grabbing the unsorted PriorityQueue iterator and then sorting it, instead of using PriorityQueue.tell(). I suspect it has to do with mutable PriorityQueue object, but 1) there is some mutable state already (in plus/reduce) 2) there seem to be other options, like copying the PriorityQueue prior to tell()ing available.

--
You received this message because you are subscribed to the Google Groups "algebird" group.
To unsubscribe from this group and stop receiving emails from it, send an email to algebird+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.