Google Groups

Re: fold over a sequence


Stuart Sierra Mar 12, 2013 11:26 AM
Posted in group: Clojure
Hi Paul,

This might be an interesting contribution to clojure.core.reducers. I haven't looked at your code in detail, so I can't say for sure, but being able to do parallel fold over semi-lazy sequences would be very useful.

-S



On Tuesday, March 12, 2013 9:34:43 AM UTC-4, Paul Butcher wrote:
So this turned out to be pretty easy. I've implemented a function called "foldable-seq" that takes a lazy sequence and turns it into something that can be folded in parallel. I've checked an example program that uses it to count words in a Wikipedia XML dump into GitHub:


The code for foldable-seq is here:


On my 4-core MacBook Pro, I see a 40 second runtime without parallel-seq, 13 seconds with.

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher

On 11 Mar 2013, at 13:38, Paul Butcher <pa...@paulbutcher.com> wrote:

On 11 Mar 2013, at 11:00, Marko Topolnik <marko.t...@gmail.com> wrote:

The idea is to transform into a lazy sequence of eager chunks. That approach should work.

Exactly. Right - I guess I should put my money where my mouth is and see if I can get it working...

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher