parallel algorithms in spire

[Erik Osheim]

Hey folks,

Despite the silence on this list, things are definitely moving. I'll
try to write a status update soon, but for now, I'm interested in
people's opinions on an important topic which I've been deferring for
awhile now:

How should Spire support parallel numerical algorithms?

The email today about segmented sieves (and the link to this paper
[1]) got me thinking about ways to parallelize Spire's segmented sieve
[2]. But I realized that there isn't currently any consensus over how
Spire should support possible parallelization in general.

What do folks think? Should Spire support an implicit
ExecutionContext? Actors? ForkJoinPool? Raw threads? Something else?

I think we should have pluggable/flexible strategy because:

1. Some applications will want things to "just work"
2. Some applications will need/want to use custom pools/settings
3. Some applications won't want to use parallelism at all
4. Spire may end up being used as an example by other projects

I'm a bit leery of adding an external dependency (e.g. on Akka), but
would definitely consider doing so, especially if we could limit it to
a spire-parallel subproject/jar.

What do you all think? Am I overthinking things? Is there an obvious
strategy? Please share your thoughts.

Thanks!

-- Erik

[1] http://www.scribd.com/doc/228155369/The-Segmented-Sieve-of-Zakiya-SSoZ

[2] https://github.com/non/spire/blob/master/core/src/main/scala/spire/math/prime/Siever.scala

[Rex Kerr]

My recommendation would be to hold off for a while to see how various reactive / CPS / streamy / etc. stuff pans out.  You'd like to adopt the easiest thing that still is plenty fast, will cover most use cases, and won't immediately disappear.  It's far from clear right now what that will be.

More important I think is to give hooks to allow independent computations to be done independently.  Then you can adapt it to whatever framework seems reasonable or people can roll their own.

  --Rex

[Jonathan Merritt]

What do people currently use for the kind of "embarrassingly parallel" problems that Rex mentions in his last paragraph?  I've had some success using Scala's parallel collections, but most of my tasks are (very) CPU-bound with nice, large, consistent chunks of computation per element.

--

Jonathan Merritt