Parallelizing EVP solves

[evan...@colorado.edu]

Hi everyone,

Last week, Ben and I discussed that it would be useful to set up a dedalus script that runs a number of eigenvalue problems (for the same equations but at different parameters/wavenumbers) in parallel.  Doing so would allow for mass exploration of an problem space at high resolution in essentially no wall time since it's an embarassingly parallel problem.

While I was looking into that last week, I ran into a number of problems with the built-in parallelization of dedalus domains/problems when running in MPI.  Today I hacked around in my local copy of dedalus and got a parallel EVP solver up an running for MY computer, but at this point some of my files are...ugly.

So my question is: is this a feature (parallelization of eigenvalues over a parameter space) that you would like to include in dedalus for anyone to grab and use?  If so, I'll fork the repo, polish my edits, and make a general function with documentation that (hopefully) anyone can use.  If not, I'll keep a (messy) local version around.  Just wanted to check on what everyone else thought before I ran too deep down the rabbit hole.

-Evan

[j s oishi]

Hi Evan,

Susan Clark and I have worked on this a bit. I've copied Susan on this email; she probably has some thoughts.We did it without MPI parallelism. I think Susan used python multiprocessing, but I'm not 100% sure.

j

--
You received this message because you are subscribed to the Google Groups "Dedalus Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-dev...@googlegroups.com.
To post to this group, send email to dedal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/e352e852-72a5-4467-9df4-8a1c3a5f0840%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Lecoanet]

Hey all,

I think I've gotten half way to getting this working myself.  I modified the distributor initialization to know if it was being run in parallel or serial.  If you tell set it up in serial, then you can use MPI to have a bunch of different cores run their own simulations without trying to talk to anyone else.  I did this so I could run postprocessing analysis tasks (like calculating volume integrals) in parallel, but I think it would work equally well for eigenvalue problems.

This was a pretty straightforward modification ~ about 10 lines of code.  If this is simpler than your solution, maybe it would be a cleaner way to put this functionality into the code.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAAOZTN92SFLGa%2BOoe%3Dyq4osZ3HYb5oBqKU%2Bz1ZfkRwp2B39hPg%40mail.gmail.com.

[Keaton Burns]

Hi all,

Can you guys say a little more about what sort of functionality you’re aiming for? As with the other solvers, solver.pencils is the list of local pencils, so I think running an EVP using MPI and looping over solver.pencils is the trivial way to solve an EVP in parallel for all the horizontal modes in a domain.

To look at specific horizontal wavenumbers, you can also do a 1D domain and use the parameters/substitutions to map dx() to any kx, etc. Daniel’s approach (or something like it, e.g. passing arbitrary MPI communicators to the distributor init) sounds like any easy way to parallelize this and other problems.

Are there other use-cases you guys have in mind?

-Keaton

> To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAJoYf%3DhgvqipyfrxghCAB9hkt5v9iOooFBJ3CK7TL3BJKLm0dg%40mail.gmail.com.

[j s oishi]

From Susan!

---------- Forwarded message ---------
From: Susan Clark <susanc...@gmail.com>
Date: Mon, Jul 6, 2015, 18:56
Subject: Re: Parallelizing EVP solves
To: j s oishi <jso...@gmail.com>

Whoops, I don't have permission to post to dedalus-dev. Can you pass this along?

--

Hi Evan / Jeff,

Yep, I used python multiprocessing to do exactly this, i.e. solve eigenvalue problems while searching a big parameter space. It's pretty straightforward -- my approach was to use a Pool object which asynchronously calls some function (containing your eigenvalue problem) with some args (your parameters). It's ~10 lines of code which I'm happy to share or explain further. 

Susan

On Mon, Jul 6, 2015 at 6:18 PM, j s oishi <jso...@gmail.com> wrote:

[evan...@colorado.edu]

Hi everyone,

Thanks for the quick responses and input!  I'll take a look at Daniel's and Susan's solutions to see which are easier -- but both sound easier than my workarounds (I added an element to the domain class and some if statements throughout the code which acted as a switch to make it parallel or not).

Keaton, in addition to exploring horizontal wavenumbers I'm interested in exploring parameters that go into my problems.  So, specifically, I'm exploring the value of Ra at which convection starts for the FC equations, and I want to be able to explore, e.g. a range of values from 10^3 - 10^6 as given by a numpy array.  I've implemented a function which does this, and I could clean it up quite easily with one of these other solutions, then (if it works for general problems) make it publicly available.

I'll keep you updated with anything exciting that comes up.

-Evan

[evan...@colorado.edu]

Hi all,

A quick note: Pools work great for parallelizing EVPs (using the pool map function), but I've heard some scary things about using python's built-in parallelization on non-private clusters.  Whereas when you use MPI you specify the number of processors and get a clear allocation for your whole run, pools just grab processors on the go.  According to Baylee, she's seen Python's built-in parallelization creep onto processors it's not supposed to on supercomputers.  So a better alternative is probably best...I'll look into MPI.

-Evan

[evan...@colorado.edu]

Daniel,

On the note of MPI -- I've looked at the documentation to try to find how to set the distributor like you said, but I've been unsuccessful in learning how to tell it to run in serial.  If you have an example of some code that does that (or a link to the functions that do that), I'd appreciate getting to look at them!

Thanks,
Evan

[Keaton Burns]

Hi all,

I’ve added the ability to specify the base communicator for a problem's distribution as a keyword argument for the domain class. Using MPI.COMM_SELF, this lets you simultaneously run different simulations on each process in an MPI environment.

To demonstrate how to do this, I also added a parallel EVP example that computes growth rates in no-slip rayleigh-benard convection over an array of horizontal wavenumbers, which are specified as problem parameters. You could analogously loop over any other parameters, like the Rayleigh number, as well.

-Keaton

> To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/02c56672-9ca3-41f4-9427-18666879012e%40googlegroups.com.

[evan...@colorado.edu]

Hi Keaton,

Apologies for the (extra super) slow reply.  I spent a week struggling with LASP's internet and another out of town.  Thanks for the addition to the EVP and the example, something like this is what I was looking for.

-Evan