Using dask for parallelizing calls to minimize

[Tomas Olsson]

Hi,

I am running several calls to minimize in a sequence, but I would like to run them in parallel.

However, I get the following errorr when ran in parallel:

AttributeError: 'MinimizerResult' object has no attribute 'params'

The sequential program is running correctly but when I try to make it run in parallel it does not work.

I use http://dask.readthedocs.io for parallelisation using calls to delayed and compute.

Best regards

Tomas

[Matthew Newville]

Hi Tomas,

Sorry, I know next to nothing about dask, or how it enables parallelization.  

Minimization methods such as leastsq / MINPACK might be a challenging problem for parallelization.  The essential problem doesn't really lend itself to parallelization very well - the solvers want to be able to run the python objective function at will to determine the next best step.  So, until someone puts it in the Fortran/C level, parallelizing any one fit will be very, very difficult.  I'm not aware of anyone even thinking about working on that.

In principle, one could do multiple independent fits simultaneously using multiprocessing.  This is not necessarily easy either, as the data and objective function have to be sent to each process using Pickle, which is pretty limited.   It might work to use the dill package instead of Pickle, as it has better support for correctly serializing complicated data such as methods. Perhaps dask could also help with this.

Hope that helps.  If you'd like more direct answer, please include actual code and the complete error message.

--Matt

[Andrew Nelson]

I've used a subclass that inherits `lmfit.Minimizer` to fit hundreds of datasets in parallel using `multiprocessing.Pool.map`. I can share the jupyter notebook if you're interested. Unfortunately I'm not familiar with dask, so I won't be able to help you with that, but if you do iron out the problems I'd be interested to know the solution.

[Tomas Olsson]

Thanks Andrew,

I would very much appreciate if you could share your code with me.

My email address is: tol @ sics.se

/Tomas

[Tomas Olsson]

Hi.

 I have actually figured out what the problem was. I created a Parameters instance for each call to minimize but it was turned into an instance of a dict when serialized and deserialised. This data structure is apparently not managed by the minimize function, and it is just ignored, without any exception until the parameters are used. So I convert the dict to a Parameters instance before suing it, and now it seems to work.

/Tomas

[Tomas Olsson]

Well, it still does not work so your code would be great to have.
/Tomas

> --
> You received this message because you are subscribed to a topic in the Google Groups "lmfit-py" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/lmfit-py/XR9e8kcAFwI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to lmfit-py+u...@googlegroups.com.
> To post to this group, send email to lmfi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/lmfit-py.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CAAbtOZfJ%2B5UsUoa62hx3vYpOyU-m%2BGQGu6L1_wcM%2BKjRW%3DyLjg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

Tomas Olsson
t...@sics.se

Swedish
SP, Swedish ICT och Innventia går samman i RISE för att skapa en samlad institutssektor och bli en starkare innovationspartner för näringsliv och samhälle. Vid årsskiftet byter vi namn till RISE. Läs mer på www.ri.se/utveckling-av-rise

English
The RISE institutes SP, Swedish ICT and Innventia are merging in order to create a unified institute sector and become a stronger innovation partner for businesses and society. At the end of the year we will change our name to RISE. Read more at www.ri.se/en/about-rise

[Daniel Schick]

I am also running into a similar problem.

I try to use dask for parallelization of a for-loop to do a lot of independent fits in parallel.

I essentially receive errors when pickleing the parameters:

TypeError: cannot pickle 'weakref' object

Has there been any progress in this direction?

It would be really great to use this great package in parallel on a multicore workstation.

Best

Daniel

[Matt Newville]

Lmfit seems to be good for demonstrating how most of the python attempts to do multiprocessing are flawed.  Basically, python's multiprocessing tools expect data shared between processes to be pickleable, but python itself does not guarantee that all objects can be pickled. Using the dill package might help, and perhaps dask or other tools might be able to have another workaround.   But lmfit is pure python and not doing anything sneaky.  It is conceivable that we could do something about this, but I don't know what that would be.

You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/5973bcfc-550a-49c4-94fc-965d0d913b02n%40googlegroups.com.

[Daniel Schick]

Hi Matt,

Thanks for your quick reply. I fear it might be even related to my custom fit function, which could be the reason why I cannot pickle the model.

However, I agree, that this a more general problem related to python and the ability to pickle objects.

I will try a bit with pickle to determine the origin of the problem.

Best

Daniel

[Michal Pelach]

Hi Daniel, Tomas,

have you found any solution or workaround in case of parallelizing. Did you just left this idea, what was your final result.

thanks

Michal

[Andres Zelcer]

Hi All

    I think there is some confusion about the "parallelizing" word. Most of the minimizing algorithms accesible by lmfit are not parallel (https://en.wikipedia.org/wiki/Parallel_algorithm) nor parallelizable.

    On the other hand, I never had issues running multiple fits in parallel (same fitting function, different datasets and parameters). For simple fits, the cost of pickling is larger than the benefits of multiprocessing. Only in the case of computationally expensive functions, or of very large datasets, running parallel fits is convenient. I do not think dask fits for this purpouse.

    If you want a simple example of running parallel fits using multiprocessing, I can upload a gist to github. In that case, it is important to be aware of what can be done and what can't be done in the fitting function.

    Best,

           Andrés

[Marc W. Pound]

Hi Andrés,

I'd be interested in seeing your gist.   I have the case where I want to run the same fitting function on each pixel of an image.  Images can be megapixels, so any method that can run faster than looping over every pixel is of interest.

thanks,

Marc

[Michal Pelach]

Hi Andrés,

well yes, thats what i thought. Minimization on its own will not get split, so that will be computed on one core, but every core can have its own minimization algorithm running. I hope thats what you meant, roughly.

Yes Andrés simple or complex example woud be really helpfull.

michal

[Andres Zelcer]

Hi, Marc, Michal.

    Here's a small snippet. I hope you find it useful: https://gist.github.com/azelcer/39c0b8acbcd18576392bb9b841a7a4d9.

    Marc: please note that if there is any kind of correlation between the pixels of your image, it is likely that this approach is *not* the way to go, as the fitting parameters for each pixel are completely unrelated to those of the other pixels.

    Michal: that's what I meant. I am sorry if I wasn't clear, but I am not a very fluent English speaker.

    Best regards,

        Andrés.

[Marc W. Pound]

Hi Andrés,

Thanks for the example.    I do understand about correlated pixels.    In the future, I will explore regularization techniques, but for now a simple example of using Pool is helpful.    I'd be happy to hear any tips on regularization that people may have to share.

best,

Marc

[Michal Pelach]

hey Andres, great you added this, i finaly had time to test this, but i have tried on two different machines and i am hving some difficulties. First it produced a RuntimeError. I tried to fix it from error description so i added if name=="main": and wrapped paralel "unit" in function and than called the function from within if name=="main": but it results in strange behaviour printing results multiple times even from sequential fit. But i guess there is some other mistake. Does this snippet work in you environment?

[Andres Zelcer]

Hi Michal

    I've updated the code so now it *should* work on windows. Sadly, I have no way of testing it now. Please let me know how it works.

       Andrés