emcee not remembering definitions using multiprocess

[Nick Sorabella]

Hello!

I'm trying to fit a simple gaussian model to some data using emcee while utilizing parallel processing. I am using multiprocess instead of multiprocessing as I am working in Jupyter notebook where multiprocessing often runs into problems for some reason. For example, I was able to substitute multiprocess for multiprocessing in the parallelization example of emcee's website and it worker perfectly.  Currently, the gaussian code is saying that one of my functions is not defined even though it was defined earlier. This same issue occurred for a much more complicated model I am working on.  Not sure why it isn't being carried over once the parallelization starts. Specifically the error occurs on the very last line saying that "log_prior" is not defined. The author of Multiprocess, Mike McKerns, reviewed the problem and stated "I see that the pool's map function is not being called within the __main__ block, emcee is calling it somewhere in that code." and suggested I reach out here at emcee for a possible solution.  The code can be found attached.  Any suggestions on how to fix this issue would be much appreciated!

[Dan Foreman-Mackey]

Unfortunately I can't reproduce your issue - running the code you sent (on google colab) works fine for me after installing multiprocess! See if you can isolate the issue even more because it seems like it's not a fundamental issue.

Best,

Dan

On Thu, Oct 27, 2022 at 11:25 AM Nick Sorabella <sorab...@gmail.com> wrote:

Hello!

I'm trying to fit a simple gaussian model to some data using emcee while utilizing parallel processing. I am using multiprocess instead of multiprocessing as I am working in Jupyter notebook where multiprocessing often runs into problems for some reason. For example, I was able to substitute multiprocess for multiprocessing in the parallelization example of emcee's website and it worker perfectly.  Currently, the gaussian code is saying that one of my functions is not defined even though it was defined earlier. This same issue occurred for a much more complicated model I am working on.  Not sure why it isn't being carried over once the parallelization starts. Specifically the error occurs on the very last line saying that "log_prior" is not defined. The author of Multiprocess, Mike McKerns, reviewed the problem and stated "I see that the pool's map function is not being called within the __main__ block, emcee is calling it somewhere in that code." and suggested I reach out here at emcee for a possible solution.  The code can be found attached.  Any suggestions on how to fix this issue would be much appreciated!

--
You received this message because you are subscribed to the Google Groups "emcee users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emcee-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emcee-users/14e125d2-e801-451b-97f0-1a8334b60ec8n%40googlegroups.com.

--

Dan Foreman-Mackey

Research Scientist

Flatiron Institute

https://dfm.io

[Nick Sorabella]

Hello!

Thanks for the response!  Did you happen to run the code in Jupyter notebook?  Honestly, I have a strong suspicion that Jupyter is part of the problem, but , unfortunately, it is my compiler of choice!

-Nick

[Nick Sorabella]

Just an update, I did run the code successfully in vscode (still not running in Jupyter notebook).  I found the parallelized version to be slower than doing it single threaded.  Not sure if this is a case of a function that is better solved single threaded or if something else is going on.  Opening task manager while running the code seems to show only 2 threads being utilized even though I have 8.