CompositeModel objects does not compatible with paralleled emcee fitting

[wenke ren]

Hi,

I major in astronomy and using your package for spectral fitting. Thans for you excellent works.

I recently try to use multi-processing to accelerate the fitting calculation but encountered some problem. I have narrowed down the problem to this script. It is clear that the problem will only raise when I try to fit with CompositeModel.

Please see the attachment for the demo.

While perform the fitting with emcee, the code works fine for simple model under any number of workers. It also compatible for CompositeModel with workers=1. The Error raise for workers>2 as follow:

----------------------------------------------------------------------------------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/wenke/data/20230803DESI/20250306CaseStudy/code/Bug_Explore.py", line 57, in <module>
    fit_result = combined_model.fit(
                 ^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/lmfit/model.py", line 1177, in fit
    output.fit(data=data, weights=weights)
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/lmfit/model.py", line 1570, in fit
    _ret = self.minimize(method=self.method, params=self.init_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/lmfit/minimizer.py", line 2330, in minimize
    return function(**kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/lmfit/minimizer.py", line 1405, in emcee
    output = self.sampler.run_mcmc(p0, steps, progress=progress, **run_mcmc_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/emcee/ensemble.py", line 443, in run_mcmc
    for results in self.sample(initial_state, iterations=nsteps, **kwargs):
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/emcee/ensemble.py", line 344, in sample
    state.log_prob, state.blobs = self.compute_log_prob(state.coords)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/site-packages/emcee/ensemble.py", line 489, in compute_log_prob
    results = list(map_func(self.log_prob_fn, p))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/multiprocessing/connection.py", line 205, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wenke/opt/anaconda3/envs/DESI/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'CompositeModel.__init__.<locals>._tmp'

----------------------------------------------------------------------------------------------------------------------------------------------

I quickly went through the conversations here and searched relavent key words in your documents. I didn't notice any relavent discussion or warning. Please let me know if this is a known problem and is there any hope to fix this.

Best wishes,

Wenke

[Matt Newville]

Hi Wenke,

I would guess that emcee needs to be told to use the `dill` package, which replaces standard Python pickling with something that works to serialize functions and methods.  If you `pip install dill`, and then add `import dill` to your script it might work. Multiprocessing with Python's built-in pickle is fragile. The `dill` package tries to help, but we cannot offer much more advice than that.

I'll also say that my ability to support the use of `emcee` within `lmfit` is minimal.   If it works, that's nice, but I cannot help if something goes wrong.  

 

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lmfit-py/5d2bcb95-d5c9-457b-8cfd-4756243dbc86n%40googlegroups.com.