Cannot use class as objective function anymore.

[Felipe Eler]

Hello, lmfit community!

I was using the package with the following versions:

lmfit=1.2.2 emcee=3.1.2 numpy=1.26.4

After I updated to the last versions:

lmfit=1.3.2 emcee=3.1.6 numpy=1.26.4

My code stopped working when using emcee as the method.

There is a curiosity: It only works if I use a single worker (no parallelism).

If workers > 1, the code returns that it can't pickle some part of my objective function.

By the way, my objective function is a class.

Is there any solution or should I stop using classes?

The class:

class PCCMatchObjective:

    def __init__(self, cpr_model, b, pc_data=None, pc_error=None, pc_weight=None):

        self.cpr_model = cpr_model

        self.b = b

        self.pc_data = pc_data

        self.pc_error = pc_error

        self.pc_weight = pc_weight

        self.counter = 0

    def __call__(self, params):

        """This function is called by the optimizer; calculate mismatch vector."""

        self.counter += 1

        # Build new capillary model from current parameters

        cpr_model = self.params_to_cprmodel(params)

        # Sw match

        swav_data = self.pc_data.swav.values

        Pc_data = self.pc_data.Pc.values

        swf = np.linspace(cpr_model.swi, cpr_model.swini, 100)

        Pc = cpr_model.calc(swf)

        swav_sim = cpr_model.calc_av(swf, self.b, Pc)

        # Interpolate swav_simulation to measured Pc

        swav_sim_inter = np.zeros(len(Pc_data))

        g = interp1d(

            Pc,

            swav_sim,

            kind="linear",

            bounds_error=False,

            fill_value=(swav_sim[-1], swav_sim[0]),

        )

        for i in range(len(Pc_data)):

            swav_sim_inter[i] = g(Pc_data[i])

        swav_mismatch = (swav_sim_inter - swav_data) * self.pc_weight / self.pc_error

        mismatch = swav_mismatch.flatten()

        return mismatch

    def params_to_cprmodel(self, params):

        v = params.valuesdict()

        swini = v["swini"]

        swi = v["swi"]

        L = v["L"]

        E = v["E"]

        T = v["T"]

        Pc_max = v["Pc_max"]

        Pt = v["Pt"]

        # We take Sorw for Pc to the right of the relperm Sorw

        cpr_model = LET_DR(swini, swi, L, E, T, Pc_max, Pt)

        return cpr_model

And my line that returns the error is:

#Excute fit

result_pcc = Minimizer(

    pcc_matchobj, params_pcc, calc_covar=True, nan_policy="omit"

).emcee(

    steps=10000,

    nwalkers=70,

    burn=100,

    workers=10,

    run_mcmc_kwargs={"skip_initial_state_check": True},

)

[Matt Newville]

Since you changed versions of both lmfit and emcee, it's not clear where the problem is. Can you install emcee 3.1.2 with lmfit 1.3.2 and see if that works?

We always ask for a minimal example that shows a problem, and a full traceback.   You included incomplete code that we really cannot 

decipher.  We really cannot tell what you are doing or what error message you might have gotten.

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/20f32b83-567f-41a7-9117-0fa42476175cn%40googlegroups.com.

--

--Matt Newville <newville at cars.uchicago.edu> 630-327-7411

[Felipe Eler]

Matt, thanks for your feedback.

Sorry for the poor code and the absence of an error message.

The error listed is this:

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[8], line 2 1 #Excute fit ----> 2 result_pcc = Minimizer(pcc_matchobj, params_pcc, calc_covar = True, nan_policy = 'omit').emcee(steps = 10000, nwalkers = 70, burn = 100, workers = 24, run_mcmc_kwargs={'skip_initial_state_check':True}) 3 # result_pcc = Minimizer(pcc_matchobj, params_pcc, calc_covar = True, nan_policy = 'omit').least_squares(diff_step=1e-4,verbose=2) 4 5 # write error report 6 print('\n\n\n') File c:\Users\felip\anaconda3\envs\curvefit\Lib\site-packages\lmfit\minimizer.py:1405, in Minimizer.emcee(self, params, steps, nwalkers, burn, thin, ntemps, pos, reuse_sampler, workers, float_behavior, is_weighted, seed, progress, run_mcmc_kwargs) 1403 # now do a production run, sampling all the time 1404 try: -> 1405 output = self.sampler.run_mcmc(p0, steps, progress=progress, **run_mcmc_kwargs) 1406 self._lastpos = output.coords 1407 except AbortFitException: File c:\Users\felip\anaconda3\envs\curvefit\Lib\site-packages\emcee\ensemble.py:450, in EnsembleSampler.run_mcmc(self, initial_state, nsteps, **kwargs) 447 initial_state = self._previous_state 449 results = None --> 450 for results in self.sample(initial_state, iterations=nsteps, **kwargs): 451 pass 453 # Store so that the ``initial_state=None`` case will work File c:\Users\felip\anaconda3\envs\curvefit\Lib\site-packages\emcee\ensemble.py:351, in EnsembleSampler.sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress, progress_kwargs) 349 state.blobs = blobs0 350 if state.log_prob is None: --> 351 state.log_prob, state.blobs = self.compute_log_prob(state.coords) 352 if np.shape(state.log_prob) != (self.nwalkers,): 353 raise ValueError("incompatible input dimensions") File c:\Users\felip\anaconda3\envs\curvefit\Lib\site-packages\emcee\ensemble.py:496, in EnsembleSampler.compute_log_prob(self, coords) 494 else: 495 map_func = map --> 496 results = list(map_func(self.log_prob_fn, p)) 498 try: 499 # perhaps log_prob_fn returns blobs? 500 (...) 504 # l is a length-1 array, np.array([1.234]). In that case blob 505 # will become an empty list. 506 blob = [l[1:] for l in results if len(l) > 1] File c:\Users\felip\anaconda3\envs\curvefit\Lib\multiprocessing\pool.py:367, in Pool.map(self, func, iterable, chunksize) 362 def map(self, func, iterable, chunksize=None): 363 ''' 364 Apply `func` to each element in `iterable`, collecting the results 365 in a list that is returned. 366 ''' --> 367 return self._map_async(func, iterable, mapstar, chunksize).get() File c:\Users\felip\anaconda3\envs\curvefit\Lib\multiprocessing\pool.py:774, in ApplyResult.get(self, timeout) 772 return self._value 773 else: --> 774 raise self._value File c:\Users\felip\anaconda3\envs\curvefit\Lib\multiprocessing\pool.py:540, in Pool._handle_tasks(taskqueue, put, outqueue, pool, cache) 538 break 539 try: --> 540 put(task) 541 except Exception as e: 542 job, idx = task[:2] File c:\Users\felip\anaconda3\envs\curvefit\Lib\multiprocessing\connection.py:206, in _ConnectionBase.send(self, obj) 204 self._check_closed() 205 self._check_writable() --> 206 self._send_bytes(_ForkingPickler.dumps(obj)) File c:\Users\felip\anaconda3\envs\curvefit\Lib\multiprocessing\reduction.py:51, in ForkingPickler.dumps(cls, obj, protocol) 48 @classmethod 49 def dumps(cls, obj, protocol=None): 50 buf = io.BytesIO() ---> 51 cls(buf, protocol).dump(obj) 52 return buf.getbuffer() TypeError: cannot pickle 'LET_DR' object

I will make the test as you suggested, and if it works, I will post it here.

Thank you.