Fitting multiple data sets using the Model interface

[Tarek H.]

Hi all,

I recently started using the lmfit framework, and I actually love it. So thank you for the nice package!

I saw in the "Fit Multiple Data Sets" example the following line:

TODO: this should be using the Model interface / built-in models!

And that's exactly what I would like! Is there any example where the combined fit is performed using the Model class?

I would be happy to provide the example myself, although given that I am rather new with lmfit, I would like some guidance in the process.

Thank you in advance!

Tarek

[Renee Otten]

hi Tarek, 

the example you mentioned works perfectly fine, but I wrote that comment because using the Model interface and built-in models would

likely make life easier for the end-user. 

I guess you want to do something like what is shown below. In the example are only two peaks, but of course that can be easily extended.

Take a look, give it a try with your own data and if you get stuck please post the code you have and we might be able to help. 

Good luck!

Renee

import matplotlib.pyplot as plt

import numpy as np

from lmfit.lineshapes import gaussian

from lmfit.models import GaussianModel

# generate synthetic data with two Gaussian lineshapes

x = np.linspace(-1, 2, 151)

noise = np.random.normal(size=x.size, scale=0.1)

data = gaussian(x, 1.0, -0.5, 0.25) +  gaussian(x, 2.5, 1.25, 0.25) + noise

# make a CompositeModel, see:

# https://lmfit.github.io/lmfit-py/model.html#lmfit.model.CompositeModel

model = GaussianModel(prefix='g1_') + GaussianModel(prefix='g2_')

# make the parameters

pars = model.make_params()

# if needed you can set initial values (and other attributes) for the

# parameters, as in:

# pars['g1_center'] = -0.4

# in this case the "default" parameter values are sufficient to have the

# fit converge

result = model.fit(data, pars, x=x)

result.plot()

# or when you want to show the individual components of the model as well

plt.figure()

comps = result.eval_components(x=x)

plt.plot(x, data, 'o', label='data')

plt.plot(x, comps['g1_'], '--', label='Gaussian 1')

plt.plot(x, comps['g2_'], '--', label='Gaussian 2')

plt.legend()

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/6c3f9e84-94fa-4f90-aa6e-cf057f8ed81eo%40googlegroups.com.

[Tarek H.]

Hi Renee,

Thank you for the extremely prompt reply!

Yes, the example does indeed work, and I'm trying to use it to implement the fit I want. But, as you said, an implementation using the Model interface would certainly be way more "elegant", and easier to implement.

I believe the example you attached does not use the same approach as the one in the example I referred to: in this example a single dataset (2 gaussians + noise) is fitted with a single composite model.

The application I need is fitting several datasets with different models (perhaps the same one, several independent times), allowing some of the fitted parameters to be free for each dataset while sharing others between datasets (as the common sigma of the different gaussians of your original example).

Is there any way to associate datasets to models and perform a single common fit? Or should I just use the minimize function as in the example?

Best,

Tarek

You received this message because you are subscribed to a topic in the Google Groups "lmfit-py" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lmfit-py/cKsQLLGkZ9c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/31DB31A4-45B7-4006-A6CF-F6D06D86808E%40macports.org.

[Matt Newville]

Hi Tarek, Renee,

I think what would be really helpful for fitting multiple datasets would be to create a different kind of Model subclass.  CompositeModel is a subclass of Model that is really pretty simple, comprised of LeftModel, RightModel, and operator.  I think we could do something similar for fitting multiple datasets.

For multiple datasets, what we probably want is a "list of models" perhaps called MultiDataModel (hopefully you can come up with a better name).   This would hold of list of models (which could be a CompositeModel, of course), and a list of datasets.   In this conception, they would have to match in order: one could imagine using a dictionary instead of a list so that each dataset had a unique name and a corresponding model and data array (and the weights array!).   I think there is no getting around the fact that the user will sort of have to keep the correspondence between dataset and the model for it, but maybe we can help facilitate that organization.  One could imagine a "DataSet" class that had a name and contained a data array, a weights array, and a Model, with methods for 'eval' and 'calculate_residual'.  With that, MultiDataModel would just be little more than a list of DataSets.

Anyway, with a /sequence/ of models and datasets, the MultiDataModel would have to override 'eval' to evaluate each model and concatenate them and also override constructing the residual to evaluate the individual models and calculate the individual residual and then concatenate those.  Again, those would be very simple with DataSet.eval and DataSet.calculate_residual.

Does that seem like a reasonable approach to you? 

--Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CAJHetVpvU%3DvQvSDBj9Bq8wjaGRh7JJGXpp7WeP%2BMkfikWD0%2B3Q%40mail.gmail.com.

[Tarek H.]

Hi Matt,

I'm afraid I won't be able to help much with the implementation, but at least it looks like what you are proposing sounds exactly like what I (and probably others like me) would need.

Note I already made the combined fit work with the approach described in the example (using "minimize"). But properly implementing such a subclass would probably make sense.

Best,

Tarek

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2B7ESbowQ-3KSkVqXO4vxLSRmKQfE9sCffBDBbG4xSSRgiMOLQ%40mail.gmail.com.

[Kévin POUNOT]

Hi everyone,

As Tarek, I would like first to thank the authors for this great package!

I am also looking at an easy way to fit 2D or 3D data from quasi-elastic neutron scattering with parameters that are global in two ('x'/energy transfer and 'q'/momentum transfer) and others that are independent for each 'q'.

I was also thinking of a subclass that inherits from Model or CompositeModel. We could probably prepend a prefix/model name to the model prefixes and have an argument to tell which parameters are global, and which are not.

Such that the program would automatically set 'expr' for the global parameters to constrain them to the one in the first model of the list/dictionary.

I don't know if there is some work ongoing already on that, but I am willing to help or start to work on it.

Best,

Kevin

[Mark Dean]

Dear Kevin,

lmfit can deal with 2D or 3D functions quite easily. There is an example here

https://github.com/mpmdean/lmfit-py/blob/master/examples/example_two_dimensional_peak.py

This will be rendered on the examples webpage of lmfit at the next release of lmfit.

Essentially you just need to tell lmfit that you have more than one independent variable.

The function itself can have free parameters that depend on either just one or multiple independent variables.

Best,

Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/bd8a3a12-ba7e-4b9c-a258-e1b111407bc3n%40googlegroups.com.

--

--------------------------------------------

Mark P. M. Dean

Brookhaven National Laboratory

Condensed Matter Physics Bldg 734

Upton NY 11973

USA

Phone: 001 631 344 7847

Email: md...@bnl.gov

[Kevin Pounot]

Dear Mark,

Thanks for the link.

What I am actually looking for is to fit also multiple parameters on the second axis, which can be of variable size.

Let's say I select 5 values in 'q' dimension out of 10, the goal would to avoid to have to write - for each possible size of q - something like:

    myFunc(x, q, a_1, a_2, a_3, a_4, a_5, center_1, center_2, center_3, center_4, center_5, width):

        width = width * q**2

        out = np.zeros_like(q)

        out[0] = a_1 * width / (np.pi * (x**2 + width**2))

        ...

        return out

Another possibility would be to generate a string, format it and use `exec` to define in a dynamic manner the function to be used with the model but it doesn't look like something really safe to do, or am I wrong?

Best,

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CABMFFfqGm_0Q2eoNquneg36UtE8bjH%2B9%3D-D9PPjPeuixeAgv%2BQ%40mail.gmail.com.

[Mark Dean]

I'm having a hard time understanding what precisely you want to do, but it sounds like you do something like this

https://lmfit.github.io/lmfit-py/examples/example_fit_multi_datasets.html#sphx-glr-examples-example-fit-multi-datasets-py

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/SV0P279MB01084F18F4517D54BD1264BFCE300%40SV0P279MB0108.NORP279.PROD.OUTLOOK.COM.

[Kevin Pounot]

Yes, indeed, I am looking to do something like this, but using the Model interface.

Best,

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CABMFFfpdawxqKSHEzOCrL5AWAuB6aC4SVNeD0%3DO_ZqKgrTASZw%40mail.gmail.com.

[Kévin POUNOT]

Hi everyone,

If there are still some people interested in this, I have attached a python script that contains a function to build a 2D model (here adapted to neutron scattering).

And a example function to build a simple Gaussian that can fit a 2D dataset either row-wise or with a global width, which have an explicit dependence on the second dimension variable.

This is similar to the link given by Mark above, but with the Model interface, such that all the 'Model' class features are available and 'CompositeModel' can be built as well.

Best,

Kevin

[Kevin Pounot]

Here is an extended version of the python code, with some corrections.

Best,

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/834854bf-4859-4752-a1d0-bd56295e090en%40googlegroups.com.

[Kévin POUNOT]

Hi everyone,

Sorry for the multiple messages, but there was a small indentation error in the previous file that I didn't spot before. This was making the fit result look weird.

Now this version works fine (at least on my data). There are several 'preset' models as well (if it can be of help).

The only incompatibility with lmfit I can see now is for plotting. I get an error stating that the plots are not available for models with two independent variables.

Best wishes,

Kevin