lmfit setup for multiple curve fits

[NovicePython]

I am hoping someone who had gotten lmfit to work inside spotfire as a data function can help me. I have a three column table, the first two columns have X,Y data and the third column has an index that defines each group of X,Y points that need to be fit to a user defined model (all groups belong to same model but might have different constraints on parameters). I don't know if lmfit has ability to take in a unique curve key and create fits and output parameters for each group in a table.

if not, do I have to setup a loop in ironpython script and then send X,Y data for each group to this data function and output parameters and collate them outside of lmfit? don't know if there is performance hit in this - we are talking about 100s to 1000s of curves in the data set. thanks

[Matt Newville]

Hi, 

I had not heard of "spotfire" before, and seeing you mention "ironpython" (which I did not know was still a viable Python interpreter), I would say that I have no idea if or how lmfit works with this environment. Can you verify that lmfit works with spotfire? If so, is that using ironpython?  Do you know which versions are being used?

Setting up a Python script or Jupyter Notebook to fit data from columns of a two-column file or spreadsheet seems like a perfectly normal thing to do, and repeating that for thousands of datasets seems normal too. I have no idea how that would be done within a framework or application like spotfire.

--Matt

On Tue, Jun 11, 2024 at 6:07 PM NovicePython <kmu.s...@gmail.com> wrote:

I am hoping someone who had gotten lmfit to work inside spotfire as a data function can help me. I have a three column table, the first two columns have X,Y data and the third column has an index that defines each group of X,Y points that need to be fit to a user defined model (all groups belong to same model but might have different constraints on parameters). I don't know if lmfit has ability to take in a unique curve key and create fits and output parameters for each group in a table.

if not, do I have to setup a loop in ironpython script and then send X,Y data for each group to this data function and output parameters and collate them outside of lmfit? don't know if there is performance hit in this - we are talking about 100s to 1000s of curves in the data set. thanks

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/3f6afae1-4fd4-4334-9618-6777e1d8444fn%40googlegroups.com.

[Jeremy DeJournett]

Without knowing anything about your application, I would add that usually model fitting is done on a fairly beefy machine, and the fit parameters are used many times to simulate a given system. With that in mind, have you considered using a simple form of IPC, like a socket, to receive model parameters from the LAN, which could be computed by a more well tested Python interpreter running in a process on the same or another machine. All of this is of course lacking the full context of why you want to embed the fitter into your Iron Python application.

I could be wrong about how people use this library, so I welcome other options if there are any.

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2B7ESboeFpjKFpEFU5Rca_PxOzvf%2B4Ti4PqxXcXPdr64XaxPOA%40mail.gmail.com.

[Matthew Newville]

Most of the people using the downstream applications that I support based on lmfit run those apps on commodity laptops or workstations, the kind a professional scientist or graduate student at a university in North America or Western Europe would expect to have.   

I (and a few others) do sometimes set up workflows or jobs to fit hundreds to thousands of spectra in batch on "advanced workstations" or small clusters.  I believe that some people do run workflows with lmfit on high-performance computing clusters.   To be clear: a fit with lmfit is basically a single-CPU process, and using multiple cores is really only useful for doing separate fits with parallel processes.

 

--Matt

 

 

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CALURN9o67xc5kKqUWC%3D5wkjPEk-fcQfYC7G432N5xrXm21sMLg%40mail.gmail.com.

[NovicePython]

Let's take spotfire and ironpython out of discussion - sorry for complicating the discussion with where I was going to implement the code in. If I just have a three column data where the third column was the curve grouping index, I am looking for some simple example code which will loop through each index value and fit the data to a user defined model and output parameters and stats to a table - this last part I am unsure of how to set up - thanks

[Matt Newville]

On Sat, Jun 15, 2024 at 1:01 PM NovicePython <kmu.s...@gmail.com> wrote:

Let's take spotfire and ironpython out of discussion - sorry for complicating the discussion with where I was going to implement the code in.

OK... knowing whether lmfit (and, say, which version) would work in that environment seems important.  You could have just said, "Yes, lmfit does work with spotfire".... or not.  Now you are sort of indicating that you would rather change the subject than answer direct questions. 

If I just have a three column data where the third column was the curve grouping index,

What is a "curve grouping index"?  Is this similar to "a unique curve key" in your earlier message? I don't know what these mean.

 

I am looking for some simple example code which will loop through each index value and fit the data to a user defined model and output parameters and stats to a table - this last part I am unsure of how to set up - thanks

What are you trying to loop over: some index pointing to some dataset?  

In Python, it is easy to loop over a list or dictionary of items.  These might contain (x, y)  data, names of files holding those values, or something else.  For a homework assignment, take one of the examples, say https://lmfit.github.io/lmfit-py/examples/example_use_pandas.html#sphx-glr-examples-example-use-pandas-py, and alter that to loop over a list of CSV files.

By "last part", do you mean "output parameters and stats to a table"?   The fit result contains all the parameters and statistics.  These are described in the documentation. If running that example above in an interactive Python or Jupyter session, try `dir(result)`.  Putting those into some structured format should be straightforward, but does depend on what sort of table (Excel, HTML, SQL database) you mean.

As you may have seen from other responses in this and similar forums, we strongly recommend including a minimal working example of what you are doing and what is not working for you.   

--Matt

On Wednesday, June 12, 2024 at 11:34:50 PM UTC-5 jcdejo...@gmail.com wrote:

Without knowing anything about your application, I would add that usually model fitting is done on a fairly beefy machine, and the fit parameters are used many times to simulate a given system. With that in mind, have you considered using a simple form of IPC, like a socket, to receive model parameters from the LAN, which could be computed by a more well tested Python interpreter running in a process on the same or another machine. All of this is of course lacking the full context of why you want to embed the fitter into your Iron Python application.

I could be wrong about how people use this library, so I welcome other options if there are any.

On Wed, Jun 12, 2024, 12:25 Matt Newville <newv...@cars.uchicago.edu> wrote:

Hi, 

I had not heard of "spotfire" before, and seeing you mention "ironpython" (which I did not know was still a viable Python interpreter), I would say that I have no idea if or how lmfit works with this environment. Can you verify that lmfit works with spotfire? If so, is that using ironpython?  Do you know which versions are being used?

Setting up a Python script or Jupyter Notebook to fit data from columns of a two-column file or spreadsheet seems like a perfectly normal thing to do, and repeating that for thousands of datasets seems normal too. I have no idea how that would be done within a framework or application like spotfire.

--Matt

On Tue, Jun 11, 2024 at 6:07 PM NovicePython <kmu.s...@gmail.com> wrote:

I am hoping someone who had gotten lmfit to work inside spotfire as a data function can help me. I have a three column table, the first two columns have X,Y data and the third column has an index that defines each group of X,Y points that need to be fit to a user defined model (all groups belong to same model but might have different constraints on parameters). I don't know if lmfit has ability to take in a unique curve key and create fits and output parameters for each group in a table.

if not, do I have to setup a loop in ironpython script and then send X,Y data for each group to this data function and output parameters and collate them outside of lmfit? don't know if there is performance hit in this - we are talking about 100s to 1000s of curves in the data set. thanks

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/3f6afae1-4fd4-4334-9618-6777e1d8444fn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2B7ESboeFpjKFpEFU5Rca_PxOzvf%2B4Ti4PqxXcXPdr64XaxPOA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/60a77cd8-d8e3-4086-821c-9b5b4a0d3815n%40googlegroups.com.