Saving multiple fit results

[Antonio Bulgheroni (toto)]

dear all, 

I have a tricky question for you. I'm doing some data analysis and for each group of data I make several fits (at least six). I would like to store them on disk, because I would need them in other analysis steps. 

For previous analyses, I was always using the dedicated save_modelresult and load_modelresult functions and I was thinking to use the same approach here, but I will soon end up with so many files to make their handling cumbersome.  At least 6 files for each group with thousand groups. 

Is there a possibility to save multiple fit results in the same file? Something similar to a HDF5 file? 

Another possibility would be to add all single fit results file to a zip archive. What do you think? 

Thanks for your help,

Cheers,

toto

[Laurence Lurio]

You could make a python array of results, and then save the python array using pickle

Larry 

--
You received this message because you are subscribed to the Google Groups "lmfit-py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/a4de6a33-ccee-4e2d-afd9-f88c0e8b8f34n%40googlegroups.com.

[Antonio Bulgheroni]

I was thinking about something like that, but not sure I could simply dump and load a model result via pickle. I was naive thinking that model_results could be saved only via the save method function because of some not standard action on the load mechanism. 

Then I would use a dictionary instead of an array, to have an easier way to access individual elements. 

Thanks :)

-----------------------------------------
Antonio Bulgheroni, PhD

You received this message because you are subscribed to a topic in the Google Groups "lmfit-py" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lmfit-py/nZeQRBMLvVo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lmfit-py+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CAK4HuLE2J3s%3D8Omh35Wz6MuZ4_4Dy0zDkGNR999qbbcXo%3DUgmQ%40mail.gmail.com.

[Laurence Lurio]

You probably can't pickle the whole result, but do you need everything?  You could probably save result.params of some other subset of result.  If you type dir(result) you can get a list of all the things saved to result, and then select the subset you want.  

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2BoFUCTTikpAhv0Tfo%2B_8hKKoZy2RCTv1ayQxzdHrkRaC9-Cuw%40mail.gmail.com.

[Matt Newville]

Hi Antonio, Laurence,

It is possible to save full Model and ModelResults (including all Parameter attributes, fit statistics, and arrays) with their `dump() or `dumps()` methods (and re-load with `load() or `loads()` methods).  These dumps are JSON-formatted blobs and not particularly nice to look at.  They will include serialized numpy arrays for best_fit, independent variables, etc.

But I agree that assessing a series of fits, it is probably more helpful to make a table (spreadsheet, csv file) with a few key fit statistics and parameter values and uncertainties for each fit.  

You could do both: dump each ModelResult to a file in a folder (or some kind of database), and also write a summary csv file.

For what it's worth, the dumps are json-formatted blobs and not particularly nice to look at.  They will include serialized numpy arrays for best_fit, etc.

I (and I think maybe Laurence too) use HDF5 for lots of large-ish experimental data sets - it's widely used at synchrotrons and similar facilities.  I would not recommend trying to put the JSON dump files into HDF5.  But, it might be interesting to think about using an HDF5 group as an alternative for JSON for this purpose. 

How to format multiple related fits for comparing results as been discussed a few times in the past.  We don't have tools to do that.  Perhaps it is time to think about doing this?

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CAK4HuLHr54LB70QkpfmMYg68XhCLYtR5QtpsbwSfhpQ0c4a%2BMQ%40mail.gmail.com.

--

--Matt Newville <newville at cars.uchicago.edu> 630-327-7411

[Antonio Bulgheroni]

Thank you all for the answers. 

I usually save the fit parameters and some fit metrics results in a database table for further analysis, but I like to have the full results object dumped to a file somewhere because I like the idea of reload it and plot it directly without having to do anything special. 

So far I have adopted my first solution, saving all fits in different files but with a good naming convention allowing the code to easily find what I am looking for. 

Thanks again,

toto

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2B7ESbodjtp%3D4iGeymESYwpwaEvW8Ck5%2B_bHf6gBzaddPfDQEw%40mail.gmail.com.

[Ray Osborn]

I think you can do what you are asking for in NeXpy (https://nexpy.github.io/nexpy/). This is an application for analyzing and viewing HDF5 data, which has a GUI interface to LMFIT. When you have completed a fit, you can click on a ’Save Fit’ button, which will then save the entire fit to a list of nested groups that can be saved to an HDF5 file and viewed within the embedded IPython shell. The data, fits, and individual components can then be plotted however you like. 

When you reload this file, it is possible to select the fit group (‘lmfit_example/entry/saved_fits’ in the image) and launch the Fit GUI again to add new functions or change fitting constraints, etc. The LMFIT interface is described in the NeXpy documentation (https://nexpy.github.io/nexpy/pythongui.html#fitting-nexus-data).

Although NeXpy is primarily aimed at NeXus files, it will open generic HDF5 files. If you don’t want to use NeXpy itself, I can point you to the code that implements this, and you are welcome to see if it can be adapted to your needs.

With regards,

Ray Osborn

-- 
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Lemont, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROs...@anl.gov

To view this discussion on the web visit https://groups.google.com/d/msgid/lmfit-py/CA%2BoFUCQVe0xnCYbdepnBUTKDBbyrE1XUhLPCrxo3-5L8PORQDA%40mail.gmail.com.