Updated Jupyter notebook UI for LMFIT

[hold...@uw.edu]

After a recent post pointing out the existence of the (mostly not-working) lmfit.ui subpackage with some boilerplate code for a Jupyter UI, I decided to give it a go try to make something useful and add some desired functionality.  I had to scrap most of the the existing subpackage that was too abstracted away to understand and make usable, but I was able to work off of the existing lmfit.ui.ipy_fitter.ParameterWidgetGroup which was quite helpful.

A demonstration jupyter notebook is attached.  Also see the attached gif for a demonstration of it's functionality.

Anybody who is interested, please give it a try and let me know how it goes.

The good parts:

• Formatting is updated so the parameter widgets look good in their layouts and can all be displayed on-screen without scrolling
  
• Using the external library (https://github.com/matplotlib/jupyter-matplotlib) allowed for smooth incorporation of in-place update-able plots (with `%matplotlib widget` jupyter magic command)
• Support for making composite models!  Additional models of the same type can also be added.  The new parameter widgets are created automatically
• It works! Running the fit updates the plot and gives the updated parameters in the parameters widgets.  The fit output can be accessed as the "fit" attribute of the "FitUI" class.
  

The need-some-work parts:

• Haven't tested all the models, but most seem to be usable.  A couple I've found are giving errors (namely ConstantModel and asteval warnings from DampedHarmonicOscillatorModel).
• Requires use of the external library (https://github.com/matplotlib/jupyter-matplotlib).  Maybe the additional dependency isn't a concern.
• The ability to modify the parameters 'expr' has not been added.  This could be somewhat cumbersome handling for error parsing and I don't have experience with asteval.
• Has not been extensively tested yet.  Surely bugs will need to be sorted out.
• I wanted to add automatic updating of the widgets displays of the parameters if the parameters were modified externally or manually in another cell.  I attempted to use the traitlets package to monitor for changes, but the internal structure of lmfit.parameter.Parameter made this incompatible for now.
  

[hold...@uw.edu]

Bummer the .gif isn't rendering as a video.  It was playing automatically after uploading to the post form on the google group webpage. Sigh. Here's a better static screenshot.

[hold...@uw.edu]

After a bit more playing around, fixed a few of the more obvious bugs, improved the parameter naming, and added the individual plot components as dashed lines on the plot.  A version 2 of the demonstration notebook is attached. 

I'm actually surprised, it's quite stable and usable.  I was able to do a more complicated 5-peak voigt fitting with relative ease.  After achieving the desired fit, I was unable to pickle the fit object itself, but I was able to pickle the parameters object.  Or the fit report could be used to record the results instead.

If there's enough interest to keep developing on this, I'm considering that perhaps the best thing to do would be to start my own repository to house the project.  The main lmfit project is quite stable and I think would be cluttered by trying to include the development of this interface, which is likely to take a lot of updates to get into a stable, reliable form.  Also, it could be nice to keep the extra dependencies separate from the core package (as it looks like the IPython stuff has already caused some headaches in the past).

Does anybody have thoughts on taking it in this direction, starting a separate lmfit_jupyter_ui repository?  Perhaps it's time to deprecate the ui subpackage of lmfit and remove those dependencies from the main project?

[Matt Newville]

Hi William,

On Sat, Feb 9, 2019 at 8:08 PM <hold...@uw.edu> wrote:

After a bit more playing around, fixed a few of the more obvious bugs, improved the parameter naming, and added the individual plot components as dashed lines on the plot.  A version 2 of the demonstration notebook is attached. 

I'm actually surprised, it's quite stable and usable.  I was able to do a more complicated 5-peak voigt fitting with relative ease.  After achieving the desired fit, I was unable to pickle the fit object itself, but I was able to pickle the parameters object.  Or the fit report could be used to record the results instead.

If there's enough interest to keep developing on this, I'm considering that perhaps the best thing to do would be to start my own repository to house the project.  The main lmfit project is quite stable and I think would be cluttered by trying to include the development of this interface, which is likely to take a lot of updates to get into a stable, reliable form.  Also, it could be nice to keep the extra dependencies separate from the core package (as it looks like the IPython stuff has already caused some headaches in the past).

Does anybody have thoughts on taking it in this direction, starting a separate lmfit_jupyter_ui repository?  Perhaps it's time to deprecate the ui subpackage of lmfit and remove those dependencies from the main project?

That's very nice!  I think it would be great to make this widely available, and I'm sure many people would use them.

I would have no problem with this part of lmfit, but having a separate package is also fine.   It does seem like the IPython/Jupyter world moves pretty fast and is willing to drop support for some componens, no doubt moving to newer, better code.   

FWIW, there is a possibly-related PR (https://github.com/lmfit/lmfit-py/pull/524) to better format the fit report for HTML notebooks.  It would probably be helpful to include that for your code too.   We're not at allo opposed to making changes to lmfit for Jupyter notebooks, even if it might be faster/easier to develop fitting widgets at a separate project and pace.   

I have some (wxPython) GUI components using lmfitm and I'm also unsure whether these should be included in lmfit or kept in separate projects.   The GUI app I use these in are for fitting X-ray absorption spectra, but it would probably be not too much work to make something that started to look like Fityk from these.

My sense is that scientific programmers seem prefer the Jupyter notebooks, whereas more casual users would prefer a standalone fitting app.

--Matt

[hold...@uw.edu]

Hi Matt,

Thanks for the reply!  I think I'll go the separate package route for now, and if something clean/stable comes out of it in the long run, we could merge it in.

Yes that PR html-formatting for the parameters looks very nice, I'll throw that in manually until it gets merged into lmfit/main.

Are the wxPython GUI components you mentioned part of the xraylarch project, or in another repository?

I hadn't heard of Fityk before.  I tried it out and the UI is very nice, although the zooming is a bit clunky.  They're drag and drop peak addition/manipulation is certainly impressive.  I didn't see an obvious way to set parameter constraints (like lmfit's "expr") or max/min values in the fitting.  Are those unique to lmfit?

William

[hold...@uw.edu]

Ah nevermind, I found the Fityk ability for expressions and bounds.  Although a little clunky perhaps having to do it through the CLI.