New Feature requests - Fits

[Patrick Kearney]

David, thanks again for IMD!  I had several features that I was hoping for in the next release and many of them are here! 

One thing I'd like that I can't figure out a way to accomplish:

1. Doing fits where both the incident angle and wavelength change.  For example, I often get reflectivity data vs. wavelength at several incident angles.  I can fit one angle (or wavelength) easily and the software finds a structure that fits the data at that angle easily.  But when I change to another angle, the fit is off, and I end up jumping between the two angles and solutions.  The simplest way (for me) to get out of this situation would be to load the reflectivity vs. wavelength and angle.  I see some things like this in IMD*Multiplot, but I don't think I can fit from there.  I'd be willing to be restricted to regularly spaced points in both angle and wavelength, in fact I could probably live with just 2 angles if I could pick them judiciously.    Am I missing a way to do this in the current software?  Is there a better way to set up the problem that I'm not thinking of? 

Thanks again

Pat

 

 

[David Windt]

Hi Pat,

Thanks for the feedback. The feature you have in mind is a good one indeed - other people have expressed an interest in something like that too. Another application would be to fit the same structure using two types of measurements, like XRR + neutron reflectance, or perhaps XRR + EUV reflectance.

I'm imagining for a future release something called IMD*Multifit, where you'd load in two or more data sets but use a common structure and common fit parameters to fit everything simultaneously. It's already at the top of my To-do list, but I can't promise when it will be ready to go.

Anyway, thanks again for the suggestion.

I'm curious: have you - or anyone else - tried the genetic fitting algorithms in IMD? If so, are they working as you might have hoped? I know there are other programs out there for fitting, e.g., XRR data, but I've never actually used any of them, so I have no idea how IMD compares. I hope you'll let me know if there are any deficiencies in IMD, or any obvious niceties in other fitting programs that I haven't thought of...

David

[Robbert van de Kruijs]

Hi, David,

 

I'm happy to hear you're interested in genetic algoriths, and per you request I'll try to give you some initial feedback here, on what I did and how it compares with other software.

 

First off, I'm comparing IMD 5.01 with GenX 2.0b.

My system: Si sub / Ru / RuOx / RuOx

Aim: extract thickness of Ru and both RuOx layers, and some hint about densities and roughnesses would also be nice :)

Problem: Ru thickness and RuO2 thicknesses are largely "exchangeable", i.e. classic gradient method can easily go in wrong direction... Basically, 12 parameters, several strongly correlated...

Solution: Genetic algoriths :)

 

I'm using same population size (~300 members, optimized from many GenX runs).

 

First thing I noticed was the slower calculation time of IMD per generation, factor 10 slower compared to GenX Disabled parallelism in GenX, with parallel computing, the difference is even larger!...
Both simulations on same "experimental data grid" ( and done without convolution for instrumental resolution for fair comparison)

(both fits use what I guess is the same chi^2 for both GenX and IMD, i..e  chi^2/errorbar^2
Note that the new IMD is actually much faster than the old one for specific cases (multilayer GIXR simulations 10.000 data points), but apparently things could be further optimized? Is IDL using multicore? (my PC runs at 50% CPU load, suggesting it does not :)

 

The final solution was a bit "worse" fit than with GenX, but I guess I still have to play with all the settings to see if I can get closer.

Thinknesses and densities were similar though, especially no "non-physical" results, where parameter limits would certainly allow them, so that's goos news!
To be honest, I also do not understand all the algorith settigns of GenX, so I cannot compare exactly the same situations in terms of fit stability, time, etc...
Maybe the IMD fit was also simply not finished yet, GenX was "good" within an hour, IMD "crashed" after 10 hours, fit looked decent by that time, but stopped due to some weird error...

 

One point: In GenX you have the option to not use the initial guess as part of the first polulation, it's unclear to me if that initial guess is part of your first population or not, maybe I should read the manual :)
But would be good to have the option to include or exclude it (you can give a guess or start really from random starting point.)

For reference: I did not use the initial guess as part of the population in GenX.

 

Now for some "frustrations" :)

 

When you abort a fit (I never reached convergence, stil playing...), IMD "pushes" the results of the fit parameters to the actual values in the main window, overwriting starting values! This really should be asked if this should be dome or if opriginal values should be kept! (GenX does ask this :)

 

We also (and I cannot stress this enough) *need* a fit residue!!!

Only then can you judge if the method chosen for fit quality (chi^2, logartihmic, errorbars, ...) actually focusses on the range where the information is, or e.g. only on the total reflection range or Bragg peaks.
Note that it may be interesting to add some more "fitness quality" methods, e.g. sin^4(theta) is also often used, since reflectivity generally scales with sin2^4

GenX has more of these options, although here I chose the one compatible with IMD.

(note that GenX also has no fit residue...)

 

Finally, I would really like the option to see the full final polulation, the best canditate is nice, but there may be others which are equally nice (and maybe even more physically relevant), that's the purpose of a genetic algorithm, to keep "different genes" as long as possible, up to and including in the final population! Then it's the scientists choice to discard or keep what he wants for futher analysis (e.g. continue with gradient method on several "best candidates" from final population)

 

All right, so far for some first feedback, summarized, things are looking interesting, still more comparisons to be done, but the speed is what's worrying me... Maybe I should just order a new PC and see if things improve :)

If you need more information, I can provide it, either here or through mail, including some screenshots.

I'll see if I can add some more stuff in the next week(s), it's a bit time consuming, I'll have to redo tens of fits from GenX to test general "consistency"...

 

P.S. love the live view of all the individuals, nicely chaotic :)

 

P.P.S. I second the suggestion of Patrick for combined hard and soft x-ray analysis! ("multi wavelength analysis")

We're about to publish a paper on this using some home-written (LevMar) fit, showing the benefit of combining the two, I'll add a link here when it's published, for those interested to read it :)

And just a funny added note: I would especially like combined x-ray and neutron analysis ... and a time machine to go back 20 years and redo my PhD analysis! :) :) :)

 

Cheers,

Robbert van de Kruijs

 

 

 

 

 

 

 

[David Windt]

Hi Robbert,

Please see below.

David

On Mon, Oct 28, 2013 at 9:18 AM, Robbert van de Kruijs wrote:

My system: Si sub / Ru / RuOx / RuOx

Aim: extract thickness of Ru and both RuOx layers, and some hint about densities and roughnesses would also be nice :)

Problem: Ru thickness and RuO2 thicknesses are largely "exchangeable", i.e. classic gradient method can easily go in wrong direction... Basically, 12 parameters, several strongly correlated...

Solution: Genetic algoriths :)

 

I'm using same population size (~300 members, optimized from many GenX runs).

Why so many members (300) with so few fit parameters (12)?  From what I've read on genetic algorithms, a rough rule-of-thumb is (number of members) = (a few) x (number of parameters). Do you not achieve good fits using ~10x fewer members?

 

First thing I noticed was the slower calculation time of IMD per generation, factor 10 slower compared to GenX Disabled parallelism in GenX, with parallel computing, the difference is even larger!...

Very sorry to hear this. But I'm confused about disabling parallelism in GenX - wouldn't that make GenX slower, and thus a smaller difference between GenX and IMD? In any case, I'd like to understand better where the slowness in IMD is coming from. Is it the time to compute reflectance for each individual? Any more details you can provide would be helpful.

Both simulations on same "experimental data grid" ( and done without convolution for instrumental resolution for fair comparison)

(both fits use what I guess is the same chi^2 for both GenX and IMD, i..e  chi^2/errorbar^2
Note that the new IMD is actually much faster than the old one for specific cases (multilayer GIXR simulations 10.000 data points), but apparently things could be further optimized? Is IDL using multicore? (my PC runs at 50% CPU load, suggesting it does not :)

IMD doesn't use multi-core capabilities at this point, unfortunately. Maybe in the future. (It's non-trivial!)

 

 

The final solution was a bit "worse" fit than with GenX, but I guess I still have to play with all the settings to see if I can get closer.

Thinknesses and densities were similar though, especially no "non-physical" results, where parameter limits would certainly allow them, so that's goos news!
To be honest, I also do not understand all the algorith settigns of GenX, so I cannot compare exactly the same situations in terms of fit stability, time, etc...
Maybe the IMD fit was also simply not finished yet, GenX was "good" within an hour, IMD "crashed" after 10 hours, fit looked decent by that time, but stopped due to some weird error...

IMD includes two different genetic algorithms, including the Differential Evolution (DE) algorithm that I understood was used in GenX. The paper by Bjorck describing that algorithm reports that it can be much faster than other algorithms. But there will definitely be performance variations depending on the algorithm settings. I quoted Bjorck in IMD.pdf, and I'll repeat it here:

“Using DE in a black-box manner bears the risk of obtaining poor performance. To achieve optimal performance of the algorithm, its control parameters may need to be carefully tuned depending on the complexity and difficulty of the problem.”

So first, were you using IMD's GA or DE genetic algorithm?  I think it can make a big difference. Second, I would strongly suggest that you have a look at section 8.6 of IMD.pdf, and also the Bjorck paper cited therein, so you can try to tune the settings for best performance.

 

One point: In GenX you have the option to not use the initial guess as part of the first polulation, it's unclear to me if that initial guess is part of your first population or not, maybe I should read the manual :)
But would be good to have the option to include or exclude it (you can give a guess or start really from random starting point.)

For reference: I did not use the initial guess as part of the population in GenX.

The initial guess is indeed part of the first generation in IMD, and there is no option to exclude it. (Section 8.6.5 of IMD.pdf.) When I wrote the code, I couldn't imagine why one would want to exclude it ... but that may just be a failure of my imagination! If you can convince me why that might be advantageous, I will make it an option!

 

Now for some "frustrations" :)

 

When you abort a fit (I never reached convergence, stil playing...), IMD "pushes" the results of the fit parameters to the actual values in the main window, overwriting starting values! This really should be asked if this should be dome or if opriginal values should be kept! (GenX does ask this :)

Personally, I prefer to abort the fit manually in most cases, so the behavior you describe is not problematic for me. Also, I try to save a .imd file before I start the fit, after I have configured everything, so I can always revert to the initial configuration by opening that .imd file. But now that you mention it, having the option of NOT over-riding the actual values with the fit parameters after an abort is a good idea - I'll do it!  (Keep in mind, however, that the initial values of the fit parameters do not get overwritten at the end of the fit - those only change when you do so manually, or when you explicitly change parameter values in the main window.)

 

We also (and I cannot stress this enough) *need* a fit residue!!!

Only then can you judge if the method chosen for fit quality (chi^2, logartihmic, errorbars, ...) actually focusses on the range where the information is, or e.g. only on the total reflection range or Bragg peaks.

Great idea!  I'll do it.

Note that it may be interesting to add some more "fitness quality" methods, e.g. sin^4(theta) is also often used, since reflectivity generally scales with sin2^4

GenX has more of these options, although here I chose the one compatible with IMD.

(note that GenX also has no fit residue...)

The genetic algorithms in IMD use a figure of merit that allows you to set the exponent value arbitrarily - is that not functionally equivalent to what you're suggesting?

 

 

Finally, I would really like the option to see the full final polulation, the best canditate is nice, but there may be others which are equally nice (and maybe even more physically relevant), that's the purpose of a genetic algorithm, to keep "different genes" as long as possible, up to and including in the final population! Then it's the scientists choice to discard or keep what he wants for futher analysis (e.g. continue with gradient method on several "best candidates" from final population)

Interesting idea. But a bit trickier to implement, I think, at least in the current version of IMD. Anyway, let me think about it further...

 

 

All right, so far for some first feedback, summarized, things are looking interesting, still more comparisons to be done, but the speed is what's worrying me... Maybe I should just order a new PC and see if things improve :)

If you need more information, I can provide it, either here or through mail, including some screenshots.

I'll see if I can add some more stuff in the next week(s), it's a bit time consuming, I'll have to redo tens of fits from GenX to test general "consistency"...

 

P.S. love the live view of all the individuals, nicely chaotic :)

Nothing to do with IMD, but check this out! https://www.youtube.com/watch?v=kPRA0W1kECg&feature=youtube_gdata_player
 

 

P.P.S. I second the suggestion of Patrick for combined hard and soft x-ray analysis! ("multi wavelength analysis")

We're about to publish a paper on this using some home-written (LevMar) fit, showing the benefit of combining the two, I'll add a link here when it's published, for those interested to read it :)

And just a funny added note: I would especially like combined x-ray and neutron analysis ... and a time machine to go back 20 years and redo my PhD analysis! :) :) :)

Still working on the time machine option ... but if you had one, I sincerely hope that re-doing your PhD would not be the first item on your to-do list!

Thanks again for all your comments and feedback!! I really do appreciate it.  Please keep them coming!

[Robbert van de Kruijs]

Hi, David,

 

Great to see you responding already, I'll give a few answer below:

 

-Population size 280 is indeed overkill, I know about the factor between population and parameters, I just wanted to play it "safe". The factor you mentioned ("a few") was mentioned but that is valied for an "optimized" situation, i.e. exactly the right mutation and crossover rates for the problem considered. Also it depends on the "expected gene range" i.e. if the boundaries are far away from the solution, you need a lot of "genetic information" ("more bits") I also read that to play it safe a factor of 10 would always be good, and I even could afford to double that. (280 was 20 times 14 parameters) Especially due to the high correlation in the parameters, I did not want to leave out any "genes" from the start. In practice, half an hour fitting time was also "convenient", I think that was the real reason for the size of 280. We increased it ridicously for runs over the weekend, and of course nothing happened. Much less than hundred (I think below 50) started to give some problems in reproducibillity of some runs.

Having said that, it's still interesting to try less polulation size for both methods, and even your second genetic algorithm that I did not touch yet.

 

- the factor 10 is *without parallelism*, if I enable parallelism in GenX, I have a factor 20 difference!... (dual core system here :) Finding the difference (and solving it) would be great, but "the individual" is "instantly" calculated (tens to hunderds per second, population of 280 in 1 second for GenX, 10 seconds for IMD), so I don't know how to quantify this or maybe we should compare for "extended" number of data points (now 600 data points, make it 60.000? I remember old IMD had lots of problems extending to large numbers, maybe due to memory swapping or such, so that maybe risky...

I did notice "getting scattering factors for..." and "computing optical constants for .." for each calculation, maybe that's delaying things?... I might try fixed n,k, not "by density and factors" see if that speeds thing up, see if there's a delay there.

 

- So far all I reported was on general DA method, since that was best comparison with GenX, I definitely planned to try with Bjorck's algorithm, I did glance at the manual, hope to spend some more time on it in the near future.

 

- I really liked the option to exclude the first guess from the GA in GenX, maybe in the end you know your system very well, but for a "unknown" system it's best to start from scratch, i.e. large polulation size and no "best guess", otherwise you run the risk that other genes will not be allowed into the population because they penalize the FOM too much and you basically end op with an "expensive" gradient method around your first guess. If it's possible to eliminate it, why not provide the user with the option, like in GenX?

(note that indeed solutions are found faster with a best guess, but I would only do that in really time consuming fits)

 

- (skipping a few remarks on you implementing my suggestions: great to hear, thanks!)

You mentioned that the FOM can be tweaked by the power of the exponent, but does that not only tweak the "importance" of outlying points? low power does not penalize deviations from fit, while high power really adds a lot of "penalty" to bad points. What I'm talking about is a strategy on where your fit is most sensitive to, without logaritmic FOM you emphasize the high intensity parts, while different FOM functions may give different emphasis on different angle of incidence ranges. To be honest, so far ((R_meas-R-calc) / error)^2 seems to do fine for me, but your mileage may vary. GenX just has lots of added FOM functions, that's why I remarked upon it.

 

Well, these were just some answers to your questions, as mentioned I'll play with IMD some more, I really like the interface much better than GenX, just have to get some more experience with it and I'll hopefully add to this.

Maybe one final silly thing, but GenX can incorporate finite sample size effects, nice for including proper critical angle range in fit, maybe IMD has it, if not it should not be so difficult to add?

 

Cheers,

Robbert

 

 

[David Windt]

Hi Robbert,

On Monday, October 28, 2013 1:05:47 PM UTC-4, Robbert van de Kruijs wrote:

Hi, David,

 

Great to see you responding already, I'll give a few answer below:

 

-Population size 280 is indeed overkill, I know about the factor between population and parameters, I just wanted to play it "safe". The factor you mentioned ("a few") was mentioned but that is valied for an "optimized" situation, i.e. exactly the right mutation and crossover rates for the problem considered. Also it depends on the "expected gene range" i.e. if the boundaries are far away from the solution, you need a lot of "genetic information" ("more bits") I also read that to play it safe a factor of 10 would always be good, and I even could afford to double that. (280 was 20 times 14 parameters) Especially due to the high correlation in the parameters, I did not want to leave out any "genes" from the start. In practice, half an hour fitting time was also "convenient", I think that was the real reason for the size of 280. We increased it ridicously for runs over the weekend, and of course nothing happened. Much less than hundred (I think below 50) started to give some problems in reproducibillity of some runs.

Having said that, it's still interesting to try less polulation size for both methods, and even your second genetic algorithm that I did not touch yet.

 

- the factor 10 is *without parallelism*, if I enable parallelism in GenX, I have a factor 20 difference!... (dual core system here :) Finding the difference (and solving it) would be great, but "the individual" is "instantly" calculated (tens to hunderds per second, population of 280 in 1 second for GenX, 10 seconds for IMD), so I don't know how to quantify this or maybe we should compare for "extended" number of data points (now 600 data points, make it 60.000? I remember old IMD had lots of problems extending to large numbers, maybe due to memory swapping or such, so that maybe risky...

I did notice "getting scattering factors for..." and "computing optical constants for .." for each calculation, maybe that's delaying things?... I might try fixed n,k, not "by density and factors" see if that speeds thing up, see if there's a delay there.

 

Just before releasing IMD 5, I made a somewhat hasty (and not entirely thought-out) change (in order to fix a different problem) that results in IMD loading optical constants/scattering factors more frequently than it needs to in some cases. Perhaps that is the source of the slowness that you see?  I will try to come up with a better plan for the next version.

 

- So far all I reported was on general DA method, since that was best comparison with GenX, I definitely planned to try with Bjorck's algorithm, I did glance at the manual, hope to spend some more time on it in the near future.

 

I've not used GenX, so I don't know for sure, but I thought it used only the DE algorithm - or does it offer the user a choice? If it only uses the DE algorithm than a fair comparison with IMD would require using the DE algorithm in IMD as well.
 

- I really liked the option to exclude the first guess from the GA in GenX, maybe in the end you know your system very well, but for a "unknown" system it's best to start from scratch, i.e. large polulation size and no "best guess", otherwise you run the risk that other genes will not be allowed into the population because they penalize the FOM too much and you basically end op with an "expensive" gradient method around your first guess. If it's possible to eliminate it, why not provide the user with the option, like in GenX?

(note that indeed solutions are found faster with a best guess, but I would only do that in really time consuming fits)

 

Fair enough - I'll include it that option.

 

- (skipping a few remarks on you implementing my suggestions: great to hear, thanks!)

You mentioned that the FOM can be tweaked by the power of the exponent, but does that not only tweak the "importance" of outlying points? low power does not penalize deviations from fit, while high power really adds a lot of "penalty" to bad points. What I'm talking about is a strategy on where your fit is most sensitive to, without logaritmic FOM you emphasize the high intensity parts, while different FOM functions may give different emphasis on different angle of incidence ranges. To be honest, so far ((R_meas-R-calc) / error)^2 seems to do fine for me, but your mileage may vary. GenX just has lots of added FOM functions, that's why I remarked upon it.

 

Can you please provide more details about this? I'd like to better understand what you have in mind.
 

Well, these were just some answers to your questions, as mentioned I'll play with IMD some more, I really like the interface much better than GenX, just have to get some more experience with it and I'll hopefully add to this.

Maybe one final silly thing, but GenX can incorporate finite sample size effects, nice for including proper critical angle range in fit, maybe IMD has it, if not it should not be so difficult to add?

It is relatively easy to add, of course. Personally I've never seen the appeal, however: aren't the sample/beam size effects near 0 deg very sensitive on alignment? Isn't it safer to just limit the range of the measured data?

 

Cheers,

Robbert

 

 

[Robbert van de Kruijs]

Hi, David,

 

GenX should use the DE method, I guess your "DE (Bjorck)" method might have something to do with the author of GenX, Matts Björck :)

(just confirmed the use of DE at the GenX wiki :)

So I'll do my next tests with the DE of IMD to be fair.

It still does not solve the slowness of calculating a single population, when I have some time I'll do it without calculating optical constants and report back on the results to see if those "not entirely thought out" changes may contribute to delays.

 

About the diffrent merit functions, GenX has "R1, R1bars, R2bars,  chibars, chi2bars, diff, log, logR1, logR2, logbars, sintth4, sqrt" as choice.

I did not find information in the manual, and Chi2bars and logbars often seem to work well, but since also in GenX I cannot judge the merit "per data point" (no fit residue :) it's hard to say which function effectively does what. Maybe you could drop Matts an email, but I guess there should be tons of literature out there on the value of merit funtions. To be honest, chi2bars (or different power) might be what everyone needs, but why did Matts include so many other options then?...

 

Indeed sample size is an issue for alignement, but critical for getting a good fit of the critical angle, and therefore everage density. limiting the fit to higher angles kills the accurate determination of average density.

And regarding alignement errors, I'm using an Empyrean system with a 2D detector, and it helps tremendously to look at the shape of the reflected peak near the small angles to at least identify if you're suffering from alignement.

Especially for thick samples there is no alignement error from strain induced substrate curvature, and sample alignement on higher order Bragg peaks allows a proper alignement of theta vs. two theta, which *should* be also perfect al small angles as long as your surface is really flat (which for thick substrates it is) In those cases you can (and should!) properly include the fit of the critical edge (and even part of the total reflection range for that determines absorption coefficient)

Of course we can "reconstruct" the proper experimental data by correcting the data for geometry, but that should be done including instrumental angular resolution, much nicer if that's simply included in the simulations (like in GenX :)

 

Cheers,

Robbert

 

 

[David Windt]

Hi Robbert,

Thanks for the additional info.  I'll include finite sample size in the next release, and will look into additional merit functions as well.

David

[Robbert van de Kruijs]

Hi, David,

Small update:

Just simply removed the fit paramters of density in my model (while keeping the model parameters based on elements and density, so it should still calculate it at least once :) and IMD *flies*!

Now the difference between IMD and Genx is maybe 10% (sadly still GenX wins out the contest) but that we can live with.

So I strongly support you fixing this for the next release :)

On another note, after various "unable to create pixmap" errors I finally created an .IMD file that "always" crashed! (maybe I should not be so happy? :)

It always crashed immediately after generation 288 ... at my work PC, but not at my home PC!?

I even closed all running programs at work, still crashed at the same generation.

At home it did not crash at least until generation 1000, when the algorith stopped (max generations reached)

I'll see if I can find situations that it does *not* crash at work after 288 generations, maybe that'll help us further...

Cheers,

Robbert

 

[David Windt]

Hi Robbert,

That's good news about the speed-up - I'll try to track down the problem and fix it for the next release.  Let me know if you're able to get a reliably-faulty .imd file so that I can track the pixmap problem as well.

David

[Serhiy Danylyuk]

Hi David,

thanks for the excellent software, using it routinely for a few years already. Wanted to ask if there is any progress on IMD*Multifit idea, as this is one thing we are really missing. Most of our data are R(wavelength, angle) datasets and it takes a lot of time to fit each angle separately and find satisfactory common solution.

Another thing that would also be extremely useful is a possibility to load a set of R(wavelength) curves for different incidence angles and directly calculate  n,k curves for all wavelengths. Currently IMD can do that only for single R(angle) dependence, so hundreds wavelength points per measurement we typically have would demand a lot of patience and time from us.

Thanks again for the great work,
Serhiy