Offset, sigma, and an explanation of the columns in the results table

[Kyle Douglass]

Hello everyone,
I was wondering what exactly the offset and sigma columns represent? How is sigma different from uncertainty?

The reason I ask is that I often use other localization software to create the CSV files and import them into ThunderSTORM. To do this, I have to set the proper names and units for each column. Unfortunately, I encounter errors when using some post-processing analyses, such as merging, when I do not correctly name a column.

I would suggest adding an explanation of the standard columns to the documentation in the GitHub wiki since questions like these seem to come up often. It would also aid those who use ThunderSTORM's post-processing tools but not the localization fitting.

Thanks! I really like ThunderSTORM so far.

Best,
Kyle

[Christophe Leterrier]

Hi Kyle (nice to see you outside of Twitter :)

I am extensively using ThunderSTORM as a batch renderer, using localization files obtained from the Nikon N-STORM software mostly. Actually for reconstruction, the sigma and offset columns are not used (I don't think they are in any processing steps like merging, drift correction etc as long as you import the number of photons from your input file). I think sigma is the standard deviation of the individual psf in nanometers, i.e. there is the approximative relationship uncertainty (sd) ~ sigma / sqrt (Nphotons). For the offset I'm not sure, I'm using the background value in the N-STORM localization file, but maybe I'm confusing it with the camera offset (100 ADU for an Andor camera). In any case these are not used if you directly get the number of photons (the intensity [photons] column in the ThunderSTORM file) from your input file.

You can look at the ImageJ script code I'm using to translate file to ThunderSTORM format here, and maybe adapt it to your needs:

https://github.com/cleterrier/ChriSTORM/blob/master/F-NStxtTranslate.js

By the way I'd be happy to have some input from Martin about the background/offset columns, just for the sake of clarity

Christophe

[Martin Ovesný]

Hi Kyle,

Christophe has already provided a good explanation. Sigma is indeed the standard deviation of the Gaussian model used for fitting. It is related to FWHM ~ 2.35 * sigma. I like your suggestion to include the description of columns in our wiki, because it seems to confuse people.

Below is a very poor attempt of a quick drawing :-) Green line represents real data. Red line is a residuum after subtraction of fitted model from the green line. Camera baseline is not shown in the results table, because it's same for all molecules. It is a number user sets in camera settings dialog.

It would be helpful for resolving the issue you reported at Github to attach the zipped file with localizations that causes the problem. If it's possible, do it directly at Github so it's tracked with the issue. Otherwise you can attach it to your message here. If nothing works, you can send it to my gmail.

By the way, Christophe is right that most of the columns are not used. Only x and y are required for merging. However, after the molecules are merged, a "parent" molecule is created and new values of all parameters are recalculated. This is very likely the issue. I'll know more when I see the data.

Best,

Martin

[Kyle Douglass]

Hi Christophe and Martin,
Thanks for the detailed responses! This does clear up some basic misconceptions I had about what the columns actually represent. The user interface to ThunderSTORM really is quite good, which is why I am working so hard to use it.

Christophe -- Thanks for the link to your .js file. I'm actually doing something similar already ;)

Martin -- The picture is perfect and helps a lot. I will attach the localization file to the GitHub issue by the end of today. The explanation of the columns is the number one thing I and others in my lab are missing right now :)

Kyle

[Kyle Douglass]

GitHub will not allow me to attach .zip files to issue tracker posts, so I'm attaching the example data set here that leads to the error. Please note that most operations seem to work; only merging produces the error.

Also, the sigma and bkgstd columns are mislabeled (which is why I made this post :) Sigma should actually be uncertainty and bkgstd should be offset.

Shouldn't merging also require the frame column, and not just x and y?

[Martin Ovesný]

Yes, of course, frame is also needed. I forgot to mention that.

Thanks for the data. I'll solve it this weekend. The problem wasn't in what I thought it would be. I'll have to take a closer look.

[Martin Ovesný]

Try to update now. I hope it helps. It should work. If it doesn't, try to remove the spaces in front of commas as described here: https://github.com/zitmen/thunderstorm/issues/25#issuecomment-127023949

Let me know if it helped. Thanks.

[Kyle Douglass]

Hi Martin,
Thanks! I discovered the spaces in the headers and tried it last week with the old version but I still received errors.

I will try it with the new version tomorrow and let you know what happens.

Thanks again,
Kyle

[Kyle Douglass]

Hi Martin,
I confirmed that the current stable version 1.3 2014-11-08 is unable to perform merging on the data when I remove the spaces in the column headers. Removing the spaces and repeating the same Merge operation (detailed here https://github.com/zitmen/thunderstorm/issues/25) now produces a null pointer error.

I then changed the column names by changing "sigma [nm]" to "uncertainty [nm]" and "bkgstd [photon]" to "offset [photon]". Repeating the merge operation resulted in the attached error message.

I next uninstalled the stable version of ThunderSTORM and installed lastest development build (downloaded August 4, 2015). Merging will successfully work in the development build, but the visualization no longer works.

Should I open a new issue with visualization problems? Thanks!

Kyle

[Martin Ovesný]

I really don't know. This error should never occur. It was part of the change I made in the last build. If a similar error occured, it would be in a log window and not in a pop-up dialog. Did you restart ImageJ after the update?

I have no idea where this error came from. It simply does not exist in the new version anymore. And unfortunatelly I'm not able to reproduce it.

Unless I have more information, I cannot fix it. If you manage to make it work, please let us know.