Re: Asylum ARDF to USID translation

[Tommaso Costanzo]

Tommaso:

 I have got something working for the AR hdf5 to pycroscopy hdf5. So far I have trimmed the extension segment to have all the data of the same length. This in my opinion is a better approach than just fill the data with NaN or symilar. To keep the entire information I will include as an extra dataset how many points where removed at every position. Now I have a couple of questions: first in the Asylum file there is a group called "Images" where images are saved (e.g. the height map or adhesion map), however I do not know the best way to incorporate them in the pycroscopy hdf5 because they are not Channel but they are pert of the Measurement. Is it ok to have a subgroup of Measurement called 'Images'?

Second to be as general as possible I have made the spectroscopic dimension only as a function of 'Time', which will be independent from the measurement type (e.g. it does not matter if it is PFM, cAFM, Force and so on). However, this preclude a direct analysis of the real information that one might look for, like deflection as a function of Bias. My idea is to have dedicated function/method to use depending on what analysis should be done. My question is where these functions should be placed: somewhere with the analysis tools or as methods of the class I have made for the translation? In addition to this, these function will produce new datasets (for example from DART PFM measurements it is possible to use the SHO model to reduce the original information to Amp, Phase, Freq, Q fact at resonance, similarly to BE data), and I would like to ask where it is the best place to store them in the file?

Suhas:

To answer your questions:
1. Here is how I would lay out the data:
/Measurement_000/Channel_000/ <-- this contains your 3D dataset
/Measurement_000/Channel_001/ <--- contains the height image
/Measurement_000/Channel_002/ <-- contains adhesion…

Can you differentiate a PFM spectroscopy file from a cAFM, Force Spectroscopy file? If so, this knowledge should be embedded into the Spectrosocpic Values - it should be in nanometers for the Force, Volts for the PFM bias, and so on

The goal is to ensure that the data is as self explanatory as possible while respecting the fact that the raw data is stored as it was acquired from the instrument with minimal preprocessing

Anything that uses knowledge of physics to reduce (BE / DART -> SHO) or further analyze the data should go into pycroscopy.analysis

[Tommaso Costanzo]

Ok I will places Images in separate Channels!

About the differantiation of PFM, cAFM etc is not completely straightforward (from a programming point of view). I decided to use time because it is actually the real way in which the raw data are recorded. In addition to this, is some variables (e.g. bias) does not extend for the entire time colelcted.

My idea was to store in the hdf5 the information required to successively reprocess the data and obtain the desired information. This allow a better flexibility especially if someone collect data in some custom way. In this case he will still be able to use the translator and then just develop his own custom function to analyze the data.

Let me know what you think!

[suhas.somnath]

In that case, the data can be written with time as the spectroscopic dimension for now. However, this is not ideal. One work-around is to write a base ARDF class that only sets the spectroscopic axis to time. Children classes can extend this base class and modify things once more information is available for being more specific about the spectroscopic dimension. This will also allow users to save custom data to ARDF files and then simply extend the base ARDF python class to change the spectroscopic axis as necessary. 

[Tommaso Costanzo]

Yes, that is what I was thinking. Moreover, what I can think as children classes are are all analysis tools. I am thinking to develop something for DART PFM since are the data I am mostly working with, but before going there I will clean up and test a bit more the translator.

So far, I just cloned the master pycroscopy and pyUSID and added my edits there. What do you suggest to do before creating a pull request? Is it better to move my edits to a new branch, or some developer branch?

[Suhas Somnath]

You should not need to clone or edit pyUSID - this package only contains the engineering tools. Please see these instructions for adding your code to pycroscopy

At the moment, you probably have the basic code that will form a Translator class. This class can exist in its own file and should go to pycroscopy.io.translators

Your physical-model based analysis code should go to pycroscopy.analysis. You are recommended to use the Process class for  structuring your processing / analysis code. 

Let me or Chris know if you are having any trouble

[Tommaso Costanzo]

Yes I am sorry for the confusion but the translator is in pycroscopy.io.translators and I will put the analysis in pycroscopy.analysis.

[Suhas Somnath]

No worries. We are happy to guide. If you are ever confused, consider visiting our websites. There is a good chance that your question may be answered in the documentation somewhere.