Updated GIV notebook

[Suhas Somnath]

We do not guarantee that the notebook available within the GitHub project would work for versions past or prior to the version for which the notebook was written. If the notebook uses legacy functions which are causing probems, you may need to downgrade your pycroscopy version (preferably uninstall what you have and install an older version. 

As part of another effort, I had recently updated the notebook a while ago for a more recent pycroscopy / pyUSID version. The vast majority of this notebook should work without any problems. Please feel free to update / correct the notebook as appropriate.

[Wei Cai]

My current environment is:

pycroscopy         0.60.1

pyUSID             0.0.4

Thanks for update the notebook file. In this new notebook, I can run to this line:

fig_lab, fig_centroids = plot_cluster_results_separately()

The error message is NameError: name 'plot_cluster_results_separately' is not defined.

And before this line, some functions like plot_utils.plot_map() and twin_img_plots() seems have no x_size property, so I commented the x_size out, then the corresponding cell can run but the plotted images would not have the x_size values?

Please help. Thanks a lot!

[Chris Smith]

At the top of the notebook, it specifies that it should work with pycroscopy version 0.59.4.  Have you tried installing this version and running the notebook?  See the link provided above by Suhas on how to install a legacy version.

[Wei Cai]

Hi Chris, thanks for pointing out the version. I installed the 0.59.4 using pip and then ran the notebook above. This time I am stucked at this line:

from pycroscopy.processing.signal_filter import SignalFilter

sig_filt = SignalFilter(h5_main, frequency_filters=frequency_filters, 

                           noise_threshold=noise_tol, write_filtered=True,

                           write_condensed=False, num_pix=1)

The error message is:

KeyError                                  Traceback (most recent call last)

<ipython-input-13-05aba438e9f4> in <module>()

      9 sig_filt = SignalFilter(h5_main, frequency_filters=frequency_filters, 

     10                            noise_threshold=noise_tol, write_filtered=True,

---> 11                            write_condensed=False, num_pix=1)

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\processing\signal_filter.py in __init__(self, h5_main, frequency_filters, noise_threshold, write_filtered, write_condensed, num_pix, phase_rad, **kwargs)

    108 

    109         self.process_name = 'FFT_Filtering'

--> 110         self.duplicate_h5_groups = self._check_for_duplicates()

    111 

    112         self.data = None

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\processing\process.py in _check_for_duplicates(self)

    109             List of groups satisfying the above conditions

    110         """

--> 111         duplicate_h5_groups = check_for_old(self.h5_main, self.process_name, new_parms=self.parms_dict)

    112         if self.verbose:

    113             print('Checking for duplicates:')

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\io\hdf_utils.py in check_for_old(h5_base, tool_name, new_parms, verbose)

   1535     """

   1536 

-> 1537     groups = findH5group(h5_base, tool_name)

   1538 

   1539     for group in groups:

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\io\hdf_utils.py in findH5group(h5_main, tool_name)

    338     for key in parent_grp.keys():

    339         if dset_name in key and tool_name in key:

--> 340             groups.append(parent_grp[key])

    341     return groups

    342 

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

~\Anaconda3\envs\py36\lib\site-packages\h5py\_hl\group.py in __getitem__(self, name)

    175                 raise ValueError("Invalid HDF5 object reference")

    176         else:

--> 177             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)

    178 

    179         otype = h5i.get_type(oid)

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5o.pyx in h5py.h5o.open()

KeyError: 'Unable to open object (bad object header version number)' 

Please help! thanks a lot.

[Chris Smith]

Unfortunately, it looks like your h5 file has been corrupted.  If you delete the existing file and re-run the notebook, it should create a new file and be able to run the analysis.

[Suhas Somnath]

Also @Wei, please check to make sure that your notebooks are not the source of any problems. You can find both the original notebook submitted to the journal (pycroscopy v. 0.59.) and the updated one (0.60.1) available here: http://nbviewer.jupyter.org/github/pycroscopy/papers/tree/master/Notebooks/SPM/AFM/

I checked the latter and did not face the problems that you listed above. 

[Wei Cai]

    Thanks for the Suhas’s suggestions. I setup two separate environments in Anaconda. One is py36 + pycroscopy (0.60.1) + pyUSID (0.0.4) (Environment I) and the other is py36 + pycroscopy (0.59.4) + pyUSID (0.0.1) (Environment II). And make sure the h5 date file is original (Thanks Chris, the h5 file has been corrupted indeed) The revised notebook (2018 revised version) seems still not running to the end in both environments.
    In environment I, it stucked at the compute() function. The details are pasted as below, please help! And in environment II, the notebook can run more, especially it can run the compute() function for about one and a half hours. However, it then stucked at the plot_cluster_labels function in the part of “Main trends in the Current”. It seems that the module 'px.viz' has no attribute 'cluster_utils' in this version. Update px to 0.60.1 might solve this problem, but as mentioned above it can not run compute(), so please help! Thanks a lot!


Environment I
py36
pycroscopy         0.60.1
pyUSID             0.0.4

h5_bayesian_grp = i_cleaner.compute()

Resuming computation in group: /Measurement_000/Channel_000/Raw_Data-FFT_Filtering_000/Filtered_Data-Reshape_000/Reshaped_Data-Bayesian_Inference_000
You maybe able to abort this computation at any time and resume at a later time!
    If you are operating in a python console, press Ctrl+C or Cmd+C to abort
    If you are in a Jupyter notebook, click on "Kernel">>"Interrupt"
Starting computing on 3 cores (requested 3 cores)
Finished parallel computation
Starting computing on 3 cores (requested 3 cores)
Finished parallel computation
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-16-b38c4dfa1f34> in <module>()
----> 1 h5_bayesian_grp = i_cleaner.compute()

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\analysis\giv_bayesian.py in compute(self, override, *args, **kwargs)
    326         del(self._bayes_parms['freq'])
    327
--> 328         return super(GIVBayesian, self).compute(override=override, *args, **kwargs)

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\core\processing\process.py in compute(self, override, *args, **kwargs)
    331                 print('Time remaining: ' + format_time(time_remaining))
    332
--> 333             self._write_results_chunk()
    334             self._read_data_chunk()
    335

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\analysis\giv_bayesian.py in _write_results_chunk(self)
    259         self.h5_results_grp.attrs['last_pixel'] = self._end_pos
    260
--> 261         self.hdf.flush()
    262
    263         print('Finished processing up to pixel ' + str(self._end_pos) + ' of ' + str(self.h5_main.shape[0]))

AttributeError: 'GIVBayesian' object has no attribute 'hdf'


Environment II
py36
pycroscopy         0.59.4
pyUSID             0.0.1

AttributeError                            Traceback (most recent call last)
<ipython-input-9-97e14fba2ece> in <module>()
      1 ###plot_labels = partial(px.viz.cluster_utils.plot_cluster_labels,
----> 2 plot_labels = partial(px.viz.cluster_utils.plot_cluster_labels,
      3                       x_label='X ($\mu$m)', y_label='Y ($\mu$m)',
      4                       title='K-Means Cluster Labels')
      5 plot_centroids = partial(px.viz.cluster_utils.plot_cluster_centroids,

AttributeError: module 'pycroscopy.viz' has no attribute 'cluster_utils'

[Suhas Somnath]

Wei - 

Error on environment 1 - I caught and fixed the same bug on the master branch of pycroscopy a few weeks ago. It looks like we haven't updated the version of pycroscopy you get when you pip install. We will update pip soon. Until then, please install pycroscopy directly from the master branch instead of pip by following these instructions. You will need to install git if you don't already have it. Since the example is shown for the dev branch and not master, you will need to change the second command to:

pip install -U git+https://github.com/pycroscopy/pycroscopy@master

Error on Environment 2 (0.59.4) - It is great that you set up two separate environments. Please bear in mind that the notebook labeled 2018 would only work with pycroscopy 0.60.1 (environment 1) while the notebook labeled 2017 would only work with pycroscopy 0.59.4. it appears that you ran the 2018 notebook instead of the 2017 notebook on this environment. Nonetheless, it makes more sense time-wise for us to fix the newer notebook than the older one. 

[Wei Cai]

Thanks Suhas, After I installed git and reinstalled gycroscopy using the command

pip install -U git+https://github.com/pycroscopy/pycroscopy@master

the notebook can run more. Thank you so much!

Currently, it stopped at the “Dielectric Constant” part. It seems that the values of d and e0 are missing. I tried to set the value of d to 90E-9 and e0 to 8.854187817E-12, then the plotted images especially the second one seems blank, is it OK?

The next cell (sIV part) can load the sIV h5 file quite well, then the notebook got stuck at this line:

decomposer = px.processing.SVD(h5_siv_raw)

The error massage is pasted as below. Please help and many thanks!

 

Consider calling test() to check results before calling compute() which computes on the entire dataset and writes back to the HDF5 file

 

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

<ipython-input-52-a5d291e7a13b> in <module>()

----> 1 decomposer = px.processing.SVD(h5_siv_raw)

 

~\Anaconda3\envs\py36\lib\site-packages\pycroscopy\processing\svd_utils.py in __init__(self, h5_main, num_components)

     45         self.num_components = num_components

     46         self.parms_dict = {'num_components': num_components}

---> 47         self.duplicate_h5_groups, self.partial_h5_groups = self._check_for_duplicates()

     48

     49         # supercharge h5_main!

 

~\Anaconda3\envs\py36\lib\site-packages\pyUSID\processing\process.py in _check_for_duplicates(self)

    103         if len(duplicate_h5_groups) > 0:

    104             for index, curr_group in enumerate(duplicate_h5_groups):

--> 105                 if curr_group.attrs['last_pixel'] < self._end_pos:

    106                     # remove from duplicates and move to partial

    107                     partial_h5_groups.append(duplicate_h5_groups.pop(index))

 

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

 

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

 

~\Anaconda3\envs\py36\lib\site-packages\h5py\_hl\attrs.py in __getitem__(self, name)

     58         """ Read the value of an attribute.

     59         """

---> 60         attr = h5a.open(self._id, self._e(name))

     61

     62         if is_empty_dataspace(attr):

 

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

 

h5py\_objects.pyx in h5py._objects.with_phil.wrapper()

 

h5py\h5a.pyx in h5py.h5a.open()

 

KeyError: "Can't open attribute (can't locate attribute: 'last_pixel')"