Trubble open .ibw format

[Niklas W]

Hello, 

I am supervising an AFM from Asylum research and since than most of my users used Gwyddion to evaluate the recorded data. To make this evaluation more quantitative I would like to access the measurement data using Python. The data is stored in the .ibw format is there a simple way to import the data to python (sample data https://1drv.ms/u/s!ArojbO7a7fVYnUT62q46hDK5vYZI?e=mOQ5GC)? I am very new to python, worked previously with MATLAB and Java Script, so I would apprichiate any advice how to do it. I already tried using the python module "igor" but had no luck. For reference I am using Sypder 3.3.6 and Python 3.7.6 on MacOS Catalina. 

Thanks, 

Niklas  

[Niklas W]

If I do the folowing: 

import pycroscopy as pyc

a = pyc.io.IgorIBWTranslator('/Users/Name/Desktop/test.ibw')

i get: 

  File "/Users/Name/Documents/Python/test.py", line 12, in <module>

    a = pyc.io.IgorIBWTranslator('/Users/Name/Desktop/test.ibw')

  File "/usr/local/anaconda3/lib/python3.7/site-packages/pyUSID/io/translator.py", line 36, in __init__

    self.max_ram = min(max_mem_mb * 1024 ** 2, 0.75 * get_available_memory())

TypeError: '<' not supported between instances of 'float' and 'str'

[Rama Vasudevan]

HI Niklas,

I think this might be because you have instantiated the class with the file path as an argument. Can you try

import pycroscopy as px

translator = px.io.IBWTranslator()

file_path = r'/Users/Name/Desktop/test.ibw'

translator.translate(file_path)

and let us know what happens?

Kind Regards,

Rama

[Niklas W]

Hello, 

first of all thank you for the fast response 

import pycroscopy as px

translator = px.io.IgorIBWTranslator()

file_path = r'/Users/Name/Desktop/test.ibw'

a = translator.translate(file_path)

worked fine, but did not solve the problem how to actually load the data.

What to do next? 

Kind Regards, 

Niklas 

[Niklas W]

Maybe to be a bit more precise: 

The translator seems to be able to "translate" the given data, and the script outputs a translator object, that contains the folowing: 

[Rama Vasudevan]

The translator creates a hdf5 file with the data and standardizes it according to the USID model.  You should proceed to inspect it using pyUSID (read here). 

Example:

import os

import h5py

import pyUSID as usid

import pycroscopy as px

file_path = r'/my_folder/a10.ibw'

#Do the translation

translator = px.io.translators.IgorIBWTranslator()

h5_path = translator.translate(file_path) #returns path of translated file

#Open the translated file

h5_f = h5py.File(h5_path, 'r+')

#Inspect the h5 file

usid.hdf_utils.print_tree(h5_f)

#Get all the main datasets

main_dsets = usid.hdf_utils.get_all_main(h5_f)

#Print out the list of main datasets

print(main_dsets)

[Niklas W]

Thank you so much, I will have a closer look into this format. Is it possible to contact you in case i have some further questions? 

 

[Rama Vasudevan]

Certainly! You can email me ( vasudevanrk at ornl dot gov) for general questions. If you run into a bug, please do report it here so that we can track and resolve issues that others may also have.

Rama

[Suhas Somnath]

Does it make sense to add your example script on how to translate files to this page - https://pycroscopy.github.io/pycroscopy/translators.html ?

[Niklas W]

I think so too :)

[Rama Vasudevan]

Done

[Rajiv Giridharagopal]

Hi Niklas,

Welcome! As a follow-up, here are some quick tips (making a "cookbook" is still on my to-do list...) from the data.

Looking at your test file, it looks like it has 6 channels of information. For easier access, it helps to use some of benefits of pyUSID (the actual data format/interfacing package):

https://pycroscopy.github.io/pyUSID/auto_examples/beginner/plot_usi_dataset.html#creating-a-usidataset

So, from Rama's code, we have main_dsets, which is a list of all the "main" datasets. The first one is height, so we could do

height = main_dsets[0]

If you just type "height" on the command line, it shows a lot of cool information about your dataset! If you wanted to just plot it, you could do so:

height.visualize()

And that's it! Visualize takes lots of common Python plotting arguments, so if you were to write "cmap = inferno" it would look like the conventional AFM coloscale. There are lots of colormap, of course.

To access your data directly, you extract the array from your dataset.

height_array = height[()]

However, while it's a 512 x 512 image,  the way it's saved in the H5 file is actually [512^2,1] for reasons you can read about in the USID documentation. So height_array is actually not in an image matrix. For analysis, you often want the matrix form, in which case the function "get_n_dim_form()" part is what converts it back into a matrix for imaging. See here: https://pycroscopy.github.io/pyUSID/auto_examples/beginner/plot_usi_dataset.html#reshaping-to-n-dimensions

height_array = height.get_n_dim_form()

Rama and Suhas can probably suggest further steps as you need! :)

Best,

Raj

[Suhas Somnath]

Raj / Rama - it would be great to add these lines of code + comments to the documentation. 

[Niklas W]

Thanks guy's! You helped quit a lot and probably more than you know! @Rama if you want you can also link the file that I attached as a test file. 

[Niklas W]

Hello Rama, 

thank you for the help with opening .ibw files in python using pycroscopy. I am currently struggeling with accessing the meta data stored in the .ibw file (sample data https://1drv.ms/u/s!ArojbO7a7fVYnUT62q46hDK5vYZI?e=mOQ5GC) such as scan speed, scan size, setpoint etc. 

Thank you in advance, 

Niklas 

[Rama Vasudevan]

HI Niklas,

You can access attributes to any dataset by calling usid.hdf_utils.get_attributes(dset) 

where dset is a dataset or a group. In your case (continuing on from the code block I wrote in the previous message), for example try

h5_main = main_dsets[0]
print('\nMetadata or attributes in a datagroup\n------------------------------------')
for key, val in usid.hdf_utils.get_attributes(h5_main.parent.parent).items():
    print('{} : {}'.format(key, val))

You may find all of this information within pyUSID's documentation:

https://pycroscopy.github.io/pyUSID/auto_examples/beginner/plot_hdf_utils_read.html

Hope this helps.

Rama

[Niklas W]

Thanks! Now literally everything is solved :) 

[niklas....@googlemail.com]

Hey,

I just had to reinstall python on my PC and now pycroscopy seems to be "broken". When executing this code

import os

import h5py

import pyUSID as usid

import pycroscopy as px

file_path = r'/my_folder/a10.ibw'

#Do the translation

translator = px.io.translators.IgorIBWTranslator()

h5_path = translator.translate(file_path) #returns path of translated file

#Open the translated file

h5_f = h5py.File(h5_path, 'r+')

#Inspect the h5 file

usid.hdf_utils.print_tree(h5_f)

#Get all the main datasets

main_dsets = usid.hdf_utils.get_all_main(h5_f)

#Print out the list of main datasets

print(main_dsets)

I get

AttributeError: module 'pycroscopy' has no attribute 'io'

Can you help me how to fix this issue? I assume something fundamentally has changed in the pycroscopy package

Cheers,

Niklas

[niklas....@googlemail.com]

Hello,

I solved the problem, apparently something fundamentally has changed over the last iterations of pycroscopy :(

The new method to import .ibw files to Python looks something like this:

import SciFiReaders as sc

class afmdata:
    def __init__(self):
        self.unit = ''
        self.width = ''
        self.heigt = ''
        self.map = []
        self.xaxis = []
        self.yaxis = []
        self.metadaten = {}

# Load File

path = ''

filename = 'file.ibw'
file = sc.IgorIBWReader(path+filename)

# Convert File to dict imho more readable

dsets = file.read()

data = {}
for channel in dsets:
    name = channel.quantity
    data[name] = afmdata()
    data[name].metadaten = channel.original_metadata
    data[name].width =  data[name].metadaten['FastScanSize']
    data[name].heigt = data[name].metadaten['SlowScanSize']
    points = data[name].metadaten['ScanPoints']
    lines =  data[name].metadaten['ScanLines']
    data[name].xaxis = np.arange(points) * data[name].metadaten['FastScanSize']/points
    data[name].yaxis = np.arange(lines) * data[name].metadaten['SlowScanSize']/lines
    data[name].map = np.rot90(np.array(channel))
    data[name].unit = channel.units 

type(channel) =  sidpy.sid.dataset.Dataset has some powerful features, that I did not use here, for example you could plot an individual channel by

#Plot channel as map

channel.plot()

I hope this is somewhat helpful and prevents some headaches I have had while untangling the datastructur!

Cheers

Niklas