could not get ICA working

[Nina]

I have just installed the windows 64bit version and the program seems
working properly. But I had some problems when trying to run ICA on my
spectrum image data (a line scan). When I tried to plot the residual
the residuals between the original data and the reconstructed data,
it showed "Object does not have any residual information. Is it a
reconstruction created using either pca_build_SI or ica_build_SI
methods?". Also, I could not get the reconstructed spectrum image
saved properly, the syntax "save('filename.rpl')" doesn't seem to be
working for me; the resulting rpl and raw files opened in DM looks
nothing like a spectrum image.

Hope somebody can help me on this problems. Thanks a lot.

Na Ni


The lines I have entered are as follows:

s=load()
EELSSpectrum
Title: C:\Users\NaNi\post doc work\resutls\SiC\TEM\sample
2\EELS 2011-10-20\eels\0.3 dispersion 38 mm cl 2nd ca beta 15.1\3_EELS
Spectrum Image_si+b calibrated.dm3
Signal type: EELS
Data dimensions: (31L, 2048L)
Data representation: spectrum

s.remove_spikes()
Spike detected in (0,)
Spike detected in (1,)
Spike detected in (2,)
Spike detected in (3,)
Spike detected in (4,)
Spike detected in (5,)
Spike detected in (6,)
Spike detected in (7,)
Spike detected in (8,)
Spike detected in (9,)
Spike detected in (10,)
Spike detected in (11,)
Spike detected in (12,)
Spike detected in (13,)
Spike detected in (14,)
Spike detected in (15,)
Spike detected in (16,)
Spike detected in (17,)
Spike detected in (18,)
Spike detected in (19,)
Spike detected in (20,)
Spike detected in (21,)
Spike detected in (22,)
Spike detected in (23,)
Spike detected in (24,)
Spike detected in (25,)
Spike detected in (26,)
Spike detected in (27,)
Spike detected in (28,)
Spike detected in (29,)
Spike detected in (30,)

s.principal_components_analysis()

Performing principal components analysis
Performing PCA with a SVD based algorithm
Undoing data pre-treatments

s.plot_lev()
Out[6]: <matplotlib.axes.AxesSubplot at 0xe437978>

s.independent_components_analysis(3)


s2=s.ica_build_SI(3)

s2.plot()

s2.plot_residual()
Object does not have any residual information. Is it a reconstruction
created using either pca_build_SI or ica_build_SI methods?

s2=s.ica_build_SI(2)

s2.plot_residual()
Object does not have any residual information. Is it a reconstruction
created using either pca_build_SI or ica_build_SI methods?

s2.save(3_PCA)
File "<ipython-input-17-fba54aa09ffb>", line 1
s2.save(3_PCA)
^
SyntaxError: invalid syntax


s2.save('3_PCA')
The 3_PCA.hdf5 file was created

s2.save('3_PCA.rpl')
The 3_PCA.rpl file was created

[Francisco de la Peña]

Dear Na Ni,

The residuals issue is a bug, thank you for reporting it. We'll try to fix it soon but, in the meantime, you can use pca_build_SI instead that should produce exactly the same result with the difference that plotting the residuals works.

To export the PCA results you must use the export_pca_results method, e.g. s.export_pca_results(3) . The save method stores the pca results in the hdf5 file, but you cannot open this kind of files with DM. In contrast, the export_pca_results method produces Ripple files that should open correctly in DM with the Ripple plugin.

By the way, be careful while using the remove_spikes method; remember that you have to previously determine the threshold using the spikes_diagnosis method. Also, you'll get better results by normalizing the noise when perfoming pca, i.e. s.principal_components_analysis(True)

Best,

Francisco

2011/11/1 Nina <na.ni...@gmail.com>

[na ni]

Dear Francis,

Thank you for your prompt help. But is the export_pca_results method supposed to save the reconstructed spectrum image as well? It looks it  only gives me the components but not the spectrum image.

Best wishes,

Na

[Michael Sarahan]

No. The export_pca_results function gives you the factors and scores
from PCA. The reconstructed SI is a matrix multiplication of these
two things.

Saving the reconstructed SI itself would be a command like:
s.principal_components_analysis(True)
s2=s.pca_build_SI(5)
s2.save('filename.rpl')

Does this work for you? If not, there's a bug in the Ripple reader
that I probably introduced trying to get the PCA results to work. Let
me know, and I'll look into it soon.

Best,
Mike

[na ni]

No. I don't think it works. I have previously tried the save (filename.rpl) method, which created the rpl and raw files but gave wired spectra.

Best wishes,

Na

[Francisco de la Peña]

Na,

I am writing  to confirm that, as Mike pointed out, the files created with Hyperspy are fine, but there is a bug in the ImportRPL DM plugin.

Mike, are you working on it? Otherwise I will...

Best,

Francisco

2011/11/1 na ni <na.ni...@gmail.com>

[Michael Sarahan]

Not at the moment. Stuck on the microscope for another day or two.
I'm pretty sure it's transposes that I did to make the PC's and scores
look right. This is the byte order problem that I always run into
with DM. Whatever you do, make sure that both the SI and the
PCs/factors both look OK. I thought I did that, but I guess I missed
something.

-Mike

[na ni]

Thanks both Francisco and Mike. Take your time on fixing the problem. Just drop me a notice when it is done.

Best wishes,

Na

[Francisco de la Peña]

Dear Na,

We have just released a new version of the DM plugin to read Ripple files. Could you confirm that it fixes the bug that you have reported?

[na ni]

Hi Francisco,

I will let you know as soon as I check the data again. Thanks.

Best wishes,

Na