Problem with large dataset

[Julian Reitz]

spring --version && springenv e2version.py
Spring environment loaded.
GUI from package Emspring-0.84.1470
Spring environment loaded.
EMAN 2.1 alpha2 (CVS 2013/08/07 17:01:09)
Your EMAN2 is running on:  Ubuntu 14.04.4 LTS 3.13.0-92-generic x86_64
Your Python version is:  2.7.2

I am facing issues in processing a large dataset. I wanted to test different parameters for "segment".
When I want to analyze the results with "segmentexam" I get an error message that can be seen in the attached log file.
I tried the same operation with a reduced dataset of only 10 micrographs and this works fine. Therefore I assume, that the large size of the project (452 micrographs; 130 GB .hdf file) is causing problems.
Is there a way to solve this issue?

Thank you very much,
Julian Reitz

[cyanpku]

Hi Julian,

I'm not sure what's the error mean, but if you suspect the file size is an issue, double check how much space is available at the /tmp/ directory. Or bin your input image stack. You don't need 2A resolution at this point.

Jing

[Julian Reitz]

Hi Jing,

thank you for your answer. The available space in /tmp is clearly more than enough.
As far as I can see the "segmentexam" operation does not have an binning option or?
Or do you suggest to bin during the "segment" operation?

Thank you,
Julian

[cyanpku]

yes during the segment, choose binning by a factor of 2 or 4, then use the bin stack for your input.

Another thing I noticed is your power spectrum resolution cutoff in the segment exam is NONE, not sure if spring freaks out by that...

For hlx reconsctruction, you don't need that many micrographs to achieve high resolution. I would choose less than 100 best images to try out the best parameters. 

[Simon Fromm]

Hi Julian,

to add my bit to Jing's suggestions which I all agree with, SPRING shouldn't have a problem with the power spectrum cutoff. As long as you don't specify anything ('NONE'), SPRING will use a default parameter, in this case 0.15 for the power spectrum cutoff.

Your original error message states:

RuntimeError: ImageReadException at /home/eman64/EMAN2/src/eman2/libEM/hdfio2.cpp:475: error with '/tmp/reitz-18_Jul_2016_15_06_44-4513/infilestack-masked.hdf': 'HDF5 file has no image data (no /MDF group)' caught

This points to a problem in writing to the tmp directory as Jing already suspected. Binning during the segmentation could help there.

Cheers, Simon

--
You received this message because you are subscribed to the Google Groups "emspring" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emspring+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Reitz]

Thank you for the answer. Binning during the segmentation seems to solve the problem.

Best,
Julian