Skip to first unread message
Misha Kudryashev
Jul 28, 2014, 5:45:56 AM7/28/14
to emsp...@googlegroups.com
HI!
I use Emspring-0.81.1282dev and getting a strange error
logged on Mon, 28 Jul 2014 10:23:42
.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 1110, in main
segment_stack.perform_iterative_projection_matching_and_3d_reconstruction()
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 1051, in perform_iterative_projection_matching_and_3d_reconstruction
ref_cycle_id, info_series, fsc_lines, required_byte, pixelinfo)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 497, in project_including_masking_and_filtering_mpi
each_reference.ref_file, pixelinfo, each_reference.helical_symmetry, prj_prefix)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_project.py", line 531, in project_through_reference_volume_in_helical_perspectives
projection_stack, helical_symmetry)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 446, in generate_projection_stack_mpi
self.copy_image_stack_to_new_stack(projection_stack, local_projection_stack)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.81.1282dev-py2.7.egg/spring/segment3d/refine/sr3d_project.py", line 485, in copy_image_stack_to_new_stack
img_count = EMUtil.get_image_count(projection_stack)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/parts/EMAN2/lib/EMAN2db.py", line 527, in db_get_image_count
raise Exception,fsp
Exception: projection_stack009.hdf
Rank 0 on machine usi35.cluster.bc2.ch with process ID 54639 raised an error message. Find the corresponding error message above. All other MPI processes are aborted. Ignore the error messages thereafter as they can arise as a result of the aborting procedure.
INFO:kudryash:/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/parts/openmpi/bin/mpirun -v --hostfile nodefile.dat -np 168 /home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/bin/springenv /home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/bin/python cleanup.py
the file projection_stack009.hdf exists and its just fine, has 15 images.
Anyone had an issue like this?
best,
Misha
Carsten Sachse
Jul 28, 2014, 6:00:28 AM7/28/14
to emsp...@googlegroups.com
Dear Misha,
I suspect this is a problem with disk space. The previous SPRING version you are using required a lot of temporary disk space. This has been greatly improved in the latest version. I highly recommend updating as I will not be able to debug old issues.
Read instructions here: https://groups.google.com/forum/?hl=en#!topic/emspring/GmKqpbA-us8
Best wishes,
Carsten
Misha Kudryashev
Jul 29, 2014, 12:10:32 PM7/29/14
to emsp...@googlegroups.com
Hi Carsten,
Thanks for the help, I updated the version and it is now package Emspring-0.82.1339dev, re-extracted the particles, cleaned everything up, re-initialized the project with the default values and I think the same error persists. Anything else I could check? (log file below)
best,
Misha
....
16519) (268, 16520) (269, 16521) (270, 16522) (271, 16523) (272, 16524) (273, 16525) (274, 16526) (275, 16527) (276, 16528) (277, 16529) (278, 16530) (279, 16531)
INFO:CPU0:filter_and_mask_reference_volume:
INFO:kudryash:The volume will be helically symmetrized: 4.88, 21.8, 29.4, 0.8, 152 (pixelsize, rise, rotation, z_section, z_height)
INFO:kudryash:The following filter coefficients have been computed:
normalized spatial frequencies filter coefficients
-------------------------------- ---------------------
0 1
0.00666667 1
0.0133333 1
0.02 1
0.0266667 1
0.0333333 1
0.04 1
0.0466667 1
0.0533333 1
0.06 1
0.0666667 1
0.0733333 1
0.08 1
0.0866667 1
0.0933333 1
0.1 1
0.106667 1
0.113333 1
0.12 1
0.126667 1
0.133333 1
0.14 INFO:CPU122:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
0.16 1
0.166667 1
0.173333 1
0.18 1
0.186667 1
0.193333 1
0.2 1
0.206667 1
INFO:CPU41:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
FO:CPU6:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
0.246667 1
0.253333 1
0.26 1
0.266667 1
0.273333 1
0.28 INFO:CPU25:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29INFO:CPU51:project_tINFO:CPU127:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
1
0.32 1
0.326667 1
0.333333 1
0.34 1
0.346667 1
0.353333 1
0.36 INFO:CPU58:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
0.38 1
INFO:CPU102:project_through_reference_volume_in_helical_INFO:CPU63:project_INFO 0.999999
INFO:kudryash:The volume was projected according to the following parameteINFO:CPU92:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
----- ---------
2 0 67.2 INFO:kudryash:The volume was projected according to the following parameteINFO:kudryash:The volume was projected according to the following parameters:
stack_id local_id phi theta psi x-shift y-shift
---------- ---------- ----- ------- ----- --------- ---------
4 0 134.4 90 270 0 0
logged on Tue, 29 Jul 2014 18:04:41
rough_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
FO:kudryash:The volume was projected according to the following parameters:
stack_id local_id phi theta psi x-shift y-shift
---------- ---------- ----- ------- ----- --------- ---------
5 0 168 90 270 0 0
logged on Tue, 29 Jul 2014 18:04:42
most recent call last):
File "/home/phil2/kudryash/procs/spring/csachseINFO:CPU71:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on TueINFO:kudryash:The volume was projected according to the following parameters:
stack_id local_id phi theta psi x-shift y-shift
---------- ---------- ----- ------- ----- --------- ---------
3 0 100.8 90 270 0 0
logged on Tue, 29 Jul 2014 18:04:41
g to the following parameters:
stack_id local_id phi theta psi x-shift y-shift
---------- ---------- ----- ------- ----- --------- ---------
6 0 201.6 90 270 0 0
logged on Tue, 29 Jul 2014 18:04:42
fo)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.82.1339dev-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 539, in project_including_mINFO:CPU89:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
2.7/site-packages/emspring-0.82.1339dev-py2.7.egg/spring/segment3d/refine/sr3d_project.py", line 531, in project_through_reference_volume_in_helical_perspectives
projection_stack, helical_symmetry, rotational_sym)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.82.1INFO:CPU86:project_through_reference_volume_in_helical_perspectives: in progress ...
logged on Tue, 29 Jul 2014 18:04:41
_to_new_stack(projection_stack, local_projection_stack)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/lib/python2.7/site-packages/emspring-0.82.1339dev-py2.7.egg/spring/segment3d/refine/sr3d_project.py", line 485, in copy_image_stack_to_new_stack
img_count = EMUtil.get_image_count(projection_stack)
File "/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/parts/EMAN2/lib/EMAN2db.py", line 527, in db_get_image_count
raise Exception,fsp
Exception: projection_stack001.hdf
Rank 0 on machine usi13.cluster.bc2.ch with process ID 118518 raised an error message. Find the corresponding error message above. All other MPI processes are aborted. Ignore the error messages thereafter as they can arise as a result of the aborting procedure.
INFO:kudryash:/home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/parts/openmpi/bin/mpirun -v --hostfile nodefile.dat -np 128 /home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/bin/springenv /home/phil2/kudryash/procs/spring/csachse-emspring-454a5fe5b8d1/bin/python cleanup.py
Carsten Sachse
Jul 29, 2014, 2:45:45 PM7/29/14
to emsp...@googlegroups.com
Hi Misha,
Alright. Could you send me your parameter file to have a closer look? The log so far is not conclusive. Do you have the full stderr? The log message can be very convoluted from the different CPU messages.
When does this error occur after the first refinement cycle? How many files are written?
Best wishes,
Carsten
...
Misha Kudryashev
Jul 31, 2014, 3:54:42 AM7/31/14
to emsp...@googlegroups.com
Hi Carsten,
Thanks, it crashed suring the first iteration, in the meantime it makes the files:
nodefile.dat parameters.par projection_stack001.hdf recvol_ref_001.hdf report.log spring.db
I am using qsub submission system which just submits the .py script with 128 processors. the .par file is below...
Any ideas?
Thanks!
Misha
Image input stack refinement = /home/phil2/kudryash/data_sheath/segment_29_Jul_2014_12_18_48_117076/protein_stack_bin4.hdf
Output volume name = recvol.hdf
Diagnostic plot prefix = diagnostic_plot.pdf
Number of iterations = 20
Reference structure option = True
Reference volume = /home/phil2/kudryash/data_sheath/segmentrefine3d_18_Jul_2014_23_14_02_130271/recvol_244apix_004.hdf
spring.db file = /home/phil2/kudryash/data_sheath/segment_29_Jul_2014_12_18_48_117076/spring.db
Continue refinement option = False
refinement.db file = refinement.db
Assemble refinement strategy = True
LR - Low resolution aim = False
LR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)
LR - X and Y translation range in Angstrom = (50, 23)
MR - Medium resolution aim = True
MR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)
MR - X and Y translation range in Angstrom = (100, 15)
HR - High resolution aim = True
HR - azimuthal and out-of-plane search restraint in degrees = (20.0, 20.0)
HR - X and Y translation range in Angstrom = (100, 15)
MaxR - Maximum resolution aim = True
MaxR - azimuthal and out-of-plane search restraint in degrees = (2.0, 2.0)
MaxR - X and Y translation range in Angstrom = (7, 4)
Absolute X and Y translation limit in Angstrom = (100, 30)
Independent half-set refinement = False
Half-set refinement start = Medium
High-pass filter option = False
Low-pass filter option = True
High and low-pass filter cutoffs in 1/Angstrom = (0.001, 0.09)
B-Factor = 0
Custom filter option = False
Custom-built filter file = filter_function.dat
Automatic FSC filter = True
Micrographs select option = False
Include or exclude micrographs = Include
Micrographs list = 1-9, 11, 13
Helices select option = False
Include or exclude helices = Include
Helices list = 1-9, 11, 13
Segments select option = False
Include or exclude segments = Include
Segment file = stackid_file.dat
Classes select option = False
Include or exclude classes = Include
Class type = Class_id
Classes list = 1-9, 11, 13
Persistence class option = False
Persistence class length in Angstrom = 700
Class occupancy threshold = 0.5
Straightness select option = False
Include or exclude curved helices = Include
Curvature range = (0.0, 1e-05)
Layer line correlation select option = False
Include or exclude segments based on layer-line correlation = Include
Correlation layer line range = (0.6, 1.0)
Defocus select option = False
Include or exclude defocus range = Include
Defocus range = (10000, 40000)
Astigmatism select option = False
Include or exclude astigmatic segments = Include
Astigmatism range = (0, 4000)
Projection correlation select option = False
Include or exclude segments based on projection correlation = Include
Correlation projection range = (1.0, 1.0)
Out-of-plane tilt select option = False
Include or exclude out-of-plane tilted segments = Include
Out-of-plane tilt range = (-5, 5)
Shift normal to helix select option = False
Include or exclude segments with shift normal to helix = Include
Shift normal to helix in Angstrom = 5.0
Keep intermediate files = False
Estimated helix inner and outer diameter in Angstrom = (50, 280)
Pixel size in Angstrom = 2.44
Symmetrize helix = True
Helical rise/rotation or pitch/number of units per turn choice = Rise/rotation
Helical symmetry in Angstrom or degrees = (21.8, 29.4)
Rotational symmetry = 6
Helix polarity = Polar
Force helical continuity = True
Limit in-plane rotation = True
Delta in-plane rotation angle = 10.0
Out-of-plane tilt angle range = (-12, 12)
Number of projections azimuthal/out-of-plane angle = (7, 1)
Image alignment size in Angstrom = 600
Step size of segmentation in Angstrom = 70
3D CTF correction = True
3D CTF correction intensity = Low
MPI option = True
Number of CPUs = 128
Temporary directory = /tmp
...
Carsten Sachse
Jul 31, 2014, 4:54:09 AM7/31/14
to emsp...@googlegroups.com
Hi Misha,
Looking at your parameter file I see that you are deviating strongly from the recommended defaults. Go back to the defaults as close as possible and only change helix-specific parameters like symmetry etc.
The culprit is the low number of requested projections:
Number of projections azimuthal/out-of-plane angle = (7, 1)
The defaults are (90, 7)
i.e. you have request 7 projections around the helix, which is way too little for any sensible reconstruction. The program actually crashes because it cannot distribute 7 projections on 128 nodes.
Other comments:
1. You should always use 'Low resolution refinement aim' because it will speed up the alignment and convergence.
2. Shift translations (100, 15) are quite excessively high and low.
I hope this helps.
Best wishes,
Carsten
...0.333333 1  
Misha Kudryashev
Jul 31, 2014, 7:29:03 AM7/31/14
to emsp...@googlegroups.com
HI Carsten,
Thanks for the mail again!
I tried to fix the parameters and it finally works. however I am not sure what is the exact fix. Just FYI uploading .par file that works...
looks like 10 projections are efficiently enough spread over 128 processors (at least not erros). I have C6 and high rotational symmetry so the idea is just to do initial alignment and then increase the number of projections. It did a refinement of 36k particles in 20 mins and wrote an .db file, hope it runs through!
All the best
Misha
Image input stack refinement = /home/phil2/kudryash/data_sheath/protein_stack_bin4.hdf
Output volume name = recvol.hdf
Diagnostic plot prefix = diagnostic_plot.pdf
Number of iterations = 20
Reference structure option = True
Reference volume = /home/phil2/kudryash/data_sheath/recvol_488apix_007.hdf
spring.db file = /home/phil2/kudryash/data_sheath/spring_segment.db
Continue refinement option = False
refinement.db file = refinement.db
Assemble refinement strategy = True
LR - Low resolution aim = True
LR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)
LR - X and Y translation range in Angstrom = (50, 23)
MR - Medium resolution aim = True
MR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)
MR - X and Y translation range in Angstrom = (21, 10)
HR - High resolution aim = True
HR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)
HR - X and Y translation range in Angstrom = (100, 15)
MaxR - Maximum resolution aim = True
MaxR - azimuthal and out-of-plane search restraint in degrees = (2.0, 2.0)
MaxR - X and Y translation range in Angstrom = (7, 4)
Absolute X and Y translation limit in Angstrom = (100, 15)
Independent half-set refinement = False
Half-set refinement start = medium
High-pass filter option = False
Low-pass filter option = True
High and low-pass filter cutoffs in 1/Angstrom = (0.001, 0.09)
B-Factor = 0
Custom filter option = False
Custom-built filter file = filter_function.dat
Automatic FSC filter = True
Filter layer-lines option = False
Micrographs select option = False
Include or exclude micrographs = include
Micrographs list = 1-9, 11, 13
Helices select option = False
Include or exclude helices = include
Helices list = 1-9, 11, 13
Segments select option = False
Include or exclude segments = include
Segment file = stackid_file.dat
Classes select option = False
Include or exclude classes = include
Class type = class_id
Classes list = 1-9, 11, 13
Persistence class option = False
Persistence class length in Angstrom = 700
Class occupancy threshold = 0.5
Straightness select option = False
Include or exclude curved helices = include
Curvature range = (0.0, 1e-05)
Layer line correlation select option = False
Include or exclude segments based on layer-line correlation = include
Correlation layer line range = (0.6, 1.0)
Defocus select option = False
Include or exclude defocus range = include
Defocus range = (10000, 40000)
Astigmatism select option = False
Include or exclude astigmatic segments = include
Astigmatism range = (0, 4000)
Projection correlation select option = False
Include or exclude segments based on projection correlation = include
Correlation projection range = (1.0, 1.0)
Out-of-plane tilt select option = False
Include or exclude out-of-plane tilted segments = include
Out-of-plane tilt range = (-5, 5)
Shift normal to helix select option = False
Include or exclude segments with shift normal to helix = include
Shift normal to helix in Angstrom = 5.0
Keep intermediate files = False
Estimated helix inner and outer diameter in Angstrom = (50, 260)
Pixel size in Angstrom = 2.44
Symmetrize helix = True
Helical rise/rotation or pitch/number of units per turn choice = rise/rotation
Enforce even phi option = False
Release cycle even phi = 8
Pitch enforce even phi = 8.0
Bin cutoff of phi angles = 100
Helical symmetry in Angstrom or degrees = (21.8, 29.4)
Rotational symmetry = 6
Helix polarity = polar
Unbending option = False
Force helical continuity = True
Limit in-plane rotation = True
Delta in-plane rotation angle = 10.0
Out-of-plane tilt angle range = (-2, 2)
Number of projections azimuthal/out-of-plane angle = (10, 1)
Image alignment size in Angstrom = 600
Step size of segmentation in Angstrom = 70
3D CTF correction = True
3D CTF correction intensity = low
MPI option = True
Number of CPUs = 128
Temporary directory = /tmp
...0.32 &n
Reply all
Reply to author
Forward
0 new messages