segmentrefine3d with movie

cyanpku

unread,

Jul 26, 2016, 1:59:43 PM7/26/16

to emspring

Hi Carsten,

I'm stuck by the frame motion correction during the segmentrefind3d... With that option "False", the alignment with the movie stack segments runs well, except the polarity of many hlx flipped. My movies were recorded at 1e/A2 per frame, in case the dose it's too low to align individual segment, I'd like to try the average of 4 frames for the motion correction. Following the tutorial, I turn on the "frame motion correction" option, choose "frame average window size" as 4 and the "frame local average" as 565A, the same as the "image alignment size". The segmentrefine3d would ran well for many hours then the segmentrefind3d quit without any error message:
-------------------------------
INFO:CPU0:select_segments_based_on_specified_criteria_mpi:        in progress ...
    logged on Tue, 26 Jul 2016 17:46:24
INFO:CPU0:prepare_databases_for_selection:

INFO:CPU0:prepare_databases_for_selection:        in progress ...
    logged on Tue, 26 Jul 2016 17:46:24
INFO:wangji:/scicore/home/basler/wangji/src/spring/parts/openmpi/bin/mpirun -v --hostfile nodefile.dat -np 128 /scicore/home/basler/wangji/src/spring/bin/springenv /scicore/home/basler/wangji/src/spring/bin/python cleanup.py
    logged on Tue, 26 Jul 2016 17:48:40
----------------------------------

Finally I found something in the error output of the queuing system:
---------------------------------------------------
Traceback (most recent call last):
File "/scicore/home/basler/wangji/src/spring/bin/segmentrefine3d_mpi", line 152, in <module>
    sys.exit(spring.segment3d.refine.sr3d_mpi.main())
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 1304, in main
    segment_stack.perform_iterative_projection_matching_and_3d_reconstruction()
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 1262, in perform_iterative_projection_matching_and_3d_reconstruction
    unbending_info, translation_step, ref_cycle_id, each_info, pixelinfo, reference_files)
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 661, in select_segments_based_on_specified_criteria_mpi
    self.perform_local_frame_averaging_and_ref_database_update_mpi(ref_cycle_id)
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment3d/refine/sr3d_mpi.py", line 615, in perform_local_frame_averaging_and_ref_database_update_mpi
    stack_ids, x_shifts, y_shifts = self.perform_local_averaging_across_frames(session, ref_session, helix_ids)
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment3d/refine/sr3d_select.py", line 1114, in perform_local_averaging_across_frames
    stack_ids = np.vstack(stack_ids)
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/numpy/core/shape_base.py", line 228, in vstack
    stack_ids = np.vstack(stack_ids)
File "/scicore/home/basler/wangji/src/spring/lib/python2.7/site-packages/numpy/core/shape_base.py", line 228, in vstack
    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
ValueError: all the input array dimensions except for the concatenation axis must match exactly
--------------

It sounds like a python problem, but I'm wondering if you have any advice over this array dimension error? Is it related to the "frame local average distance" ? Can it be random number or it has to match the segment box size?

Thanks a lot in advance,

Jing

Carsten Sachse

unread,

Jul 27, 2016, 3:24:13 AM7/27/16

to emspring

Hi Jing,

On Tuesday, July 26, 2016 at 7:59:43 PM UTC+2, cyanpku wrote:

Hi Carsten,

I'm stuck by the frame motion correction during the segmentrefind3d... With that option "False", the alignment with the movie stack segments runs well, except the polarity of many hlx flipped.

Could you provide the relevant parameters in a list to see which 'False' option you are referring to?

Frame motion correction = True

Frame average window size = 4

Frame local averaging distance = 565

Could you use higher local averaging distances? I suspect that you will have a large number of e.g. in-plane rejections that there are no values to average anymore. As an alternative do you complete the run if you set the 'Frame local averaging distance' to 0?

Best wishes,

Carsten

cyanpku

unread,

Jul 27, 2016, 3:56:48 AM7/27/16

to emspring

Hi Carsten,

Thanks for your advice! I'll try either increase the local average distance or set it to 0 and will let you know which works better.

Cheers

Jing

Below is my segmentrefine3d.par. I meant turning the "frame motion correction" False seems to allow the program to run until complete.

------------------------------------------------------

Image input stack refinement = /scicore/pfs/wangji/titan_2016June/gfp/segment_25_Jul_2016_20_16_53_65760/stack_4_recon.hdf

Output volume name = vol.hdf

Diagnostic plot prefix = diag.pdf

Number of iterations = 4

Reference structure option = True

Reference volume = /scicore/pfs/wangji/titan_2016June/gfp/segmentrefine3d_20_Jul_2016_22_39_22_115868/vol_2078apix_012.hdf

spring.db file = /scicore/pfs/wangji/titan_2016June/gfp/segment_25_Jul_2016_20_16_53_65760/spring_0-11frames.db

Continue refinement option = True

refinement.db file = /scicore/pfs/wangji/titan_2016June/gfp/segment_25_Jul_2016_20_16_53_65760/refinement_0-11frames.db

Assemble refinement strategy = True

LR - Low resolution aim = False

LR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)

LR - X and Y translation range in Angstrom = (70, 38)

MR - Medium resolution aim = False

MR - azimuthal and out-of-plane search restraint in degrees = (180.0, 180.0)

MR - X and Y translation range in Angstrom = (40, 18)

HR - High resolution aim = False

HR - azimuthal and out-of-plane search restraint in degrees = (15.0, 15.0)

HR - X and Y translation range in Angstrom = (12, 8)

MaxR - Maximum resolution aim = True

MaxR - azimuthal and out-of-plane search restraint in degrees = (2.0, 2.0)

MaxR - X and Y translation range in Angstrom = (5, 3)

Absolute X and Y translation limit in Angstrom = (100, 40)

Frame motion correction = False

Frame average window size = 4

Frame local averaging distance = 565

Independent half-set refinement = False

Half-set refinement start = Max

High-pass filter option = False

Low-pass filter option = True

High and low-pass filter cutoffs in 1/Angstrom = (0.001, 0.09)

B-Factor = 0

Custom filter option = False

Custom-built filter file = filter_function.dat

Automatic FSC filter = True

Micrographs select option = False

Include or exclude micrographs = Include

Micrographs list = 1-9, 11, 13

Helices select option = False

Include or exclude helices = Include

Helices list = 1-9, 11, 13

Segments select option = False

Include or exclude segments = Include

Segment file = stackid_file.dat

Classes select option = False

Include or exclude classes = Exclude

Class type = Class_id

Classes list = 3

Persistence class option = False

Persistence class length in Angstrom = 1000

Class occupancy threshold = 0.5

Straightness select option = True

Include or exclude straight helices = Exclude

Persistence length range = (0, 20)

Layer line correlation select option = False

Include or exclude segments based on layer-line correlation = Include

Correlation layer line range = (60, 100)

Defocus select option = True

Include or exclude defocus range = Include

Defocus range = (8000, 30000)

Astigmatism select option = True

Include or exclude astigmatic segments = Include

Astigmatism range = (0, 1500)

Projection correlation select option = False

Include or exclude segments based on projection correlation = Include

Correlation projection range = (60, 100)

Out-of-plane tilt select option = True

Include or exclude out-of-plane tilted segments = Include

Out-of-plane tilt range = (-10, 10)

Shift normal to helix select option = False

Include or exclude segments with shift normal to helix = Include

Shift normal to helix in Angstrom = 30.0

Keep intermediate files = False

Estimated helix inner and outer diameter in Angstrom = (10, 220)

Pixel size in Angstrom = 1.039

Symmetrize helix = True

Helical rise/rotation or pitch/number of units per turn choice = Rise/rotation

Helical symmetry in Angstrom or degrees = (37.66, 23.52)

Rotational symmetry = 6

Helix polarity = Polar

Force helical continuity = True

Limit in-plane rotation = True

Delta in-plane rotation angle = 10.0

Out-of-plane tilt angle range = (-15, 15)

Number of projections azimuthal/out-of-plane angle = (90, 8)

Image alignment size in Angstrom = 376

Step size of segmentation in Angstrom = 38

Choose out-of-plane tilt amplitude correlation = False

Amplitude correlation out-of-plane tilt range = (-12, 12)

3D CTF correction = True

3D CTF correction intensity = Low

MPI option = True

Number of CPUs = 128

Temporary directory = /scicore/scratch/jing

--------------------------------------------------------------

cyanpku

unread,

Jul 28, 2016, 9:12:37 AM7/28/16

to emspring

Hi Carsten,

Segmentrefind3d still quit without error when I used a longer distance (1730A, a full helix repeat of my sample). Setting the "local average distance" to 0 worked, with much less polarity flipping segments but now I get a fsc that drops down and maintain at 0.2. Attached diagnose plots are for the reconstruction using individual frames with the "frame motion correction" option False (individual_frames_diag_stat_1039apix_015.pdf) and using the frames with "average window size" 4 (diag_stat_1039apix_015.pdf). Do you have any clue about what's messing up the fsc? Should I use the "independent half-set refinement" to get a normal healthy looking fsc curve? Or is there something else that I should change?

Thanks a lot,

Jing

diag_stat_1039apix_015.pdf

individual_frames_diag_stat_1039apix_015.pdf

Carsten Sachse

unread,

Jul 30, 2016, 2:15:36 AM7/30/16

to emspring

Hi Jing,

Thank you for sharing your results. It is not entirely clear where the messed up FSC comes from. Using helical symmetry, this can be observed if there is a large reuse of overlapping data. Maybe this is the issue here, when you turn on the averaging option that overlapping pieces of data are compared with one another in the FSC. If you check the independent FSC you should have less of an issue. To be honest, we have not used the frame correction option excessively due to the internal complexity and computational expense. It may still contain issues we did not sort out properly. In your attempt to increase resolution, I would also consider using classification to identify the best classes and exclusively reconstruct those.

Best wishes,

Carsten

cyanpku

unread,

Jul 31, 2016, 5:46:33 AM7/31/16

to emspring

Hi Carsten,
Thanks for your suggestions!
It's indeed taking quite a long time for the frame motion correction .... So far I've been using the 2d classification to exclude the worst class based on the diffraction pattern. It's a great idea to select only the best class instead. I'll try that and hopefully with less segment in total, I can attempt to play with the motion correction later on.
Cheers,
Jing

Reply all

Reply to author

Forward