Spring tutorial

[Johannes Haataja]

Hi,
    about the spring tutorial http://www.sachse.embl.de/emspring/_images/tutorial.pdf
which is a nice documentation, where emphasis has also been put to explain the theoretical/practical points. Nevertheless I do find the tutorial to be time to time too implicit rather than explicit for fluent progression.

For instance in step 4. "Segmentation of TMV particles" it is stated that

Stack for classification
For the first, select phase-flip as a CTF correction and the perturb step option (the reason why we need to use this option will become apparent during the classification).

Stack for reconstruction
Tick the CTF correct option and select CTFTILT , and then Convolve in the next drop-down menu. Tick also the Astigmatism correction , and the Invert, Normalization, Row normalization, Centering and Rotation options. Since the Unbending will introduce interpolation and thereby resolution loss, we will not use it at this stage and leave the option un-ticked

To what does "For the first" (line 2) refer to? To the first stack (classification) - does one need to run the stack generation twice and if so how should these output files be passed as inputs in the next steps?

I assume that

"select phase-flip as a CTF correction"

equates  (labels in the GUI have changed since the tutorial was made?)
* Select "Intermediate" for "Additional parameters (level)",
* check "CTF correct option"
* for "CTFFIND or CTFTILT" select "Ctffind"(?)
* for "convolve or phase-flip" select "Phase-flip"

+

one should probably specify(?)
    * the <output.hdf> for  "Image output stack"
    * some folder for "Temporary directory"
    * check "Spring database option"
    * specify input for "spring.db file"

Should one then proceed ( <=> press "OK") before moving on the "stack for reconstructions"?

- J.

I do apologize if I sound like someone looking for idiot proof  instructions =S

[Carsten Sachse]

Dear Johannes,

On Wednesday, March 2, 2016 at 3:49:46 PM UTC+1, J. Haataja wrote:

Hi,
    about the spring tutorial http://www.sachse.embl.de/emspring/_images/tutorial.pdf
which is a nice documentation, where emphasis has also been put to explain the theoretical/practical points. Nevertheless I do find the tutorial to be time to time too implicit rather than explicit for fluent progression.

For instance in step 4. "Segmentation of TMV particles" it is stated that

Stack for classification
For the first, select phase-flip as a CTF correction and the perturb step option (the reason why we need to use this option will become apparent during the classification). 

Stack for reconstruction
Tick the CTF correct option and select CTFTILT , and then Convolve in the next drop-down menu. Tick also the Astigmatism correction , and the Invert, Normalization, Row normalization, Centering and Rotation options. Since the Unbending will introduce interpolation and thereby resolution loss, we will not use it at this stage and leave the option un-ticked

To what does "For the first" (line 2) refer to? To the first stack (classification) - does one need to run the stack generation twice and if so how should these output files be passed as inputs in the next steps?

Indeed, you will run SEGMENT twice and as a result generate two stacks: one for the classification and second for the 3D structure refinement. I realize that this step is not obvious to many users. I will name the advantages:

The classification stack:

Here, we are interested in generating a high signal image with lots of 2-D features for visual analysis in real and Fourier space. First, to restrain the alignment, we pre-rotate the segments according to the picked coordinates. Second, to maximize the low-resolution signal in the images, we flip the phases as a means of CTF correction. Third, we perturb the step size as the regular step size will introduce artificial repeats into the classes visible by overemphasized streaks in Fourier analysis. 

The 3-D structure refinement stack:

This stack is really meant for maximizing the resolution of in the 3-D structure refinement. Here, first we keep the images unrotated in order to avoid multiple interpolation steps when preparing them for the 3-D reconstruction. Second, we convolve the images by the CTF as a means of CTF correction as this imposes a more correct amplitude correction, followed by the 3-D CTF^2 deconvolution (for more details check our Sachse et al. 2007 paper). Third, we will keep the step size constant as a way to take advantage of every asymmetric unit in the 3-D reconstruction.

I assume that

"select phase-flip as a CTF correction"

equates  (labels in the GUI have changed since the tutorial was made?)
* Select "Intermediate" for "Additional parameters (level)",

The levels are an attempt to reduce complexity of parameters. In most cases, the beginner parameters are really essential to provide. If intermediate parameters are not provided defaults will be filled that make sense for the simplest operations but require modification for more advanced tasks. You are already performing more advanced image processing operations!

* check "CTF correct option"
* for "CTFFIND or CTFTILT" select "Ctffind"(?)
* for "convolve or phase-flip" select "Phase-flip"

Yes, this is correct. 

+

one should probably specify(?)
    * the <output.hdf> for  "Image output stack"
    * some folder for "Temporary directory"
    * check "Spring database option"
    * specify input for "spring.db file"

Correct. 

Should one then proceed ( <=> press "OK") before moving on the "stack for reconstructions"?

OK is correct. OK launches the process. I hope this helps. All of these steps will be repetitive once you have learned how to work with the SPRING GUI. That is why we left them out. The tutorial aims to focus on the conceptual issues of the processing rather than on technical ones. Thanks for the feedback.

Best wishes,

Carsten 

[J. Haataja]

Dear Carsten,
     Thank you for the clarification. I got bit further with tutorial. I didn't actually realize, that all the default parameters for each step were listed in the Appendix. Unfortunately I did run into trouble in step 6. Classification. Apparently the program fails to create averages.hdf output:

INFO:username:/home/username/programs/spring_v0-84-1470/parts/openmpi/bin/mpirun -np 5 /home/username/programs/spring_v0-84-1470/bin/segmentalign2d_mpi --f parfile_7524.par --d segmentalign2d01
logged on Tue, 08 Mar 2016 16:23:18
INFO:CPU0:spring_align2d:progress state: 56 %  [======================>                  ]
INFO:CPU0:sx_kmeans:

INFO:username:/home/username/programs/spring_v0-84-1470/parts/openmpi/bin/mpirun -np 5 /home/username/programs/spring_v0-84-1470/bin/springenv /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py segmentalign2d01/tmv_4class2binnedali.hdf sxk_means02 rectmask.hdf --K=10 --maxit=500 --rand_seed=-1 --crit=D --MPI
logged on Tue, 08 Mar 2016 16:23:43
INFO:CPU0:sx_kmeans:springenv /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py segmentalign2d01/tmv_4class2binnedali.hdf sxk_means02 rectmask.hdf --K=10 --maxit=500 --rand_seed=-1 --crit=D --MPI
logged on Tue, 08 Mar 2016 16:23:49
Traceback (most recent call last):
File "/home/username/programs/spring_v0-84-1470/bin/segmentclass", line 152, in <module>
sys.exit(spring.segment2d.segmentclass.main())
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 732, in main
stack.classify()
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 706, in classify
avgstack, varstack = self.sx_kmeans(aligned_stack, self.maskfile, self.noclasses)
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 451, in sx_kmeans
external_kmeans_run.check_expected_output_file(program_to_be_launched, avgstack)
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/csinfrastr/csproductivity.py", line 543, in check_expected_output_file
raise IOError(error_message)
IOError: /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py did not finish successfully. The output file sxk_means02/averages.hdf was not found, please check logfile of /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py for details.

Perhaps this error is related(?) to the fact that I skipped reference image option

Reference image option                   = False
Image reference stack                    = ref720pix.hdf

and custom filter option

Custom filter option                     = False
Custom-built filter file                 = filter_function.dat

since I don't known when/where such files are generated/located... =S

 - J.

[Carsten Sachse]

Dear Johannes,

On Tuesday, March 8, 2016 at 3:28:35 PM UTC+1, J. Haataja wrote:

Dear Carsten,
     Thank you for the clarification. I got bit further with tutorial. I didn't actually realize, that all the default parameters for each step were listed in the Appendix. Unfortunately I did run into trouble in step 6. Classification. Apparently the program fails to create averages.hdf output:

INFO:username:/home/username/programs/spring_v0-84-1470/parts/openmpi/bin/mpirun -np 5 /home/username/programs/spring_v0-84-1470/bin/segmentalign2d_mpi --f parfile_7524.par --d segmentalign2d01
logged on Tue, 08 Mar 2016 16:23:18
INFO:CPU0:spring_align2d:progress state: 56 %  [======================>                  ]
INFO:CPU0:sx_kmeans:

INFO:username:/home/username/programs/spring_v0-84-1470/parts/openmpi/bin/mpirun -np 5 /home/username/programs/spring_v0-84-1470/bin/springenv /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py segmentalign2d01/tmv_4class2binnedali.hdf sxk_means02 rectmask.hdf --K=10 --maxit=500 --rand_seed=-1 --crit=D --MPI
logged on Tue, 08 Mar 2016 16:23:43
INFO:CPU0:sx_kmeans:springenv /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py segmentalign2d01/tmv_4class2binnedali.hdf sxk_means02 rectmask.hdf --K=10 --maxit=500 --rand_seed=-1 --crit=D --MPI
logged on Tue, 08 Mar 2016 16:23:49
Traceback (most recent call last):
File "/home/username/programs/spring_v0-84-1470/bin/segmentclass", line 152, in <module>
sys.exit(spring.segment2d.segmentclass.main())
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 732, in main
stack.classify()
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 706, in classify
avgstack, varstack = self.sx_kmeans(aligned_stack, self.maskfile, self.noclasses)
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/segment2d/segmentclass.py", line 451, in sx_kmeans
external_kmeans_run.check_expected_output_file(program_to_be_launched, avgstack)
File "/home/username/programs/spring_v0-84-1470/lib/python2.7/site-packages/emspring-0.84.1470-py2.7.egg/spring/csinfrastr/csproductivity.py", line 543, in check_expected_output_file
raise IOError(error_message)
IOError: /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py did not finish successfully. The output file sxk_means02/averages.hdf was not found, please check logfile of /home/username/programs/spring_v0-84-1470/parts/EMAN2/bin/sxk_means.py for details.

  For explanation, SEGMENTCLASS cycles through SPARX's  k-means clustering and subsequent alignment of the segments to those clusters. In this case, the error message is fairly clear: double check the log file within the sxk_means02 directory. I suspect K-means clustering may have crashed due to some memory problem. Therefore, I recommend use binned data whenever possible. The size of your segment stack will also affect the memory usage.

Perhaps this error is related(?) to the fact that I skipped reference image option

Reference image option                   = False
Image reference stack                    = ref720pix.hdf

and custom filter option

Custom filter option                     = False
Custom-built filter file                 = filter_function.dat

since I don't known when/where such files are generated/located... =S

 The custom filter option is an advanced option that allows you to specify the filter coefficients of your choice in the provided table file. Which would not be related to the error message at all.