Low resolution sub-tomogram ribosome averaging in M

[thomas.la...@gmail.com]

Hello fellow list mmembers,

I am working with some bacterial ribosomes from  lamellae collected at ~3.46 A/px, 4-5.5 um defocus, +/-60 in steps of 3, and total exposure of ~120 e/A^2.

From ~17k ribosomes across 9 tomos, Warp -> RELION -> ~20 A average (seems a reasonable estimate).

Since hope springs eternal, I am sending the particles and the 20 A average through M, but seem to get nothing but over-fit results to come out. 

I've tried both tight and generous binary masks in the species folder.

Also tried changing the default CTF high-pass filter to 35A rather than 20 A, without much luck.

Best result so far has been temporal sampling of 1, solving just for particle poses and a 1x1 image warp grid.

The input volume is on the top row of the attached image and output from M on the bottom. The strong peripheral density of the output looks like textbook over-fitting to me.

Has anyone else tried sending low-res ribosomes through M? with success?
If 'yes', any considerations that should be taken on refinement parameters or is this potentially a lost cause?

Before I forget, happy St. Patty's day!
-Tom

[teg...@gmail.com]

Hi Tom,

As a first sanity check, can you roll back everything to before M (old versions are stored in the 'versions' folder), and run one iteration with all options turned off to check if you get ca. the original 20 A map back? Starting with 20 A shouldn't be a problem in general.

Cheers,

Dimitry

[thomas.la...@gmail.com]

Thanks for the quick reply and suggestion!

Running the original species import without anything checked resulted in a "100A" blob.

Turns out, that I messed up when attempting to make my relion-v3.1+ .star to something "3ish", so the shifts were still in A and not pixels...😅

Re-importing  with specifying the appropriate units for the shifts did the trick.
3x3, particle poses, tilt-angles, 3x3x2x10 for a 3 sub-iterations and at a temporal sampling of 1 lead to 20 A -> 13 A 🥳

6A/px sampling rate probably not doing it justice, but this was what I was hoping for!
Now to up-sampling refine some more!

Thanks!

-Tom

[Lorenzo Gaifas]

Hey Thomas,

Glad you solved it! For future reference, dynamo2m's relion_star_downgrade might be useful for this purpose!

Best,

Lorenzo

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/55de7a78-07fc-4b16-9448-8c7ef771bb4fn%40googlegroups.com.

[Matthias Vorländer]

Hi there,

I am having a similiar problem and I am not sure I understand the conventions correctly. From a single tomogram I got a 38 angstrom reconstruction from 130 particles and want to see if I can improve it.That is what I have done so far:

Pixel size of the raw data is 1.35. I did tilt series pre-processing in warp, tilt series alignment in etomo (bin 4), reconstructed the tomogram in warp (10 A/pixel size), performed template matching and then reconstructed subtomos in warp with 5 ang/pix followed by 3D classification and refinement of selected class in relion. Then I downconverted the run_data.star file using dynamo2m. Now, in M I create a species and it reads in the star file seemingly correctly.

My question is: What do I specify during import in the "coordinates use A/px" and "shifts use A/px" fields? I thought I would need to set this to the pixel size of the subtomos (5 A/px). When I do this and refine I get a 100A blob and warp always crashes during the "finishing refinement" stage( divided by 0 error). When I leave it at 1, I get at 60 Angstrom map (However using a wider mask then the one I used in relion). Is this expected, or do I need to set the original pixel size of the tilt series (1.35)?

Thanks in advance,

Best,

Matthias

[Alister Burt]

Hi Matthias,

Your workflow looks right and your thinking about the pixel size was correct, it should be the pixel size of the subtomograms you did the refinement with (5A/px)

Things to check

- revert everything (.mrc.xml files in the versions folder of your Warp dir) after any bad run in M

- extract particles from your downgraded run_data.star file in Warp, do a reconstruction in RELION reconstruct and check that everything looks okay

Good luck!

Alister

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/2c26a271-af2f-41e2-8280-fce6d8206e1en%40googlegroups.com.

[Matthias Vorländer]

Hi Alister,

thanks a lot for these suggestions! Indeed, when I reconstruct after downgrading the starfile and then exporting subtomos again I only get a very noisy volume (when i heavily lowpass filter, I do see a blob that isn't in the center of the box). 

This is what the starfiles look like at each stage:

Original starfile that warp writes out after subtomo export:

data_

loop_

_rlnCoordinateX #1

_rlnCoordinateY #2

_rlnCoordinateZ #3

_rlnMicrographName #4

_rlnAutopickFigureOfMerit #5

_rlnMagnification #6

_rlnDetectorPixelSize #7

_rlnCtfMaxResolution #8

_rlnImageName #9

_rlnCtfImage #10

_rlnAngleRot #11

_rlnAngleTilt #12

_rlnAnglePsi #13

_rlnPixelSize #14

_rlnVoltage #15

_rlnSphericalAberration #16

   617.033   464.307   34.635  TS_01.mrc.tomostar  6.951328  10000.0  5.00100  5.1  ../subtomo/TS_01.mrc/TS_01.mrc_0000000_5.00A.mrc  ../subtomo/TS_01.mrc/TS_01.mrc_0000000_ctf_5.00A.mrc  0  0  0  5.00100  300.000  2.700

run_data star file from relion after refinement:

data_optics

loop_ 

_rlnOpticsGroup #1 

_rlnOpticsGroupName #2 

_rlnSphericalAberration #3 

_rlnVoltage #4 

_rlnImagePixelSize #5 

_rlnImageSize #6 

_rlnImageDimensionality #7 

           1 opticsGroup1     2.700000   300.000000     5.001000          180            3 

 

# version 30001

data_particles

loop_ 

_rlnCoordinateX #1 

_rlnCoordinateY #2 

_rlnCoordinateZ #3 

_rlnMicrographName #4 

_rlnAutopickFigureOfMerit #5 

_rlnCtfMaxResolution #6 

_rlnImageName #7 

_rlnCtfImage #8 

_rlnAngleRot #9 

_rlnAngleTilt #10 

_rlnAnglePsi #11 

_rlnPixelSize #12 

_rlnOpticsGroup #13 

_rlnGroupNumber #14 

_rlnOriginXAngst #15 

_rlnOriginYAngst #16 

_rlnOriginZAngst #17 

_rlnClassNumber #18 

_rlnNormCorrection #19 

_rlnLogLikeliContribution #20 

_rlnMaxValueProbDistribution #21 

_rlnNrOfSignificantSamples #22 

_rlnRandomSubset #23 

  297.730000    69.106000   147.197000 TS_01.mrc.tomostar    16.052830     5.100000 subtomo/TS_01.mrc/TS_01.mrc_0000285_5.00A.mrc subtomo/TS_01.mrc/TS_01.mrc_0000285_ctf_5.00A.mrc    44.375362     8.417129    -42.55984     5.001000            1            1     0.999756     0.999756     -0.87562            1     1.000000 2.339946e+07     0.014047           77            2 

downconverted star file:

# Created by the starfile Python package (version 0.4.4) at 10:48:45 on 22/03/2021

data_

loop_

_rlnMicrographName #1

_rlnCoordinateX #2

_rlnCoordinateY #3

_rlnCoordinateZ #4

_rlnAngleRot #5

_rlnAngleTilt #6

_rlnAnglePsi #7

TS_01.mrc.tomostar 297.929911 69.305911 147.021911 44.375362 8.417129 -42.559840

After reextracing subtomos in warp

data_

loop_

_rlnMicrographName #1

_rlnCoordinateX #2

_rlnCoordinateY #3

_rlnCoordinateZ #4

_rlnMagnification #5

_rlnDetectorPixelSize #6

_rlnCtfMaxResolution #7

_rlnImageName #8

_rlnCtfImage #9

_rlnAngleRot #10

_rlnAngleTilt #11

_rlnAnglePsi #12

_rlnPixelSize #13

_rlnVoltage #14

_rlnSphericalAberration #15

  TS_01.mrc.tomostar  248.275    57.755  122.518  10000.0  6.00000  5.1  ../subtomo/TS_01.mrc/TS_01.mrc_0000000_6.00A.mrc  ../subtomo/TS_01.mrc/TS_01.mrc_0000000_ctf_6.00A.mrc  0  0  0  6.00000  300.000  2.700

Note: I changed the pixel size when I re-extracted the subtomos to prevent warp from overwriting the original files. 

I noticed that the rotations get lost when warp reextracts the subtomos. Is this normal? I do tick the "pre-rotate particles" box in the warp dialogue. I am not sure where things are going wrong. Any input would be very much appreciated!

Many thanks in advance,

Best,

Matthias

[Alister Burt]

Hi Matthias,

There’s your issue!

If you use the 'pre-rotate particles’ option the Euler angles in your STAR file after alignment describe how to rotate a pre-rotated particle to align it to the reference. When you try to extract from the tomogram it needs to know how to rotate the particle in the tomogram to align it to the reference. See some discussion here.

Unfortunately the easiest way to fix this is to go back to your original particle poses, reextract your particles without the ‘pre-rotate’ option and redo your alignments.

Good luck!

Alister

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/568f0bee-71e6-46f3-9620-334c3aa57b97n%40googlegroups.com.

[Matthias Vorländer]

Hi Allister, 

Thanks a lot! Just to make sure that I understand correctly, this means that I should extract the original template matched subtomos in Warp, redo the classification and refinement in relion, and then things should work out?

Thanks for the kind help!

Best, Matthias 

[Alister Burt]

Hi Matthias,

Yep! Specifically, always extract the subtomos without the ‘pre-rotate particles’ option and you won’t run into this issue.

Cheers,

Alister

[xu-...@mails.tsinghua.edu.cn]

Hi all,

I used isolated ribosome tomograms to run Warp-Relion-M subtomo average. The pre-processing is proceseed by Warp. Relion refines a 7.4A map using about 12,000 subtomograms. However, M produced a 50A map. I uncheck every box and run 1 iteration using 100% particles to see if M will re-produce a similar map like Relion. However, M still produce a 50A map.

I used Alister's scripts to down-version the star file from Relion3.1 and check the units of shift is right. The XX_fsc.star produced by M looks strange because the correlation of 2 half maps is negative between 50A to 20A. sometimes, the correlation is -1. I attach the XX_fsc.star below.

Is there any clue to do the trouble-shooting? Really appreciate your advice.

Best,

Jialu

--------------------------------------------------------------------

data_

loop_

_wrpResolution #1

_wrpFSCUnmasked #2

_wrpFSCRandomized #3

_wrpFSCCorrected #4

_wrpFSCMasked #5

  Infinity              1              1              1              1

     601.6      0.9943152      0.9999955      0.9999962      0.9999962

     300.8      0.9981717       0.999985      0.9999865      0.9999865

  200.5333      0.9987051      0.9999562      0.9999584      0.9999584

     150.4      0.9965022      0.9998554      0.9998587      0.9998587

    120.32      0.9830461      0.9993832      0.9994031      0.9994031

  100.2667      0.9691125      0.9985872      0.9986564      0.9986564

  85.94286       0.960398       0.998244      0.9982107      0.9982107

      75.2       0.923426      0.9971949      0.9967543      0.9967543

  66.84445      0.8371596      0.9951468      0.9939438      0.9939438

     60.16      0.7283027      0.9902782       0.988903       0.988903

  54.69091      0.4963332      0.9808248       0.981636       0.981636

  50.13334       0.308367      0.9760745     -0.1899881      0.9715289

  46.27692      0.2324384      0.9709504     -0.5673173      0.9544701

  42.97143     0.09406988      0.9647993     -0.6492484      0.9419454

  40.10667     0.01786287      0.9554538     -0.5947073      0.9289619

      37.6    0.005603485      0.9522445     -0.7235698      0.9176901

  35.38824      0.0414065      0.9371525     -0.9449662      0.8777637

  33.42223     0.01443157      0.9134809      -1.053136      0.8223646

  31.66316     0.05133304      0.8911093     -0.9195514      0.7909788

     30.08     0.01301006      0.8412236      -0.873557      0.7025234

  28.64762     0.02175037      0.7996053     -0.8255575       0.634168

  27.34546     0.01004299      0.7466243      -0.770351      0.5514361

  26.15652   -0.005229754      0.6616352     -0.6510722      0.4413353

  25.06667    0.001195486      0.5952121     -0.3978781      0.4341559

    24.064     0.01380409      0.5373061     -0.2791049      0.4081659

  23.13846    0.002647326      0.5037481     -0.3446139      0.3327327

  22.28148    0.004194609      0.4655601     -0.3480181      0.2795654

  21.48572    0.004434231      0.3675427     -0.1062797      0.3003253

[Alister Burt]

Hi Jialu,

Could you try to update your dynamo2m install? ‘pip install —upgrade dynamo2m’

I think I was applying the inverse of the translation to the particle positions (caught by Linhua Tai), pushed a fix yesterday :)

Make sure to reset your tilt series xml files to the pre-M versions so you’re starting from a ‘good’ state, not a weird local minimum from the failed run of M.

Good luck!

Alister

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/800b090e-7098-4964-9666-0b49baf74483n%40googlegroups.com.

[Alister Burt]

Also - your masked FSC never drops below 0.143 in the portion of the STAR file shown - only the solvent corrected FSC  does which indicates there might be some problems with your mask too - did you supply a binary mask?

Cheers,

Alister

On 8 May 2021, at 13:58, xu-...@mails.tsinghua.edu.cn <xu-...@mails.tsinghua.edu.cn> wrote:

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/800b090e-7098-4964-9666-0b49baf74483n%40googlegroups.com.

[xu-...@mails.tsinghua.edu.cn]

Thank you Alister,

I will update the dynamo2m and re-try it. The mask is produced by M and it is binary mask. Sorry uploading the fsc.star failed so I just attach some part of it to avoid the post is too long. The fsc is below 0.143 at resolution of 7.42, which is agree with the initial resolution calculation. 

Best,

Jialu

------------------------------------------------------------

data_

loop_

_wrpResolution #1

_wrpFSCUnmasked #2

_wrpFSCRandomized #3

_wrpFSCCorrected #4

_wrpFSCMasked #5

  9.549207    0.005892009   -0.008834957     0.06556683     0.05731115

  9.400001    0.003521391  -0.0008626399       0.051301     0.05048261

  9.255385  -0.0008592578   -0.000882139     0.05619914     0.05536658

  9.115152     -0.0033814   -0.006894249     0.06343257     0.05697564

  8.979105    0.003706617   -0.003891442     0.05941412     0.05575389

  8.847059    0.002114395     0.01138176     0.04782028     0.05865777

  8.718842    0.004605231     0.02984608     0.02714639     0.05618225

  8.594286    0.007310051     0.03604798    0.004850686     0.04072381

   8.47324    0.009354907     0.02141975     0.00687904     0.02815144

  8.355556    0.007124708    0.009642054     0.01312597     0.02264146

  8.241096     0.00111503     0.01130918     0.01245979     0.02362807

   8.12973    0.005169528    0.008652166     0.02288256     0.03133674

  8.021334    0.009657587    -0.01483861       0.049108     0.03499809

   7.91579     0.01114292    -0.03728709     0.06353269     0.02861455

  7.812987   0.0004334226    -0.04009082     0.06251358     0.02492899

  7.712821   -0.001635364    -0.02693613     0.05884594      0.0334949

  7.615191     0.00610011   -0.009358552      0.0424704      0.0335093

      7.52     0.01067502    0.002748551      0.0126279     0.01534174

  7.427161    0.001553749     0.00545337    0.001248693    0.006695253

 7.336586   7.172808E-05     0.00117982     0.01623025     0.01739092

  7.248193    0.001946011   -0.004694068     0.03170177     0.02715651

  7.161905    0.003438541   -0.003161411     0.02630834      0.0232301

  7.077648    0.001283173  -8.089687E-05     0.01546517     0.01538552

  6.995349     0.00481265   -0.007321243      0.0150252    0.007813958

  6.914943    0.006894507    -0.01457009     0.01692339    0.002599883

  6.836364    0.002734036    -0.01483152     0.01517456   0.0005680961

  6.759551    0.002310931    -0.01602387     0.01370477   -0.002099492

  6.684445     0.00315314    -0.01593411    0.007480766   -0.008334146

[Reika Watanabe]

Dear all,

Using my cellular FIB-ET tomogram (2.7A/pix, defocus -5 micron, total dose 120e/A2, K3 full, ten frames/tilt), through Warp, Dynamo, Relion pipeline, I have obtained 20A resolution structure (3A/pix, 112x112x112 voxel, roughly 320 A diameter). The final particle number to be used for M is only 3000 after two rounds of Relion 3d classification.  

After 1st M (image 4x3, volume 2x 3x2x10), particle poses, stage angles checked, I think I get a moderate improvement of the map (although M reports resolution 20A, similar to Relion). Whenever I continued to perform the more elaborate M processing as suggested in the #TEAMTOMO tutorial, the resolution went down (worse than 30A).

After bad M, I replaced the tilt series xml file found in the version folder for a fresh restart. (I went back to the original frame and subtomogram data saved in the frame and subtomogram folders just in case).

Am I doing something wrong during the operation because I could not see the clear resolution improvement by M? 

Or is it normal that we cannot expect improvement by M due to small particle numbers? Or just the nature of molecules (very diverse molecules?)

I plan to repeat the whole warp/M processing from scratch; what are the critical parameters to pay attention to (CTF estimation, image/volume parameter, mask, et al.)? Any input will greatly help me on this!

I am essentially following the tutorial shown in Gotta Warp 'M all, #TEAMTOMO and using dynamo2M scripts (Thanks to everyone who contributes this great effort). 

Thank you very much in advance!

Reika 

[Reika Watanabe]

I am analyzing long filamentous structures but not ribosomes. 

I forgot to specify in my previous email and it might influence alignment by M?

Have a great weekend to all!

Reika 

REIKA WATANABE CASTILLON Ph.D.
La Jolla Institute for Immunology 
9420 Athena Circle La Jolla, CA 92037 
http://www.lji.org

You received this message because you are subscribed to a topic in the Google Groups "Warp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp-em/ZMr-uW8bBzU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/90f63f6a-293c-4cda-abde-0d5336fdab60n%40googlegroups.com.

[Pranav Shah]

Hi Reika,
It has hard to provide concrete advice without knowing the precise
nature of the problem.
Does your complex have any symmetry that you could leverage? As you
rightly pointed out, your complex may be too flexible or you may have
too few (3000) particles. I have not managed to do this yet, but you
could get a sense for how flexible your particles are using some form
of PCA based classification (3D classification in relion can be a bit
janky) to get more homogeneous classes (this could potentially lead
you down a path where you have self-consistent classes but very small
populations) and try the M routine again. Alternatively, you could
also try picking and refining ribosomes in your dataset which should
be scattered around in your fields of views that could serve as
fiducials and help in image and volume warp based estimations.

HTH.
Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

> To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/FD282D77-FEE2-4ECA-9F49-B87B7D741381%40gmail.com.

[Reika Watanabe]

Hi Pranav,

Thank you very much for your prompt and insightful suggestions.

The structure of interest is helical structures, and I know the approximate value of rise and twist.
I have never applied any helical symmetry during subtomogram processing. Please let me know if you know of any suitable software package that allows us to do so. Relion4 will do, but I am still not starting it yet. Maybe I should!

Concerning ribosome suggestion, indeed, I will start ribosome averaging found in my data set.
Ribosomes and the structure of interests do not occupy the same area in the tomogram. Therefore, it might not serve as fiducial; however, I can verify data quality and proper processing pipeline.

Have a great day, and thanks again for your help!
With best wishes,
Reika

[Pranav Shah]

Oooh! I have never done helical symmetry so I have no idea what will
work, but helical symmetry is powerful and you stand to gain a lot
more if you could figure that bit out... I think M has a provision for
helical symmetry, but this is a question for Dr. D to answer...

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

[teg...@gmail.com]

Hi Reika,

Sorry, helical symmetry support is still very rudimentary. You'll need to know the parameters well since M doesn't refine them. You may want to switch between M and RELION a few times to establish the exact parameters. I'll copy here what I wrote to someone using helical symmetry:

"Please copy these binaries into Warp's installation folder (previously updated with the latest 1.0.9 nightly build): https://www.dropbox.com/s/cs57yslxmihvfdb/WarpHelix_20210416_2.zip?dl=0

In your .species file, update the symmetry value from "C1" to "Hel|-27.17|4.21". You can also add "<Param Name="HelixZFraction" Value="0.3" />" if you want to use a fraction of the Z axis other than the default 30% (this is equivalent to Relion's "Central Z length").

If you create a new species instead of removing redundant particles from an existing one, you should be able to select the new symmetry type in the UI. However, the Z fraction parameter isn't available there."

Cheers,

Dimitry