Request: use tilt instead of deltlt in the taSolutions log for aligned tilt series import

[Benjamin Barad]

Hi Dimitry,

I am doing warp on tilt series collected on lamellae collected dose symmetrically with a pretilt of -11º (due to the milling angle, -11 is the "flat" path through the sample). Because of the relatively large pretilt, I do all my processing in imod with a +11º tilt angle offset in order to get better tilt series alignments - the effect is surprisingly significant for getting better quality output. However, when I import the aligned tilt series into warp, the tilt angle offset is lost and the resulting tomogram quality is relatively poor. Beyond the quality issue, I need a much thicker tomogram to be able to fit the lamella, and that empty space seems to make denoising perform much worse.

As far as I can tell, the math for this happens here: https://github.com/cramerlab/warp/blob/21e804c6b362154dae6177f381e48198e599df2c/Warp/Controls/TaskDialogs/Tomo/DialogTomoImportImod.xaml.cs#L317. It reads the deltlt column (which does not have the tilt axis offset incorporated), and applies it to the original angles. I've been able to get flat and high quality tomograms out of warp by writing a script to add 11 to each deltlt angle, but that feels like a very hacky solution that could be done better. 

Could that line instead read the tilt angle (instead of deltlt) and just write it in instead of adding on? The tilt angle appears to incorporate the tilt angle offset, but has precision only to the 10s digit. Alternatively, perhaps the Angle offset could be read from one of the numerous logs that it is written to (just not taSolutions!).

Thanks,

Ben

P.S. I will also open an issue for this on github, but the listserv seems like the more active source of discussion recently.

Benjamin Barad, PhD

Postdoctoral Associate

Grotjahn Lab @ Scripps Research

He/His/Him

[Benjamin Barad]

Correction: in the second paragraph, I meant tilt angle offset, not tilt axis offset.

[teg...@gmail.com]

Hi Ben,

What would be the most robust way to get it from IMOD? I'm not the biggest fan of its metadata chaos and try to limit my reliance on its more advanced features. Also, doesn't pre-tilt correction also include a rotation around X? That isn't part of the .tomostar model right now.

Are tilt angle corrections from reference-free alignment relevant beyond the first decimal? I usually don't solve for them at all.

Cheers,

Dimitry

[Benjamin Barad]

Hi Dimitry,

Doing full pre-tilt correction using the tomogram positioning module in etomo would involve measuring a z-shift and an x-axis offset, but for lamellae the dominant feature by far is the tilt angle offset, and because we know the angle we mill at it is predictable to within maybe 0.1 or 0.2º (probably depends on the cryostage used for milling). As a result, I just do all of my processing, beginning with coarse alignment, using a tilt angle offset. I find that if I use that pre-defined tilt angle offset (for my relatively large cells, where I don't need to mill super flat, it is almost always 11º), any improvement from using the tomogram positioning routines in imod is minimal.  I imagine for most users who work on lamellae, incorporating just the tilt angle offset will be more than enough to get to a high enough quality that segmentation, denoising, particle picking, and eventually tilt series refinement will all work much better.

For my internal use, I have been grabbing the AngleOffset from `align.com` - it is relatively straightforward to parse and will work whether you input the angle offset from the start or solve for it with tomogram positioning and generate a "final alignment".

As far as the precision of the tilt axis angles - my feeling is that no, that level of precision is totally unnecessary - I would estimate the "true precision" is at best 0.1º for all but the very best of my data, and I plan to refine the tilt series geometry anyway!

Thanks,

Ben

Benjamin Barad, PhD

Postdoctoral Associate

Grotjahn Lab @ Scripps Research

He/His/Him

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/d93522dd-5d2a-45ed-b082-b0e92a621644n%40googlegroups.com.

[Wim Hagen]

Hi Ben,

My 5 cents:

When pre-tilt from milling is somewhat known, I’d argue that you want to start collecting dose-symmetric tilt series at that pre-tilt, SerialEM and TFS Tomo allow one to do that. It would prevent some unnecessary pixel interpolations needed to “tilt things during processing”, especially for the low tilts that bring most of the high-res info, 

Best,

Wim Hagen

EMBL Heidelberg

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/CAPEz%3DzqQpMHpOUsynZwZMNjVbJ5G_DoOBWbzVnm56Wg1%2B0ZuLQ%40mail.gmail.com.

[Benjamin Barad]

Hi Wim,

You are correct - I do collect dose symmetric with pretilt starting at -11º. However, serialEM reports that new "0" angle as -11 still, and since the nominal 0º angle is so far from the true zero of the lamella, I end up with non-optimal tilt series alignments and other issues I described with denoising etc. above.

I could adjust the mdocs themselves potentially (or maybe there is a best way to change settings within SerialEM), but I appreciate having the record of exactly how I collected the data (including true tilts).

Benjamin Barad, PhD

Postdoctoral Associate

Grotjahn Lab @ Scripps Research

He/His/Him

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/C6B3E802-04E9-4D52-90D2-D40F23089272%40gmail.com.

[Wim Hagen]

Hi Ben,

Right, I didn’t realize that, thanks! I’d then say this definitely is something to bring up with David Mastronarde: a pre-tilt parameter in Etomo to account for the pre-tilt from the very start?

Best,

Wim

On 22. Sep 2021, at 19:37, Benjamin Barad <benjami...@gmail.com> wrote:



[Benjamin Barad]

I'd love to have that in place - right now I enter it at 3 steps (coarse alignment, patch tracking, and fine alignment) but reducing that to one would be great - especially if it made it easier to extract the information into warp. I'll send in a feature request on the serialEM mailing list. 

I do think that `align.com` is a good place to get the information for Warp import, though, since it will work whether you have that pretilt in place for the start or you solved for it with tomogram positioning. A simpler but less robust alternative would be to read a pretilt from the tilt series mdoc file if serialEM wrote it out at the top - that fail in cases of user error (e.g., I might have forgotten to change the pretilt from 0º because I was a little short on sleep setting up tilt series!).

[Pranav Shah]

This is an interesting thread and I had meant to reply to it earlier
Ben. I have calculated subtomogram averages out to physical Nyquist
(4.0A) from TS's that were not corrected for the pre-tilt (-13˚). Not
correcting for it didnt hamper neural network based particle picking
either. Having said that, I can see how it would affect segmentations
and it would be really good to have "flat" tomos that have been
centered. You are right to suggest using the aling.log file and David
"Wiz" MAstronarde has already written a specific tool called alignlog
(https://bio3d.colorado.edu/imod/betaDoc/man/alignlog.html) for
grabbing different kinds of information from the align.log file.
Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

> To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/CAPEz%3Dzo1zA8eGRb2ZQ0JQb-gijw2kFySSo1O9P73wQLtKjsaTg%40mail.gmail.com.

[Digvijay Singh]

hi Ben and others, 

I have not perused all the points of the thread, here's how I deal with PreTilt angle. 

[1] By parsing mDoc file, the 'DoseRate' captured on Camera (MeanCounts/ExposureTime) is calculated for each tilt-image. 

[2] The tilt angle at which the captured rate is the highest is the PreTilt angle, with an accuracy of ~1° if your tilt-step is 2°.

[3] Offset/adjust all tilt-angles in the mDoc file or the .tlt file by the PreTilt calculated in [2]. 

[4] Use adjusted mDoc file in WARP when importing tilt-series. The resultant tomograms will be nearly flat and save any troubles coming from TomoPitch in IMOD etc. Also helps with better/faster processing on tomograms ( as you can truncate/restrict Z more)

Below is a Python script, that you can run in folder containing mDocs. It will generate a subfolder called Adjusted containing adjusted mdocs that can be input in WARP. You & others may find this useful

#!/usr/bin/env python

# -*- coding: utf-8 -*-

# By Digvijay

import sys

import re

import numpy as np

import glob, os

    

def ConvertMDocFiles(MRCpath,mdocFileName):

 os.chdir(MRCpath) # Go to the folder with the identified .mrc.mdoc file

 if not os.path.exists(MRCpath+"/Adjusted"):

   os.makedirs(MRCpath+"/Adjusted")

 AdjustedMRCPath=MRCpath+"/Adjusted/"

 ExposureTimeList=[]

 CountsPerElectronList=[]

 MeanIntensityList=[]

 MaxIntensityList=[]

 TiltAngleList=[]

 with open(mdocFileName, "r") as f:

  for line in f:

   match1=re.search('ExposureTime = ?(\d*.\d*)',line)

   match2=re.search('TiltAngle = ?(-?\d*.\d*)',line)

   match5=re.search('CountsPerElectron = ?(\d*.\d*)',line)

   match6=re.search('MinMaxMean = ?(\d*.\d*)\s(\d*.\d*)\s(\d*.\d*)',line)

   if match2:

    TiltAngleList.append(float(match2.group(1)))

   if match1:

    ExposureTimeList.append(float(match1.group(1)))

   if match5:

    CountsPerElectronList.append(float(match5.group(1)))

   if match6:

    MeanIntensityList.append(float(match6.group(3)))

    MaxIntensityList.append(float(match6.group(2)))

 f.close()

 DoseRateOnCameraList=[aa/(bb*cc) for aa,bb,cc in zip(MeanIntensityList, CountsPerElectronList,ExposureTimeList)]

 DoseRateAtPreTiltZero_EPS=max(DoseRateOnCameraList)

 PreTiltZeroAngle= TiltAngleList[DoseRateOnCameraList.index(max(DoseRateOnCameraList))]

 f.close()

 

 input_file = open(mdocFileName, "r")

 output_file = open(AdjustedMRCPath+os.path.basename(mdocFileName), "w")

 for line in input_file:

  match2=re.search('TiltAngle = ?(-?\d*.\d*)',line)

  match3=re.search('bidir = ?(-?\d*.\d*)',line)

  if match2:

   NewTiltAngle=float(match2.group(1))-PreTiltZeroAngle

   line = re.sub('TiltAngle = ?(-?\d*.\d*)', 'TiltAngle = '+str(NewTiltAngle), line)

  if match3:

   NewBiDirAngle=float(match3.group(1))-PreTiltZeroAngle

   line = re.sub('bidir = ?(-?\d*.\d*)', 'bidir = '+str(NewBiDirAngle), line)

  output_file.write(line)

 output_file.close()

Path=os.getcwd()

for mdocFileName in glob.glob("*.mdoc"):

 ConvertMDocFiles(Path,mdocFileName)

[Benjamin Barad]

Thanks Digvijay - I am very leery about modifying my mdocs directly - I see them as part of the raw data, and in a couple cases it is quite important to me to be able to return to the absolute geometry relative to the growth plane - but I am glad to see I am not the only one thinking about flattening tomograms, and it seems like for many cases this is indeed the simplest solution (until I convince Dimitry to read it from the imod alignments, or let me make a pull request doing as much!).

Benjamin Barad, PhD

Postdoctoral Associate

Grotjahn Lab @ Scripps Research

He/His/Him

P.S. Are you putting your code up on github? If so, I'd love to follow you there - I have learned a lot by following Pranav and Alister Burt, as well as others. It would also make it easier to cite your work if I end up using your code in a paper. I've been uploading my code mostly to the GrotjahnLab github organization - happy to help you set one up for Villa lab if that is your preference! It is free for academic labs.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/5512db61-caa0-42c1-80ac-d580d79662c5n%40googlegroups.com.

[Digvijay Singh]

Hi Ben,

Sorry for my late reply. Parenthood has been busy :) 

Adjusting the tilt angle should not cause any issues. And you are right that the original mDocs is best not altered. That is why, the code snippet I pasted generates a new set of adjusted mDocs (while retaining the original) that can be used in WARP/Relion pipeline. I have been using these adjusted mDocs and getting all the subtomo pipelines to work fine. 

Sure, I will make my Github profile public and we can follow each other. I do regularly check out the content in the profile of other wonderful members of the community like Alister, Pranav, and others. 

Cheers,

Digvijay

[Pranav Shah]

Thought I'd reopen this thread and leverage the hive mind...

I finally have managed to get Noise2Map to work with my data (IDK why it took me so long to get to it) and I would like to use these denoised  data for segmentating a few tomograms. However, the problem is that these tomograms have the pre-tilt on them which makes visualisations kind of unappealing. Now I can solve my problem the hard way or the easy way.

Easy way - Since there are only a few tomograms, I could create a new mini-project dir which contains the frames for the tomogram I wish to segment, use Digvijays script to modify the starting angle in the mdoc file and then process everything from scratch until I have the tomograms for segmentation. OR.

Hard way - Go through contortions to edit the xml file for the tomograms I wish to reconstruct so that when Warp reconstructs the tomograms, they come out flat.

I am unable to decide the best way to move forward because in Scenario 1, I am having to repeat all the pre-processing steps, which I am not sure is a good use of time, however with Scenario 2, wont changing the pre-tilt post-alignment have an impact on the alignements and shifts themselves?

[Pranav Shah]

Huzzah! A third way has emerged! I like the look of the affine
transforms especially the fact that I can trivially rotate around Z...
will give that a go tomorrow... thanks man!

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

On Wed, Nov 17, 2021 at 12:52 AM Alister Burt <alist...@gmail.com> wrote:
>
> Hi Pranav,
>
> If it’s just for visualisation and won’t affect analysis could you rotate the tomogram?
>
> https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.rotate.html
> or
> https://github.com/clEsperanto/pyclesperanto_prototype/blob/master/demo/transforms/affine_transforms.ipynb
>
> Have fun!
>
> Alister

> To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/662d2ddf-d9c2-4018-bc32-a22fd6068554n%40googlegroups.com.
>
>

[Pranav Shah]

That was a great suggestion Alister! Had to go through some trials and
errors in the installation, but managed to get it to work...!

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

[Alister Burt]

Hi Pranav,

If it’s just for visualisation and won’t affect analysis could you rotate the tomogram?

https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.rotate.html

or

https://github.com/clEsperanto/pyclesperanto_prototype/blob/master/demo/transforms/affine_transforms.ipynb

Have fun!

Alister

On 17 Nov 2021, at 00:40, Pranav Shah <p.sha...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/662d2ddf-d9c2-4018-bc32-a22fd6068554n%40googlegroups.com.

[Digvijay Singh]

Hi Pranav,

In addition to Alister's suggestions; you could also use rotatevol (from IMOD;https://bio3d.colorado.edu/imod/doc/man/rotatevol.html) or e2proc3d.py. E.g,:

rotatevol -angles z,y,x -I Tomo.mrc --ou Tomo_rotated.mrc 

rotatevol -angles 10,y,x -I Tomo.mrc --ou Tomo_rotatedby10.mrc

Cheers