WarpTools template matching

[Lorenz Grundmann]

Hi everyone,

I am playing around with WarpTools and try to wrap my head around the commands and respective settings.  After following the tutorial, I am trying to apply this workflow to in-situ data and would like to play around with the thresholding for optimization purposes, but I do not fully get the thresholding.

1. In the tutorial the --minimum is set to 6, but what is its unit?

2. Do I threshold on the _rlnAutopickFigureOfMerit in the star file or by finding local maxima and their correlation?

3. Can I somehow visualize which particles are thresholded (other than checking the previews)?

After performing the tutorial, I am trying to apply this workflow to in-situ data and would like to play around with the thresholding for optimization purposes.

Cheers,

Lorenz

[Alister Burt]

Hi Lorenz,

1. template matching results are normalized to mean=0, std=1 so the units are 'standard deviations away from the mean'

2. the peaks in the STAR file are the result of finding local maxima and their correlation (with minimum distance supplied at the CLI), you should be able to work with the peak list in the star file directly

I made a small interactive viewer for TM results in napari a while ago which might help you achieve what you're going for with 3. 

It lets you navigate a whole set of TM results, visualise the CC volume on top of the tomograms and see the effect of different thresholds on the peak lists https://gist.github.com/alisterburt/4551aaa3882898f45ef761f194c59fbd

If you end up trying to run this it should be on a local workstation or a remote desktop, not over ssh/X forwarding

Cheers,

Alister

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/c7e6c64b-579c-43ce-a9ba-43f6e5465ecfn%40googlegroups.com.

[Lorenz Grundmann]

Hi Alister,

Thank you very much for the quick reply and hinting me to this handy napari viewer, which seems to be exactly what I was looking for :).

If I understand you correctly, the AutopickFigureOfMerit is the value that is normalized? 

Suprisingly, the thresholding is working differently for me on both the tutorial data and my in-situ data. When specifying --minimun=X for the WarpTools threshold_picks command the *clean.star only contains particles with a AutopickFigureOfMerit > X.

Here plotted is the AutopickFigureOfMerit of one tomogram.star of the tutorial:

After thresholding the particles via:
WarpTools threshold_picks \
--settings warp_tiltseries.settings \
--in_suffix 15854 \
--out_suffix clean \
--minimum 6

My tomogram_clean.star only contains the two particles with AutopickFigureOfMerit > 6:

data_

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnCoordinateZ #3
_rlnAngleRot #4
_rlnAngleTilt #5
_rlnAnglePsi #6
_rlnMicrographName #7
_rlnAutopickFigureOfMerit #8
  0.5632184  0.4092827  0.41025642   83.57143  41.858788  352.49997  TS_11.tomostar   6.000019
  0.3821839  0.8523207   0.8333333  112.49999  11.715852  344.99997  TS_11.tomostar  6.2819195

In this case I was only copying and pasting from the tutorial and thus I am a bit confused what could be the issue here. Am I doing somehting wrong?

I am using WarpTools provided by sbgrid.

Best,

Lorenz

[Alister Burt]

Hi Lorenz,

Values are normalised in the CC volume and the value of the volume at peaks is what's in the STAR file under rlnAutopickFigureOfMerit, those are then thresholded in threshold_picks

https://github.com/warpem/warp/blob/ea51d2fcc4e09da8fe08fa988b22ae29481900d8/WarpTools/Commands/General/ThresholdPicks.cs#L108-L109

While values should be comparable within a dataset, peak heights will differ depending on

- TS alignment quality

- angular sampling

- template size

- similarity of template to data

Exploring your data as you're doing to pick the right threshold for your processing is a good idea :-)

Cheers,

Alister

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/41c47183-050d-4c53-a388-f20e8817a410n%40googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/41c47183-050d-4c53-a388-f20e8817a410n%40googlegroups.com.

[Lorenz Grundmann]

Hi Alister,

Thanks a lot for the detailed information, it really helps me getting a 'grip' on the warp pipeline etc.

Cheers,
Lorenz

[Peter Van]

Hi all,

I have a few questions. Are the principles behind template matching-based particle picking with WarpTools similar to those in software like StopGap? Additionally, is there a significant difference in particle picking performance between these tools? Another question is whether Warp can perform 3D CTF correction, and if so, does it lead to a noticeable improvement in particle picking in 3D CTF-corrected tomograms? Under what circumstances should we consider using 3D CTF-corrected tomograms for particle picking?
Cheers,

Peter

[Pranav Shah]

Are the principles behind template matching-based particle picking with WarpTools similar to those in software like StopGap? 

- yes, in so far as both perform sophisticated filtering to improve the matching results(see the relevant first author pubs for deets)

Additionally, is there a significant difference in particle picking performance between these tools?

- Yes… however significant good or significant bad depends on the quality of the sample and TS alignments and some level of tinkering on the users end.

Another question is whether Warp can perform 3D CTF correction, and if so, does it lead to a noticeable improvement in particle picking in 3D CTF-corrected tomograms? Under what circumstances should we consider using 3D CTF-corrected tomograms for particle picking?

- Yes, yes, all (unless all you have are only non-ctf corrected tomograms and have lost access to the raw data or data collection params)

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/5c946905-3f6a-40fa-a55e-b94433bc730en%40googlegroups.com.

[Peter Van]

 I got it. Thank you, Pranav! One more question: In the stopgap paper, I noticed that they use tomograms with 3D-CTF correction done by Nova CTF for particle picking. I didn't find the tomogram 3D-CTF correction function in Warp. I'm not sure if Warp considers 3D-CTF correction  of tomograms when using template matching for particle picking or if it directly uses tomograms without this correction.  

cheers,

Peter

[Ricardo Righetto]

Someone please correct me if I'm wrong, but my understanding is that all tomograms reconstructed by Warp are CTF-corrected by definition? Warp reconstructs tomograms by assembling several overlapping subvolumes accounting for the defocus at the center of each subvolume, which is effectively a 3D CTF correction approach. Please see the Warp paper for details.

Best wishes,

--
Ricardo Diogo Righetto

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/bebec290-01aa-4dd5-a55e-d28cc8e51929n%40googlegroups.com.

[Pranav Shah]

Thats correct Ricardo

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/CAB7-Rru%3DDAc83%3D-gjdL1pPfGqDa7Chtd3ybx886Jd9tx3Xu4XQ%40mail.gmail.com.

[Alister Burt]

Hi all,

Productive discussion! Only thing to add is that yes, Warp's template matching properly considers the per-tilt defocus (3D-CTF) for each region of the tomogram during matching.

I highly recommend the methods section of the Warp paper for an explanation of what's going on.

Cheers,

Alister

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/CAHNYQ5pmKsKXNVRrr6JEPx-Z0B%2BBk88905zeKzYjd4SzPcC%3Dpw%40mail.gmail.com.

[Peter Van]

Very useful information for me. Thank you all!

[Peter Van]

I have a further question about Warp reconstruction. Currently, I've found three options: the original tomogram, one that can undergo CTF deconvolution, and the Noise2Noise denoised tomogram. I'm wondering which of these is the optimal choice for template matching in Warp, considering that the original tomogram often has the lowest contrast, with some tomograms having barely visible contrast to the naked eye.

Additionally, I have another question regarding the original tomogram. It currently involves 3D-CTF correction with WBP reconstruction in Warp. Is it possible to have an option for 3D-CTF correction with SART reconstruction in the future? Perhaps 3D-CTF correction with SART, which might offer better contrast, could improve template matching performance.

[Alister Burt]

Hi Peter,

Contrast and template matching performance aren’t linked in the way you think they are, the filtered tomograms may boost low frequency info but are not suitable for high res template matching.

There is no WBP in Warp, tomograms are reconstructed by direct Fourier inversion of overlapping cubes with the analytically correct CTF at the center of the cube for each 2D image used.

There are no plans to add SART and it wouldn’t improve template matching performance

Cheers,

Alister

Sent from mobile - apologies for brevity

On Aug 7, 2024, at 00:42, Peter Van <m18717...@163.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/ad6168b4-31c0-45ea-aea6-709c9e5f393cn%40googlegroups.com.

[Peter Van]

 I got it, thank you very much for Alister's detailed explanation!  

Cheers,

Peter

[Peter Van]

  I have a simple question: Does warp template matching support user-defined masks, such as an elliptical or structured mask? This may be more suitable for some proteins that are not nearly spherical.  

[Alister Burt]

Hi Peter,

There is no support for user defined masks.

Cheers,

Alister

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/9df495d8-c2a8-43f3-a9ca-008069bac226n%40googlegroups.com.

[Peter Van]

Hi Alister,

I got it. Thanks again!

Cheers,

Peter