removing bad tilts before WARP

[mrJ]

hi all,

I am processing large number of tilt series and trying to think of how to optimize the pipeline for high throughput.  Currently I import frames into WARP and manually remove the bad tilts.  This is time consuming.  

I had the idea to instead use AREtomo to "preprocess" all of the tilt series, which automatically removes the bad tilts and outputs a file containing a list of bad tilts.  I could conceivably use this list to "clean" my data before importing into WARP.  I'm just a bit confused about how to do this.  I could simply remove those .tif movies corresponding to bad tilts, but presumably I need to also edit the mdoc file.

 If I'm editing the mdoc file I think I can't just remove the whole paragraph for the bad tilts.  How might I go about this, if it is at all possible?

thanks!

Jesse  

[Lorenzo Gaifas]

Hi Jesse,

I went down this rabbit hole already, and ultimately decided against it because it was error prone and complex to implement, and I still didn't trust completely Aretomo to be able to remove all bad tilts (it mostly removes "dark" tilts, but sometimes tils are bad in other ways), which partially defeated the purpose.

However, if you want to go about it, this is how I would do it:

- preprocess everything in warp, but do no manual selection, and spit out the imod stacks

- use aretomo for processing

- make a script to

    - extract the indices of the dark tilts

    - find the `xml` files for all the rejected tilts (watch out for the different order if the collection was dose-symmetric) and switch the `UnselectManual` flag to `True`.

    - mdoc shouldn't need updating AFAIK

Best,

Lorenzo

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/1376e694-fab4-4a9d-b871-f0e33b0e9a79n%40googlegroups.com.

[mrJ]

hi Lorenzo, 

that's a great strategy.  Thanks!  I guess at the very least I could do put something together which does that and it would remove the majority of bad ones.  Then I could just go through it manually afterwards and pick out anything with bad CTF.  

I'm curious, how else do you decide to remove tilts?  I come from a SPA background and I'm not used to assesing each movie one by one by it's CTF.  I guess this is just because SPA generates so many more movies.  Anyways, if you have any more metrics for how to remove bad tilts I'd love to hear them.   Usually I look for (among other things) the following:

- single tilts which have unusual  defocus relative to the rest of the series, possible CTF fit wrong

- cases where CTF estimate looks reasonable but doesn't actually match the measured CTF

thanks!

Jesse

[Lorenzo Gaifas]

Hi Jesse,

Sure, you save some clicks to disable bad tilts, but the time consuming part is really going through and checking the thumbnails, not really the "click" itself, so I found that I wasn't saving that much time :)

I usually collapse the left tab so you can see both CTF and real space; then I look for:

- really "unfittable"-looking CTF (if fit is "just" bad, sometimes it's just because the parameters need to be adjusted, so I usually do a few rounds of "fit only this CTF" on tilts with different tilt angle and defocus, to try and find a parameter set that works for everything)

- big blobs of ice coming into view. Not sure how many of these are caught by Aretomo, you'd have to test a bit

- big, sudden shifts, which are not really caught by aretomo (sometimes collection just messes up due to eucentric height and focusing issues)

- corrupted images (all black or big stripes)

This is probably not all, but that's what comes to mind now.

Best,

Lorenzo

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/059a8c41-190c-4184-94b2-cc389469e39dn%40googlegroups.com.