Strange FSC curves from validation server

[huang....@gmail.com]

Hi Steve and Muyuan, 

I have finished a few rounds of structure refinement using the old subtomogram - subtilt pipeline. Since the structure looked quite good, I wanted to validate the map prior to possible deposition to the EMDB. For some reason, there was a large discrepancy between the curves calculated by the FSC validation server and via EMAN2 original. 

While the unmasked FSC plot looks similar, the masked curve never dropped to zero, unlike the one calculated during the refinement by EMAN2. 

I have attached the images below if you wouldn't mind taking a look. I'm wondering if my masking was too tight. In any case, I am a little confused since the validation server indicates they use the EMAN2 package for FSC calculation. 

Thank you so much,

Judy

[Muyuan Chen]

This is actually somewhat expected behavior. So if you run

e2proc3d.py spt_01/threed_05_even.hdf spt_01/fsc_tmp.txt --calcfsc spt_01/threed_05_odd.hdf

you will get a funny curve (blue) that does not fall to zero. Green curve is the unmasked FSC for comparison.

To get the correct masked FSC curve, first mask the threed_raw_even/odd file then compare them.

e2proc3d.py spt_01/threed_raw_even.hdf spt_01/tmp_even.hdf --multfile spt_00/mask.hdf
e2proc3d.py spt_01/threed_raw_odd.hdf spt_01/tmp_odd.hdf --multfile spt_00/mask.hdf

e2proc3d.py spt_01/tmp_even.hdf spt_01/fsc_tmp1.txt --calcfsc spt_01/tmp_odd.hdf

you will get the red curve that should be identical to the fsc_masked_05.txt produced automatically.

This is mostly due to the sharpening processes that happen after the FSC calculation, especially if you use the local filter option. Steve may have a better theoretical lecture about this...

Muyuan

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/22a079fb-1c3b-410f-882f-eb963177fd40n%40googlegroups.com.

[Judy Huang]

Hmmmmm interesting, thank you Muyuan!

Sort of a follow-up question as FSC curves seem to be my nemesis today. a FSC curve I generated using the new workflow dips below zero (screenshot attached). I was wondering why that might be...

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqEEJA%3D2CfoWvESmb%2BOWgftKWg6-93xk2ODW3VzbScCuUw%40mail.gmail.com.

[Muyuan Chen]

I would guess it is some mask-induced bias that accumulates during the iterations. It is hard to guess the cause of the deeper cause of this just from the curve though.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAOpU3BRnCjQ8WoxmWraXuXWTOPELX0%3DPfHNUunD_%3D72LrBwSuQ%40mail.gmail.com.

[digvij...@gmail.com]

I have a related question. 

Shouldn't we be using the threed_odd_unmasked.hdf (instead of threed_raw_odd.hdf) and threed_even_unmasked.hdf for masking with a desired mask and then calculating the masked FSC between their masked counterparts? 

What are the threed_raw_odd.hdf and threed_raw_even.hdf files? I saw the .eman2log.txt files and I could not find any processes relating to them. All the postprocessing was done using threed_odd_unmasked.hdf and  threed_even_unmasked.hdf. 

[Muyuan Chen]

The raw files are directly copied from the output of spt_average or make3d so they don’t show up in the log. The unmasked files are filtered. It normally shouldn’t be a problem but some local filter operations might have mask like effects on them.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/0d509efc-6c50-473c-94b0-2902bc91c06an%40googlegroups.com.

[digvij...@gmail.com]

I see. 

I see that the threed_even_unmasked and threed_odd_unmasked are filtered by the following process from the raw odd and even averages. i.e. process filter.lowpass.tophat:cutoff_abs=0.5

But e2fsc.py or related functions use the  threed_even_unmasked and  threed_odd_unmasked, instead of  threed_raw_even and  threed_raw_odd. 
Would it not be better to use the threed_raw_even and  threed_raw_odd for e2fsc or related functions?

[Muyuan Chen]

The whole point of keeping the raw files is to have an unprocessed version in case it is needed later, like the case of validation server here. So it won’t be used as input for the post process. Also, at the beginning of post process, the threed_xx_even file is essentially the raw file, so there isn’t a reason to differentiate either...

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/27a13d00-1622-494a-b374-0249c50159b3n%40googlegroups.com.

[digvij...@gmail.com]

Thanks, absolutely, yes, I know that threed_n_even and threed_n_odd are unprocessed when they are fed to the post-process pipeline.

I got confused when I saw that unprocessed threed_0n_even/threed_0n_odd are filtered by (process filter.lowpass.tophat:cutoff_abs=0.5) to generated odd_unmasked and even_masked which are then used for e2fsc.py. I think I forgot but maybe the abs value of 0.5 in EMAN2 convention means that there is no low-pass filtering (?), so technically, this filter is probably doing nothing. I thought abs of 1 would mean no-low pass filtering but maybe I was wrong. I gotta double check. 

No worries. Thank you!

[Muyuan Chen]

Cutoff_abs=0.5 filters at Nyquist. It does almost nothing, but may help reducing some edge artifacts in fourier space I guess.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/d6e47fef-6a9f-4564-9206-1d10e073c493n%40googlegroups.com.

[digvij...@gmail.com]

Thanks for the clarification