Henderson's "true resolution" test, and "gold standard" refinement

[Steven Ludtke]

Ok, this is unprompted, but with all the discussions floating around in the cryo-EM community recently, I thought it was worth posting EMAN2's 'official' opinion on the topic here. These concepts will be included in the paper we're preparing now on EMAN2.1's refinement strategy.

There have been a lot of new buzzwords flying around the Cryo-EM community recently about resolution measures and making sure you don't over-refine your maps. First there was "gold standard" refinement (or FSC), which simply means computing the FSC as it was originally supposed to be computed, with two 'completely' independent maps. That is, you start with a split data set, and refine the two halves completely independently from each other, using independently generated starting models, then measure the resolution using the FSC between these maps. When using this more robust refinement/FSC curve, you can use Henderson's 0.143 resolution cutoff (which I maintain should actually be 0.2, due to a math error in the 0.143 paper, but it has so little effect, I'm not pushing the issue). This is the normal refinement procedure used in EMAN2.1, which has additional benefits, including automatic Wiener filtration of the final maps at their 'true' resolution.

However, this doesn't completely satisfy everyone. There is still some question about what 'completely' really means up above. Relion, for example, typically uses starting maps which aren't completely independent, but have been low-pass filtered to 0 amplitude at something like 60 Å resolution. ie - it gives the refinement a general starting shape, but contributes no high resolution information, which must be derived entirely from the data. EMAN2 generally uses a starting map which has been phase-randomized beyond 1.5 - 2x the target resolution. ie- if you are targeting 10 Å resolution, you would randomize the phases of the starting model beyond 20 Å (for example). Not only does this mean there are no correlations at high resolution, it actually biases the starting maps away from each other, forcing the refinement to get rid of the bad phases before it can fill in the good ones. In practice, both of these methods seem to work quite well.

However, there are other issues. One almost always imposes a mask of some sort on the reconstruction before computing a FSC. If these masks are not also independent (and low resolution) the masks can lead to false correlation between the maps. Now, EMAN2.1 uses an independently and automatically generated mask for each of the two output maps, which is additionally very 'soft' (Gaussian edge), so I maintain that this should be fine, and not exaggerate resolution.

Step forward in time a bit, and you'll see that Richard Henderson published an additional paper suggesting a method for proving that your refinements really are independent, and adjusting your resolution (if it turns out that they were not). The idea is straightforward, take your raw data, do a normal (in EMAN2.1 this would be gold-standard) refinement, then phase-randomize the particle data beyond some target resolution. This resolution should be worse than the resolution your normal refinement claimed to have achieved. Since the particle data no longer has any actual data/signal past some cutoff resolution, if you re-refine it exactly the same way, you should find that the FSC falls rapidly at the exact resolution where you started randomizing the phases. If this does not happen, it implies that your refinement procedure has some hidden correlation between the even/odd maps, and your resolution was over-estimated.

Of course, simple interpolation issues will mean that the curve is never going to fall off perfectly to 0 at the exact pixel where you filtered it, but hopefully it is possible to get close.  Attached you will see the plot of what happens when you do this in EMAN2.1.  As you can see, the gold-standard refinement approaches the 'perfect' result of no correlation beyond the phase-randomized resolution, so I'm quite confident that the refinement is quite robust. 

That doesn't mean you couldn't override the automatic options, and impose a mask which causes resolutions to be over-estimated again, but the default refinements should generally be pretty reliable. If you have any concerns about a particular case, it never hurts to try Richard's little test. It is quite straightforward to perform, and may give you more confidence in your map.

----------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)
Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)
Co-Director CIBR Center                          (www.bcm.edu/research/cibr)
Baylor College of Medicine                             
slu...@bcm.edu

[Paul Penczek]

Hi,

concerning what "truly independent" means: the two maps have to be aligned with respect to each other

in order to compute an FSC curve.  That decreases the degree of independency further in additional to problems

you already mentioned.

More importantly, if somewhere at the beginning the process starts from the same template it is difficult

to talk about independency in the strict meaning of the word.  For example, we can consider a refinement

method that introduces distortions/bias in a deterministic manner.  As a result, both maps will be distorted (incorrect?)

to a degree, but in the same manner, and the odd-even refinement test will not detect the problem.

Regards,

Pawel.

---

From: Steven Ludtke <slu...@bcm.edu>
To: "em...@googlegroups.com" <em...@googlegroups.com>
Sent: Thursday, September 5, 2013 8:36 AM
Subject: [EMAN2] Henderson's "true resolution" test, and "gold standard" refinement

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
 
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Steven Ludtke]

On Sep 5, 2013, at 4:14 PM, Paul Penczek <pawe...@att.net> wrote:

Hi,

concerning what "truly independent" means: the two maps have to be aligned with respect to each other

in order to compute an FSC curve.  That decreases the degree of independency further in additional to problems

you already mentioned.

While I agree with this from a purist point of view, practically it isn't an issue. I agree if you have two maps of pure noise, then align them you will get better correlation (since that's what alignment means) than if they weren't aligned, and the FSC will also become non-zero. Practically speaking, though, this would be akin to noise bias from only 2 particles. It takes a lot more averaging and alignment to create an effect that has any practical impact. Now, if the maps were averaged, and the average map were used to seed the next refinement iteration (both even and odd), THEN you would have a real problem.

More importantly, if somewhere at the beginning the process starts from the same template it is difficult

to talk about independency in the strict meaning of the word.  For example, we can consider a refinement

method that introduces distortions/bias in a deterministic manner.  As a result, both maps will be distorted (incorrect?)

to a degree, but in the same manner, and the odd-even refinement test will not detect the problem.

This was my argument against Sjors' version of the method which used maps where the complex amplitudes were 0 beyond ~40-60 Å. ie- if you view the problem as an N dimensional energy minimization, starting at the same point in the energy space with different data will bias you towards the same local minimum, even if the data is completely different. The approach of leaving non-zero amplitudes by randomizing the phases differently in the two maps I felt was a better approach. However, after a fair bit of real-world testing I did on several different structures, I convinced myself that as a practical matter, Sjors method worked as well at removing bias, at least for the situations I tested. I still prefer to really force the maps to start at different points in the energy space (though in the same low-resolution neighborhood).

The problem comes into play when (as with the recently discussed paper), they do neither of these things. That is, they start with the same map which has the same amplitudes and phases out to high resolution.

In any case, if you look at the example I posted, it should be clear that there is no significant noise or model bias in the approach EMAN2.1 is using. If there is no 'signal' present past resolution X, the FSC curve does not get (measurably) past resolution X during refinement.

Regards,

Pawel.

---

From: Steven Ludtke <slu...@bcm.edu>
To: "em...@googlegroups.com" <em...@googlegroups.com>
Sent: Thursday, September 5, 2013 8:36 AM
Subject: [EMAN2] Henderson's "true resolution" test, and "gold standard" refinement

Ok, this is unprompted, but with all the discussions floating around in the cryo-EM community recently, I thought it was worth posting EMAN2's 'official' opinion on the topic here. These concepts will be included in the paper we're preparing now on EMAN2.1's refinement strategy.

There have been a lot of new buzzwords flying around the Cryo-EM community recently about resolution measures and making sure you don't over-refine your maps. First there was "gold standard" refinement (or FSC), which simply means computing the FSC as it was originally supposed to be computed, with two 'completely' independent maps. That is, you start with a split data set, and refine the two halves completely independently from each other, using independently generated starting models, then measure the resolution using the FSC between these maps. When using this more robust refinement/FSC curve, you can use Henderson's 0.143 resolution cutoff (which I maintain should actually be 0.2, due to a math error in the 0.143 paper, but it has so little effect, I'm not pushing the issue). This is the normal refinement procedure used in EMAN2.1, which has additional benefits, including automatic Wiener filtration of the final maps at their 'true' resolution.

However, this doesn't completely satisfy everyone. There is still some question about what 'completely' really means up above. Relion, for example, typically uses starting maps which aren't completely independent, but have been low-pass filtered to 0 amplitude at something like 60 Å resolution. ie - it gives the refinement a general starting shape, but contributes no high resolution information, which must be derived entirely from the data. EMAN2 generally uses a starting map which has been phase-randomized beyond 1.5 - 2x the target resolution. ie- if you are targeting 10 Å resolution, you would randomize the phases of the starting model beyond 20 Å (for example). Not only does this mean there are no correlations at high resolution, it actually biases the starting maps away from each other, forcing the refinement to get rid of the bad phases before it can fill in the good ones. In practice, both of these methods seem to work quite well.

However, there are other issues. One almost always imposes a mask of some sort on the reconstruction before computing a FSC. If these masks are not also independent (and low resolution) the masks can lead to false correlation between the maps. Now, EMAN2.1 uses an independently and automatically generated mask for each of the two output maps, which is additionally very 'soft' (Gaussian edge), so I maintain that this should be fine, and not exaggerate resolution.

Step forward in time a bit, and you'll see that Richard Henderson published an additional paper suggesting a method for proving that your refinements really are independent, and adjusting your resolution (if it turns out that they were not). The idea is straightforward, take your raw data, do a normal (in EMAN2.1 this would be gold-standard) refinement, then phase-randomize the particle data beyond some target resolution. This resolution should be worse than the resolution your normal refinement claimed to have achieved. Since the particle data no longer has any actual data/signal past some cutoff resolution, if you re-refine it exactly the same way, you should find that the FSC falls rapidly at the exact resolution where you started randomizing the phases. If this does not happen, it implies that your refinement procedure has some hidden correlation between the even/odd maps, and your resolution was over-estimated.

Of course, simple interpolation issues will mean that the curve is never going to fall off perfectly to 0 at the exact pixel where you filtered it, but hopefully it is possible to get close.  Attached you will see the plot of what happens when you do this in EMAN2.1.  As you can see, the gold-standard refinement approaches the 'perfect' result of no correlation beyond the phase-randomized resolution, so I'm quite confident that the refinement is quite robust. 

That doesn't mean you couldn't override the automatic options, and impose a mask which causes resolutions to be over-estimated again, but the default refinements should generally be pretty reliable. If you have any concerns about a particular case, it never hurts to try Richard's little test. It is quite straightforward to perform, and may give you more confidence in your map.

----------------------------------------------------------------------------
Steven Ludtke, Ph.D.
Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)
Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)
Co-Director CIBR Center                          (www.bcm.edu/research/cibr)
Baylor College of Medicine                             
slu...@bcm.edu

<resplot.jpg>

[Paul Penczek]

---

From: Steven Ludtke <slud...@gmail.com>
To: em...@googlegroups.com
Sent: Thursday, September 5, 2013 10:04 PM
Subject: Re: [EMAN2] Henderson's "true resolution" test, and "gold standard" refinement

On Sep 5, 2013, at 4:14 PM, Paul Penczek <pawe...@att.net> wrote:

Hi,

concerning what "truly independent" means: the two maps have to be aligned with respect to each other

in order to compute an FSC curve.  That decreases the degree of independency further in additional to problems

you already mentioned.

While I agree with this from a purist point of view, practically it isn't an issue. I agree if you have two maps of pure noise, then align them you will get better correlation (since that's what alignment means) than if they weren't aligned, and the FSC will also become non-zero. Practically speaking, though, this would be akin to noise bias from only 2 particles. It takes a lot more averaging and alignment to create an effect that has any practical impact. Now, if the maps were averaged, and the average map were used to seed the next refinement iteration (both even and odd), THEN you would have a real problem.

>>>  A simple test will persuade you that small misalignment has large impact on FSC outcome.  The point is not the bias,

but that there is no such thing as "independent" FSC.  In simple words, alignment of half-volumes changes FSC significantly,

which in turn influences the filtration and finally the outcome of supposedly "independent" refinements.