Relion vs EMAN2

[Steven Ludtke]

There have been a lot of questions and comments about Relion recently, so I thought it was worth a brief post. You'll note that it is now possible to "seed" a relion refinement with the results of an EMAN2 refinement for comparative purposes. Of course, philosophically, there is nothing wrong with the concept of maximum likelihood, and it produces extremely useful results in many fields. However, the way ML is currently handled in cryo-EM has a number of shortcomings, which makes it, in the end, pretty much equivalent results-wise to current methods, but far more computationally expensive. The shortcomings I see with current ML implementations:

1) Assumption that all particles are "good" and that they are equally "good", which is demonstrably untrue

2) Assumption that noise is independent & Gaussian in real and/or Fourier space - while this may be true for the noise coming from the microscope/detector, the reconstruction algorithm also sees as noise anything in the physical specimen which isn't a perfect particle. This component of the noise is often dominant, and is highly structured.

3) When the data is good, the orientations should settle on one specific value anyway. ie- if a particle really is probabilistically spread out in orientation space, it probably has no business going into the reconstruction anyway. That is, the only benefit you get out of ML in cases where the orientations are unique is that the local "smearing" due to slight local angular uncertainty is integrated into the answer. This can be taken into account without doing a full ML integral.

Anyway, here is a quick comparison from a poster in Tahoe last week. Clearly such comparisons are specimen and data-quality dependent, but it does illustrate some of what I'm trying to say in a practical way. Structures are presented as they were produced directly by the corresponding program, without additional filtration, note also that the EMAN2 result is done with speed=1, which provides a slight resolution boost at the cost of some CPU time. A ~8.4 Å map (vs the ~7.8 Å maps below) can be done in EMAN2 in 3-4 CPU-hours. Also note that FreAlign can do a bit better than shown in mode 4, but it takes a lot longer to run, and we didn't have a chance to do it before the poster was printed:

[Steve Ludtke]

After my posting a couple of weeks ago comparing EMAN2 to Relion, but also including a less than flattering comparison with Frealign, Niko was obviously concerned, as he expected (as did I) Frealign to do better than this. After Niko played with the data a bit himself, we made some additional improvements to the EMAN2->Frealign converter, upgraded to the latest release of Frealign, and switched from mode 1 to mode 3. Stephen (my grad student) put together this updated comparison.

Note that this new comparison includes both the raw maps as they came out of each program, as well as a version filtered based on the FSC with the crystal structure (applied as a Wiener filter in each case after sharpening the maps).

Looking at the timings, you have to keep in mind that the Frealign run used the EMAN2 angles as starting points, so a straight time comparison isn't really accurate there, whereas Relion just got an initial model, and no angles. We'd have to go back and see just how bad the initial angles can be in Frealign and still converge properly to come up with a total time. However, if you want to experiment with any of Frealign's per-particle parameter refinement capabilities, or just see how things compare, it should now be easy to do. Relion is, of course, also easy to compare, but will require a little more waiting to find the results ;^)  The initial model used for the EMAN2 refinement was generated from a few class-averages in the usual EMAN2 way.