relion crashes with warp particles from large datasets

[Andres Hernandez]

I have routinely encountered a crash in relion 3.1b and 3.1 with particles from the last two releases of warp. Relion crashes, typically by iteration 3, with the following error:

"No orientation was found as better than any other. A particle image was compared to the reference and resulted in all-zero

weights (for all orientations)...."

The error has been popping up more frequently with the larger K3 datasets. Usually any dataset over 1 million particles will have a handful of particles that give the error. If I remove the flagged particles from the star file or re-extract the particles in relion, the issue goes away. Sometimes, after removing the flagged particles, a new set of particles throws the error. Any idea what causes it? I suspect it is coming from the way particles are normalized in warp, but can't explain why it seems to also be dependent on the size of the dataset.

Best,

Andres

[teg...@gmail.com]

Hi Andres,

one of my guesses is that the particle export is running out of GPU memory. I have now modified the code to limit the most expensive part (Fourier transform) to at most 128 particles at a time. It's included in this new build: http://deployment.warpem.com/nightly/Warp_20201012.zip

However, if it's only individual particles in a stack, the problem is more complex and rather on RELION's side :-) The normalization procedure is identical between Warp and RELION, but if RELION re-extracts from Warp's averages, the results are slightly different. Both versions have exposure weighting applied, but for the full micrograph average (from which RELION extracts) Warp doesn't divide the result by the sum of weights, resulting in somewhat dampened amplitudes at higher spatial frequencies. None of this should lead to RELION's error – I just wanted to point out why results would be slightly different.

The "No orientation was found as better than any other" error in RELION has been a frequent occurrence since the 2.0 version (when GPU acceleration was added). Its handling has been improved since then, and there is a fixed number of CPU fallback attempts that is set to 40 by default (--failsafe_threshold parameter). This might explain why it fails on bigger data sets since that number is more likely to be hit with more particles.

Cheers,

Dimitry