issues with subtomos reconstructed in warp

[Przemysław Dutka]

Hello, 

I have some issues with 3D refinement in Relion using subtomos reconstructed by Warp. I'm getting the following error: 

ERROR:

BUG! cannot find positive values in sigma2_noise spectrum 

When I reconstruct particles using relion_reconstruct all looks normal but when I start 3D refinement or classification error persist. I was able to work with this dataset before without any issues, the only difference now is that I changes particles spacing for extraction. I don't see how that would have any effect on subtomos reconstruction in Warp. 

I had similar issues previously with teamtomo tutorial data when reconstructing subtomos at some pixel size. However, now I'm getting this error regardless of the pixel size in reconstructed subtomos.

Thank you for your help in advance!

Cheers,

Przemek 

Full relion error below:

Estimating initial noise spectra
000/??? sec ~~(,_,">                                                          [oo]
0.62/37.02 min ~~(,_,">
1.25/37.52 min .~~(,_,">
1.90/38.02 min ..~~(,_,">
2.52/37.77 min ...~~(,_,">
3.17/38.02 min ....~~(,_,">
3.77/37.68 min .....~~(,_,">
4.40/37.73 min ......~~(,_,">
5.03/37.77 min .......~~(,_,">
5.67/37.80 min ........~~(,_,">
6.32/37.92 min .........~~(,_,">
6.97/38.02 min ..........~~(,_,">
7.62/38.10 min ...........~~(,_,">
8.23/38.02 min ............~~(,_,">
8.83/37.87 min .............~~(,_,">
9.48/37.95 min ..............~~(,_,">
10.12/37.95 min ...............~~(,_,">
10.73/37.90 min ................~~(,_,">
11.38/37.97 min .................~~(,_,">
12.03/38.02 min ..................~~(,_,">
12.68/38.07 min ...................~~(,_,">
13.32/38.07 min ....................~~(,_,">
13.95/38.07 min .....................~~(,_,">
14.57/38.02 min ......................~~(,_,">
15.20/38.02 min .......................~~(,_,">
15.82/37.97 min ........................~~(,_,">
16.42/37.90 min .........................~~(,_,">
17.07/37.93 min ..........................~~(,_,">
17.70/37.95 min ...........................~~(,_,">
18.33/37.95 min ............................~~(,_,">
18.98/37.98 min .............................~~(,_,">
19.60/37.95 min ..............................~~(,_,">
20.23/37.95 min ...............................~~(,_,">
20.90/38.02 min ................................~~(,_,">
21.52/37.98 min .................................~~(,_,">
22.13/37.95 min ..................................~~(,_,">
22.77/37.97 min ...................................~~(,_,">
23.37/37.90 min ....................................~~(,_,">
24.02/37.93 min .....................................~~(,_,">
24.65/37.93 min ......................................~~(,_,">
25.27/37.92 min .......................................~~(,_,">
25.90/37.92 min ........................................~~(,_,">
26.48/37.85 min .........................................~~(,_,">
27.12/37.85 min ..........................................~~(,_,">
27.77/37.88 min ...........................................~~(,_,">
28.38/37.87 min ............................................~~(,_,">
29.02/37.87 min .............................................~~(,_,">
29.63/37.85 min ..............................................~~(,_,">
30.25/37.83 min ...............................................~~(,_,">
30.90/37.85 min ................................................~~(,_,">
31.52/37.83 min .................................................~~(,_,">
32.13/37.82 min ..................................................~~(,_,">
32.78/37.83 min ...................................................~~(,_,">
33.38/37.80 min ....................................................~~(,_,">
34.02/37.82 min .....................................................~~(,_,">
34.63/37.80 min ......................................................~~(,_,">
35.23/37.77 min .......................................................~~(,_,">
35.87/37.77 min ........................................................~~(,_,">
36.48/37.75 min .........................................................~~(,_,">
37.07/37.72 min ..........................................................~~(,_,">
37.72/37.73 min ...........................................................~~(,_,">
37.73/37.73 min ............................................................~~(,_,">
 igroup= 5 n= 0 mymodel.sigma2_noise[igroup]=
-8.941e-09
-2.384e-09
-5.757e-10
-9.641e-11
-1.076e-10
 igroup= 5 n= 0 mymodel.sigma2_noise[igroup]=
-8.740e-11
-1.585e-10
-3.152e-10
-8.540e-11
-9.327e-12
-4.856e-12
-3.362e-12
-3.166e-12
-3.266e-12
-3.130e-12
-3.426e-12
-3.131e-12
-3.097e-12
-3.537e-12
-3.865e-12
-3.607e-12
-3.463e-12
-3.175e-12
-3.249e-12
-3.169e-12
-3.205e-12
-8.941e-09
-2.384e-09
-5.757e-10
-9.641e-11
-1.076e-10
-8.740e-11
-1.585e-10
-3.152e-10-3.255e-12
-3.012e-12
-2.828e-12
-2.800e-12
-2.651e-12
-2.390e-12
-1.338e-12


-8.540e-11
-9.327e-12
-4.856e-12
-3.362e-12
-3.166e-12
-3.266e-12
-3.130e-12
-3.426e-12
-3.131e-12
-3.097e-12
-3.537e-12
-3.865e-12
-3.607e-12
-3.463e-12
-3.175e-12
-3.249e-12
-3.169e-12
-3.205e-12
-3.255e-12
-3.012e-12
-2.828e-12
-2.800e-12
-2.651e-12
-2.390e-12
-1.338e-12

in: /tmp/sbgrid/spack-stage/spack-stage-relion-3.1.3-hppel26fj5mkjmljzrlpmg7b3guaubgp/spack-src/src/ml_optimiser.cpp, line 2445
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum
in: /tmp/sbgrid/spack-stage/spack-stage-relion-3.1.3-hppel26fj5mkjmljzrlpmg7b3guaubgp/spack-src/src/ml_optimiser.cpp, line 2445
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum
=== Backtrace  ===
=== Backtrace  ===
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x451011]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11MlOptimiser40setSigmaNoiseEstimatesAndSetAverageImageER13MultidimArrayIdE+0x819) [0x5f14c9]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN14MlOptimiserMpi10initialiseEv+0x9e5) [0x475ce5]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(main+0x55) [0x43cc75]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5aa435b555]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi() [0x43fa4f]
==================
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x451011]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11MlOptimiser40setSigmaNoiseEstimatesAndSetAverageImageER13MultidimArrayIdE+0x819) [0x5f14c9]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN14MlOptimiserMpi10initialiseEv+0x9e5) [0x475ce5]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(main+0x55) [0x43cc75]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fab2e4cf555]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi() [0x43fa4f]
==================
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum

 RELION version: 3.1.3
 exiting with an error ...

 RELION version: 3.1.3
 exiting with an error ...
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 igroup= 5 n= 0 mymodel.sigma2_noise[igroup]=
-8.941e-09
-2.384e-09
-5.757e-10
-9.641e-11
-1.076e-10
-8.740e-11
-1.585e-10
-3.152e-10
-8.540e-11
-9.327e-12
-4.856e-12
-3.362e-12
-3.166e-12
-3.266e-12
-3.130e-12
-3.426e-12
-3.131e-12
-3.097e-12
-3.537e-12
-3.865e-12
-3.607e-12
-3.463e-12
-3.175e-12
-3.249e-12
-3.169e-12
-3.205e-12
-3.255e-12
-3.012e-12
-2.828e-12
-2.800e-12
-2.651e-12
-2.390e-12
-1.338e-12

in: /tmp/sbgrid/spack-stage/spack-stage-relion-3.1.3-hppel26fj5mkjmljzrlpmg7b3guaubgp/spack-src/src/ml_optimiser.cpp, line 2445
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum
=== Backtrace  ===
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x451011]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN11MlOptimiser40setSigmaNoiseEstimatesAndSetAverageImageER13MultidimArrayIdE+0x819) [0x5f14c9]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(_ZN14MlOptimiserMpi10initialiseEv+0x9e5) [0x475ce5]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi(main+0x55) [0x43cc75]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f915e5c0555]
/programs/x86_64-linux/relion/3.1.3_cu10.2/bin/relion_refine_mpi() [0x43fa4f]
==================
ERROR:
BUG! cannot find positive values in sigma2_noise spectrum

 RELION version: 3.1.3
 exiting with an error ...
[gpunode02:32206] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[gpunode02:32206] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[Alister Burt]

Hey Przemek,

Usually these issues are related to normalisation - making sure that you use the same ‘particle diameter’ in both warp and relion is the typical advice here

Cheers and good luck,

Alister

On 25 Jan 2022, at 07:13, Przemysław Dutka <przemysl...@gmail.com> wrote:

Hello, 

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/616f7645-682a-4c8b-a498-ac4cbf83d50an%40googlegroups.com.

[Przemysław Dutka]

Hey Alister,

Thanks for the suggestion. I did more troubleshooting and I think I found the source of the error, but I don't know how to fix it. When I ran relion_image_handler --stats I found out that subtomos extracted from four of my tomograms have intensity values 0 (check attached file). I ran subtomos extraction several times in Warp and it constantly produces all black subtomos for the same four tomograms. I don't see anything special about these particular tomograms and particles extracted from these subtomograms in Dynamo look normal.  Do you (or anyone) have any suggestions? Are there any Warp log files I could check to get more information?

Thanks,

Przemek 

[Florian Fäßler]

Hi Przemek,

 

I have only experienced the issue of individual tomograms only yielding empty subvolumes after subtomogram export with warp, when those specific tomograms had a higher estimated defocus and/or higher resolution than the rest. Typically the GPU would then not be able to handle the export resulting in the empty subtomos. Took me some time to find out because warp would not crash during the export from these tomos but only after it started exporting for the next subset of tomos (So no error message, when I exported them individually). What helped in my case was to do the export for the problematic tomos using CPU and merge star files later.

 

Hope this also works for you!

 

Cheers,

 

Florian

 

 

Gesendet: Sonntag, 30. Januar 2022 um 04:47 Uhr
Von: "Przemysław Dutka" <przemysl...@gmail.com>
An: "Warp" <war...@googlegroups.com>
Betreff: Re: issues with subtomos reconstructed in warp

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/61849f86-a092-4dfd-88a4-505d3a4ec959n%40googlegroups.com.

[Jessica Rabuck-Gibbons]

Hi Przemek,

I've also had issues similar to Florian's, but in my case, it was because I was extracting subtomos at 2.5A. Warp would crash, but the only way I could really tell was that it wasn't writing out subtomos, and the Warp worker had also disappeared. The endpoint of my issue was that I didn't have enough memory to extract so many subtomos at such a high resolution, and files ended up getting corrupted when Warp crashed.

In my case, my problems were solved by (1) doing a complete restart of the computer and making sure that Warp was fully shut down, (2) extracting with a slightly larger pixel size, (3) individually extracting subtomos from each tomogram, and then (4) getting onto a better computer (long story short--I had a really weird dual boot GPU computer from like 2019 built for my postdoc lab, and it just didn't have enough juice to deal with anything but simple test cases. Eventually I was able to move to our collaborator's computer that had been built with tomography in mind). 

One more note--It can also help to basically start from scratch with your processing. I've found that it's not as much of a time suck since you can still use your imod files and particle picks, which for me were the bottlenecks in data processing. Sometimes important files end up getting corrupted, which is a super annoying bug to try to track down. Good luck!

Thanks,

Jess

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/trinity-3446f855-de0f-4665-beda-cdd287241774-1643528990999%403c-app-gmx-bs09.

[Przemysław Dutka]

Hi everyone, 

Thanks for all the advice. I tried everything and the only thing that was able to resolve the issue was to start from scratch. 

Best,

Przemek