tensorflow might not function normally in e2gmm refine

[Clotho]

Dear Muyuan,

I have recently conducted testing of the e2gmm refinement on the 'EMPIAR-10492' dataset, following the tutorials provided at 'https://blake.bcm.edu/emanwiki/EMAN2/e2gmm_refine'. However, both the 'e2gmm_guess_n.py' and 'e2gmm_refine_iter.py' commands yield the following warnings:

=============

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR

I0000 00:00:1725864668.678668  236592 service.cc:145] XLA service 0x55626c166d90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:

I0000 00:00:1725864668.678936  236592 service.cc:153]   StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0

I0000 00:00:1725864669.765751  236592 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.

WARNING:tensorflow:You are casting an input of type complex64 to an incompatible dtype float32.  This will discard the imaginary part and may not be what you intended.

============

Here is the e2version output:

EMAN 2.99.62 ( GITHUB: 2024-08-27 01:33 - commit: NOT-INSTALLED-FROM-GIT-REPO )

Your EMAN2 is running on: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17 3.10.0-957.el7.x86_64

Your Python version is: 3.12.5

Finally,  the resultant map called threed_05.hdf was as a resolution of 8 Å without any improvement. The commands and map have been provided below. Thank you in advance for any advice or suggestions you may have regarding this matter!

[Muyuan Chen]

Hi, 

I just ran one iteration of the same protocol with the latest version this morning, and confirmed the resolution is improving, so there is probably nothing significantly wrong with the code. However, since you only showed the very final result, it is hard to tell what went wrong. Can you check the following FSC curves:

r3d_00/fsc_masked_00.txt : this should get to 3.9Å, meaning the relion to eman conversion is working. 

r3d_00/threed_00_fsc.txt : this is the similarity between the GMM and the voxel map, and should be mostly above 0.9 before 3.6Å.

gmm_00/fsc_masked_01.txt : this is the first iteration of refinement, and should be slightly better than the 0th iteration one. 

This may help diagnose at which step the issue happened.

Muyuan

Dear Muyuan,

<Screen Shot 2024-09-09 at 20.31.46.png>

<Screen Shot 2024-09-09 at 20.30.09.png>

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/85a1389e-5b99-487b-9b25-515e0bc7c8e6n%40googlegroups.com.
<Screen Shot 2024-09-09 at 20.31.46.png><Screen Shot 2024-09-09 at 20.30.09.png>

[Clotho]

Dear Muyuan, 

Thanks for your reply. I attach the fsc curve plots and map snapshot here. It seems that the the FSC curve looks like normal but resolution of the final map (threed_05.hdf)

is no better than 4 Å. 

Best,

[Muyuan Chen]

oh. That might just be a sharpening issue? If you run something like:

e2spt_structfac.py --even gmm_00/threed_raw_even.hdf --res 3.5 --sharp

Do you see high resolution features in gmm_00/threed_rawsf.hdf? 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/05d69a9a-d778-40c4-8d34-8ca6d8e3ab3fn%40googlegroups.com.

[Clotho]

Dear Muyuan,

Cheers! The high resolution features now can be seen clearly. There are still questions needed to be clarify. 

(1). I did not  find any detailed explanation for the "--thr" parameter of the e2gmm_guess_n command. If I opened the conconstrcution "r3d_00/threed_00.hdf"  in chimeraX, the value of level was range from 0 to 4000. So I guessed that the "--thr 4" did not mean the level value of 4. For my own dataset, how to determine a proper value for the "--thr" parameter? 

(2). The resolution of e2gmm global refinement reported by eman2 is about 3.4 Å, but the final reconstruction-"threed_05.hdf" does not look like a 3.4 Å map compared with those raw map with similar resolution from Relion or cryoSPARC. That is to say,  for e2gmm, we should to focus on the post-processing map rather than the raw map? 

Best,

[Muyuan Chen]

1. --thr is the isosurface threshold level. It is just EMAN compress the hdf files, and chimera cannot read them properly. If you open the map with "show3d" in e2display browser, 4 should be the right number. 

2. Normally it should do the post-processing by itself, and you should get a sharpened map in threed_05. In fact, now you ran the e2spt_structfac program once, if you repeat the same e2gmm_refine_iter run, you should get correct maps in threed_xx.hdf in the new gmm folder. I don't know what went wrong in your case in the first place. I would guess there was some error message in the first e2convertrelion command that was missed that led to this, or some intermediate files was moved or deleted during the refinement...

Muyuan

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/c3062582-fe20-4936-9dc6-a171503d2b86n%40googlegroups.com.

[Steve Ludtke]

Chimera should read EMAN2 compressed HDF files correctly, the isosurface threshold will just be different...

On Sep 10, 2024, at 2:56 AM, Muyuan Chen <g5v...@gmail.com> wrote:

1. --thr is the isosurface threshold level. It is just EMAN compress the hdf files, and chimera cannot read them properly. If you open the map with "show3d" in e2display browser, 4 should be the right number. 

2. Normally it should do the post-processing by itself, and you should get a sharpened map in threed_05. In fact, now you ran the e2spt_structfac program once, if you repeat the same e2gmm_refine_iter run, you should get correct maps in threed_xx.hdf in the new gmm folder. I don't know what went wrong in your case in the first place. I would guess there was some error message in the first e2convertrelion command that was missed that led to this, or some intermediate files was moved or deleted during the refinement...

Muyuan

On Mon, Sep 9, 2024 at 11:12 PM Clotho <cloth...@gmail.com> wrote:

Dear Muyuan,

Cheers! The high resolution features now can be seen clearly. There are still questions needed to be clarify. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqHCneKzTDjVvNGWM6k4nA%2Byvh6XmrO3DL6zznN3acf_3Q%40mail.gmail.com.

[Clotho]

Hi, I just tested e2gmm on my own dataset. It is quite strange that the proper isosurface threshold value for the threed_00.hdf  generated by  e2convertrelion cmd would reach ~2000 in e2display. If I typed in "--thr  2000" for e2gmm_guess_n.py , the program reported error as follow:

dm=np.sort(d)[len(d)//2]

IndexError: index 0 is out of bounds for axis 0 with size 0

And the output log for the e2convertrelion cmd was displayed  here. Are there any difference from your case?

[Muyuan Chen]

Having threshold values reaching 2000 in e2display is very strange, since the map is by default normalized. That said, I encountered various glitches in e2display after the recent python upgrade, and sometimes ridiculous high numbers jump out in various places. Since you get the error later, it is more likely a threshold display issue than having actually a high value there. Maybe Steve can look into what went wrong? 

One thing is to click the "info" button in the e2display browser window, and look for the mean and sigma entries, and see if they are around 0 and 1. Sometimes just making a copy of the file might fix it too. "e2proc3d.py r3d_00/threed_00.hdf r3d_00/threed_00_test.hdf" 

Muyuan

On Sep 12, 2024, at 10:08 AM, Clotho <cloth...@gmail.com> wrote:

Hi, I just tested e2gmm on my own dataset. It is quite strange that the proper isosurface threshold value for the threed_00.hdf  generated by  e2convertrelion cmd would reach ~2000 in e2display. If I typed in "--thr  2000" for e2gmm_guess_n.py , the program reported error as follow:

dm=np.sort(d)[len(d)//2]

IndexError: index 0 is out of bounds for axis 0 with size 0

<Screenshot from 2024-09-12 16-47-17.png>

And the output log for the e2convertrelion cmd was displayed  here. Are there any difference from your case?

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/779f7874-9f55-4d56-af84-ae1814438f84n%40googlegroups.com.
<Screenshot from 2024-09-12 16-47-17.png><Screen Shot 2024-09-13 at 01.05.29.png>

[Clotho]

Hi Muyuan,

I checked the info of "threed_00.hdf" and the value of mean and sigma is 0.025 and 1, respectively. I also make a copy using provided command and the threshold value seems to be normal this time. It is quite strange. 

 BTW, still, the threed_00.hdf  looks like a raw map without being sharpened. I just want to know how to determine that the map has been sharpened ?  Can I use e2spt_structure.py command to sharpen this map?

Sincerely Yours,

[Muyuan Chen]

Yes I find it quite strange too. There has been quite a few strange things lately after the python upgrade. Hopefully they get fixed soon...

It is probably sharpened, but the default sharpening parameters may not be as aggressive as some other software. Running e2spt_structfac with --sharper will probably help. 

That said, the sharpening only affect the visual appearance but not the alignment results. So you can run the sharpening in the end, then re-run the e2refine_postprocess with the new sf.txt and the threed_raw files (i.e. copy the threed_raw_even to the last threed_0X_even, and rerun the last e2refine_postprocess command). Or just bring it to other software for sharpening, since there are quite a few protein feature based fancy sharpening tools out there as well. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/bd19c202-d65f-4b41-b06d-621ec05a69fan%40googlegroups.com.

[Ludtke, Steven J.]

As far as I know none of the C++ code should have been impacted by the newer dependencies. Jumping from TensorFlow 2.12 to 2.16 would be the most likely candidate for any problems, but if you want me to help fix things I don't run across myself, I'll need more details than "quite a few strange things"...  

If you find something that seems broken a simple example of how to demonstrate/replicate the problem would help massively in getting it fixed.

---

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 
Deputy Director, Advanced Technology Cores                  and Molecular Pharmacology
Academic Director, CryoEM Core
Co-Director CIBR Center

On Sep 13, 2024, at 12:14 PM, Muyuan Chen <g5v...@gmail.com> wrote:

Yes I find it quite strange too. There has been quite a few strange things lately after the python upgrade. Hopefully they get fixed soon...

It is probably sharpened, but the default sharpening parameters may not be as aggressive as some other software. Running e2spt_structfac with --sharper will probably help. 

That said, the sharpening only affect the visual appearance but not the alignment results. So you can run the sharpening in the end, then re-run the e2refine_postprocess with the new sf.txt and the threed_raw files (i.e. copy the threed_raw_even to the last threed_0X_even, and rerun the last e2refine_postprocess command). Or just bring it to other software for sharpening, since there are quite a few protein feature based fancy sharpening tools out there as well. 

On Fri, Sep 13, 2024 at 8:24 AM Clotho <cloth...@gmail.com> wrote:

Hi Muyuan,

I checked the info of "threed_00.hdf" and the value of mean and sigma is 0.025 and 1, respectively. I also make a copy using provided command and the threshold value seems to be normal this time. It is quite strange. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqGYKO%3DTDiKVCA_ZYAN%2BPU-pgStRDd%3DOR57%3Due_Fhpy8%2BQ%40mail.gmail.com.

[Steve Ludtke]

Sorry for the slow reply here, it seemed like you were mostly interacting with Muyuan, so I wasn’t interfering. While you did post the warning messages, you didn’t post the command you executed which produced the message, making the output a bit useless...

Most of the messages you showed are just normal debugging output produced by TensorFlow, and don’t indicate anything wrong. The only exception is the warning about casting a complex64 to a float32, which is certainly a bit troubling in most contexts, but since I don’t know what program you were running, with what options, it’s impossible to figure out what might be causing that.

Regarding your earlier comment about isosurface thresholds, I’m also puzzled. If you open an EMAN2 compressed HDF in Chimera/X, the threshold will, indeed, be messed up due to the way the compression is handled. Chimera can read the raw data, but it will show up as integer values rather than floating point, and thus the threshold will be much higher. If you show it in e2display, however, that shouldn’t be an issue, unless you are using a very old version of e2display predating our support of compression from several years ago...

 

On Sep 24, 2024, at 10:58 PM, Clotho <cloth...@gmail.com> wrote:

Sorry for reply late. 

I have posted the tensorflow warning snaphot here:

<Screenshot from 2024-09-25 11-14-51.png>

Also, I have checked my eman2 envrionment. The cuda version is 12.3. The version of tensorflow and numpy is 2.16.1 and 1.26.4, respectively. 

<Screen Shot 2024-09-25 at 11.47.21.png>

If you need more detailed information, please feel free to ask here. Thank you again for your reply.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/82267d3a-e19b-46f1-866e-d6c2f51e5e71n%40googlegroups.com.
<Screen Shot 2024-09-25 at 11.47.21.png><Screenshot from 2024-09-25 11-14-51.png>