Can you choose which GPU and/or multiple GPUs to use with e2gmm.py?

[Maia Azubel]

Hi Steve,

Is it possible to choose GPU and/or use multiple GPUs with e2gmm.py?

>e2version.py

EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )

Your EMAN2 is running on: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8 4.18.0-147.8.1.el8_1.x86_64

Your Python version is: 3.7.9

Thanks,
Maia

[Ludtke, Steven J.]

Nope, not without editing the code slightly. The code at the beginning of all of the GMM refinement programs dates back to the early days of Tensorflow, and device management has changed a lot since then. If you look at e2gmm_refine_point.py, you should be able to pretty easily adapt a way to force it to use a specific GPU. We're revamping how this is handled in the work towards EMAN3, as part of a complete overhaul of the NumPy and Tensorflow in EMAN.

---

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 
Deputy Director, Advanced Technology Cores                  and Molecular Pharmacology
Academic Director, CryoEM Core
Co-Director CIBR Center

-- 
-- 
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

--- 
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB66570949693379ECB39291A4DF592%40BY5PR02MB6657.namprd02.prod.outlook.com.

[Muyuan Chen]

Maybe just add a simple fix for e2gmm now? 

In other programs I use 

if "CUDA_VISIBLE_DEVICES" not in os.environ:

os.environ["CUDA_VISIBLE_DEVICES"]='0’ 

So at least users can overwrite the environment variable in terminal before running the program. It is not intuitive but works. 

Muyuan

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/B6460B42-E116-4D26-A040-CF4F280A7506%40bcm.edu.

[Andrea Dallapé]

have there been improvements on this front? Is it possible to use multiple GPUs or is it necessary to wait for the new version?

This could be a game changer.

Thank you,

Andrea

[Muyuan Chen]

I don't think Steve changed e2gmm.py for GPU specification, and I generally don't make changes to his programs. The change is easy to make yourself locally too.

https://github.com/cryoem/eman2/blob/master/programs/e2gmm.py#L62

Just change 0 to any other GPU ID, or comment out this line and specify GPU ID by the environment variable before running the program

As for multi-GPU training. it hasn't happened yet. I was hoping tensorflow will have a simple way to do multi-gpu distribution, or me finding a collaborator who does GPU optimization. Both are sort of happening but neither at a point that an easy-to-use product is produced. It is just a lot of work but not too useful for me since I don't have many GPUs. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/c0f1047f-7d8b-4b17-8477-db927c74e8b9n%40googlegroups.com.

[Ludtke, Steven J.]

Hi Andrea,

I have made the change to allow the “CUDA_VISIBLE_DEVICES” environment variable work properly in e2gmm.py and e2gmm_refine_point.py. It is now on github (ie - if you install from source, you can do a git pull and make install to get the update). 

To use it, you need to export CUDA_VISIBLE_DEVICES in the shell before running e2gmm.py. I haven’t tested it, but it’s a pretty simple change, so should be fine. 

This isn’t included in the GUI because the way things are handled, the GPU is set as the program is launched, before the GUI is loaded. Things will be different in EMAN3

---

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 
Deputy Director, Advanced Technology Cores                  and Molecular Pharmacology
Academic Director, CryoEM Core
Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqEpc3Z4S%2B7n0O_dh-WvtWAVS0J1Rx-ZZS5DGMqs2zs7ng%40mail.gmail.com.

[Andrea Dallapé]

Hi Steve,

thank you so much for your prompt modification, as usual.

This is for deciding what GPU to use, not to use multiple GPUs, right? Or am I missing something?

Thank you.

Best,

Andrea

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/736C76B3-DC16-4438-A663-C07BF73EDE7B%40bcm.edu.

[Steve Ludtke]

Ahh, sorry, I didn’t realize what you were asking.  There is no reasonable strategy for using multiple GPUs simultaneously on one dynamics job at present. This isn’t a small effort, it would require a dedicated programmer a significant amount of time to get working. While Tensorflow is beginning to support multi-device training, this only works at a very high level, and some of the operations we’re using (like the Fourier ring correlation, and making projections from Gaussian centers) doesn’t fit well into that generic framework right now. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CALnnSmd55Vu_WC-Fc74qQH0Y3eiTyFWEmrhQqVKiBV%3DbZys%2BMg%40mail.gmail.com.

[Andrea Dallapé]

Understood, thank you so much for the clarification.

I hope this would be easier to implement in the future.

In the meantime, I will follow Muyuan’s suggestion using a subtracted region of interest, it is working very well and it is not crashing because of OOM.

Congratulations for the tools.

Best,

Andrea

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/0EC4842E-0E5D-452D-92F8-0ED3C4B3BCFE%40gmail.com.