mpirun on multiple gpus

[jude vishnu]

Dear all,

I am using hoomd 3.3. I wanted to know how I can run simulations on multiple gpu's using mpirun. I read the tutorials where running simulation on multiple cpu's is mentioned. Can we do the same by changing the device to gpu ? Can someone give me an example scipt ?

Regards,

Jude

[Russell Kajouri]

Hi there

To run simulation by mpirun, you can follow up the below and read the tutorial of mpi. MPI is a independent package of HOOMD.

mpirun -np <number of process> <python script> --mode=gpu

Best

The message has been sent by phone

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/2ba3633a-04db-4289-b959-668a9ddd5fa1n%40googlegroups.com.

[jude vishnu]

You are referring to version 2. I don't think hoomd 3.3 works this way.

Regards,

Jude

[Joshua Anderson]

Jude,

Follow the steps in the tutorial and/or those appropriate to launching a MPI application on your system, but use a GPU device instead of a CPU.

https://hoomd-blue.readthedocs.io/en/v3.5.0/tutorial/03-Parallel-Simulations-With-MPI/00-index.html

And yes, the documentation states that multiple GPUs are supported:

https://hoomd-blue.readthedocs.io/en/v3.5.0/features.html#mpi

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/f6a8a249-da7d-47df-b4c9-9c7737d514d2n%40googlegroups.com.

[jude vishnu]

Dear Joshua,

I did what you asked me to. I get the following error when I switch from cpu to gpu device. 

The code is attached below with initial config. If you can let me know why this is happening this would be great.

Regards,

Jude

[Joshua Anderson]

Jude,

Your script runs fine for me:

mpirun -n 2 python lj_perf.py
notice(2): Using domain decomposition: n_x = 1 n_y = 1 n_z = 2.
1861.9754815068595
1861.9754815068595

Did you build with ENABLE_MPI_CUDA=on by chance? That code path is buggy and should not be used, leave it on the default setting of off.

If off, share your `CMakeCache.txt` file so we can see how you compiled HOOMD.

Other than that, I can't think of anything specific. We test numerous multi-GPU executions with MPI on every commit to the repository. For example: https://github.com/glotzerlab/hoomd-blue/actions/runs/3151866395/jobs/5126702861

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

--
You received this message because you are subscribed to the Google Groups "hoomd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoomd-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/c4ffc09a-3a90-4c6a-854f-8ad340784b58n%40googlegroups.com.
<random.gsd><lj_perf.py><error.txt>

[jude vishnu]

 prep_config_diamond_network_diffusive_Ntot_1155...

Hi,

I installed with hoomd3.5 with ENABLE_MPI_CUDA=OFF and it worked for the previous code. However now I am trying to run a different code using mpi on gpu's but it results in errors which I do not fully understand. I execute it by mpirun -n 3 python3 diff_sim.py and this results in errors. 

It seems to run fine when executing with mpirun -n 2 python3 diff_sim.py. Is this a bug or is the particle density not supporting this kind of decomposition?

Regards,

Jude

You received this message because you are subscribed to a topic in the Google Groups "hoomd-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hoomd-users/CewY3g3gH5w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hoomd-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/E97E8053-41BF-40B9-B0B7-C3824CE349F2%40umich.edu.

--

Yours truly,

Jude Ann Vishnu

[jude vishnu]

Hai,

I just wanted to know whether someone was able to figure this out?

Regards,

Jude

[Joshua Anderson]

Jude,

It is not possible to answer the question given the information you provided.

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/149438e7-6cb0-4861-b7a2-d62bd99dab5an%40googlegroups.com.

[jude vishnu]

Hi Joshua,

Could you please tell me what other additional information I can provide so that this problem can be solved?

Regards,

Jude

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/9D128B0E-9967-4F65-93DC-8BB574430BB9%40umich.edu.

[Joshua Anderson]

Jude,

Jude, you ask "Is this a bug or is the particle density not supporting this kind of decomposition?"

To evaluate whether or not this is a bug, I would need to run your script. You did not provide the file prep_config_diamond_network_diffusive_Ntot_1155108_vol_frac0.84991_densratio_1.52410_molfrac_0.60382_kT_4.3.gsd needed to run the script.

To evaluate whether the particle density is supported, I would need to know the box dimensions and number of particles.

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

On Oct 11, 2022, at 9:35 AM, jude vishnu <mohms...@gmail.com> wrote:

Hi Joshua,

Could you please tell me what other additional information I can provide so that this problem can be solved?

Regards,

Jude

On Tue, Oct 11, 2022 at 2:44 PM Joshua Anderson <joaa...@umich.edu> wrote:

Jude,

It is not possible to answer the question given the information you provided.

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

On Oct 11, 2022, at 7:33 AM, jude vishnu <mohms...@gmail.com> wrote:

Hai,

I just wanted to know whether someone was able to figure this out?

Regards,

Jude

On Monday, October 3, 2022 at 11:15:55 PM UTC+2 jude vishnu wrote:

 prep_config_diamond_network_diffusive_Ntot_1155...

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/CAKDDmPCMW6Rs2Gavbad4pmnx5XXuV-jm%3Dus69PJ%3DqMTp0u2N0w%40mail.gmail.com.

[jude vishnu]

Dear Joshua,

I did share the initial configuration, just like I mentioned in my mail earlier. It was shared through google drive to this group. If you are finding trouble in accessing this please let me know.

I am also putting the initial configuration again here so that you can see it. Basically all hoomd-group members should be able to access it. 

https://drive.google.com/file/d/1qme7n4dtdF5LqLYU8zVQTOyxehfa88DB/view?usp=sharing

I hope this is enough.

Regards,

Jude

[Joshua Anderson]

Jude,

I did not see that link in your message.

Running your script, I get the same error. Running with the device notice_level=10 gives this additional information at the end of the output:

notice(6): nlist: (Re-)allocating neighbor list, new size 67296884036 uints 
notice(7): GPUArray: Resizing to 261624 MB

This appears to be a bug in the Tree neighbor list, as it is allocating space for far more neighbors than your system should have. I suggest you use the Cell neighbor list. With it, your script runs to completion on my system. You have a single r_cut and your system volume is reasonable so Tree offers no advantages over Cell.

------

Joshua A. Anderson, Ph.D.

Research Area Specialist, Chemical Engineering, University of Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/hoomd-users/fd6c8c58-3403-4e40-a912-d74594640d4en%40googlegroups.com.

[jude vishnu]

Thanks Joshua. I will try this out.

Regards,

Jude