Is there any way to run mpirun command inside container for multi-host system?

[Guohua Li]

For our HPC system, really need run mpirun command inside the container on multi-host. 

How can we fix the issues? 

In my experience, when I build mpi job on multihost, everytime I change the version of MPI inside the container, I have to change the version of MPI on the host. 

Is there any solution for run mpi command inside the container on multi-host system? 

[Gregory M. Kurtzer]

Running a multi-node job from within the container carries many caveats that are difficult to resolve easily. It maybe possible using your resource manager to spin up the contained remote node MPI process daemons (for example ORTE in the case of OMPI), but that would require direct integration with the resource manager.

But with that said, the hybrid mode should work even with different versions. What MPI and versions are you trying to run?

Thanks!

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

--

Gregory M. Kurtzer

CEO, SingularityWare, LLC.

Senior Architect, RStor

Computational Science Advisor, Lawrence Berkeley National Laboratory

[Stefan Kombrink]

I've got some experience on running multi node jobs with mpirun inside the containers.
Greg is right that it will be difficult to find an general purpose solution for arbitary HPC systems and software.
I made it work for two heavily used applications on our cluster one of which used IntelMPI and the other OpenMPI 1.6
The main thing was to replace /bin/ssh in order to tweak mpirun to wrap "ssh hostname cmd" to "ssh 'singularity ContainerName exec cmd'
I had to apply some more workarounds to make IB work properly (mainly envvars to configure MPI properly) but essentially the efford was justifiable.

Sorry cant find the link to the ssh wrapper script right now but I can post it later when you are interested in it. It is really just a few lines of bash code.

Stefan

[Guohua Li]

Really thanks for your reply! 

Please post the script when you find it. 

Now I'm testing every MPI version which host installed MPI version can support much MPI versions inside the container! 

Hope your script can solve our problem!

영원히 살 것 처럼 꿈꾸고 내일 죽을 것 처럼 살자!  

This too shall pass away!

※ 이국화 LI GUOHUA
※ 건국대학교 대학원 신기술융합학과 박사과정

※ 융합 컴퓨팅 연구실 (IC Lab)
※ iIT, Department of Advanced Technology Fusion, Konkuk University
※ Industry-University Cooperation Bldg. 805

※ KonKuk University, Hwayang-dong, Gwangjin-Gu, Seoul 143-701 South Korea
※ Cell Phone: +82) 10-3666-8263
※ E-mail: heavenkong@gmail.com

[David Godlove]

I'd also like to have a look at that script and know more about the env vars you tweaked. Thanks Stefan.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Stefan Kombrink]

Here is the wrapper script which replaces ssh: https://pastebin.com/vqXXRzb5

Am Samstag, 10. Juni 2017 11:17:04 UTC+2 schrieb Guohua Li:

Really thanks for your reply! 

Please post the script when you find it. 

Now I'm testing every MPI version which host installed MPI version can support much MPI versions inside the container! 

Hope your script can solve our problem!

영원히 살 것 처럼 꿈꾸고 내일 죽을 것 처럼 살자!  

This too shall pass away!

※ 이국화 LI GUOHUA
※ 건국대학교 대학원 신기술융합학과 박사과정

※ 융합 컴퓨팅 연구실 (IC Lab)
※ iIT, Department of Advanced Technology Fusion, Konkuk University
※ Industry-University Cooperation Bldg. 805

※ KonKuk University, Hwayang-dong, Gwangjin-Gu, Seoul 143-701 South Korea
※ Cell Phone: +82) 10-3666-8263

※ E-mail: heavenkong...@gmail.com

2017-06-10 18:07 GMT+09:00 'Stefan Kombrink' via singularity <singu...@lbl.gov>:

I've got some experience on running multi node jobs with mpirun inside the containers.
Greg is right that it will be difficult to find an general purpose solution for arbitary HPC systems and software.
I made it work for two heavily used applications on our cluster one of which used IntelMPI and the other OpenMPI 1.6
The main thing was to replace /bin/ssh in order to tweak mpirun to wrap "ssh hostname cmd" to "ssh 'singularity ContainerName exec cmd'
I had to apply some more workarounds to make IB work properly (mainly envvars to configure MPI properly) but essentially the efford was justifiable.

Sorry cant find the link to the ssh wrapper script right now but I can post it later when you are interested in it. It is really just a few lines of bash code.

Stefan

Am Donnerstag, 8. Juni 2017 15:17:30 UTC+2 schrieb Guohua Li:

For our HPC system, really need run mpirun command inside the container on multi-host. 

How can we fix the issues? 

In my experience, when I build mpi job on multihost, everytime I change the version of MPI inside the container, I have to change the version of MPI on the host. 

Is there any solution for run mpi command inside the container on multi-host system? 

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Gregory M. Kurtzer]

Hi Stefan,

This is an interesting idea, and we might be able to easily include this in a container by a bind mount. Would you be willing to submit and maintain this in Singularity itself?

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Stefan Kombrink]

Hi Greg,

 I thought about it more:
I also had to bind mount /etc/ssh.
There's a caveat I haven't thought about yet: when ssh is wrapped it is possible to break out running containers just by calling

ssh_orig localhost '/tmp/run_this_code.sh'

Then, sudo and all setuid programs are accessible on the host.
In our case it does not really matter, but it is good to have that in mind!

So maybe I'll first write a tutorial. What are your thoughts on this?

Stefan

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Stefan Kombrink]

And the script works for 2.2.1 only :( - I'll have to see whether I can get the required info to port it to 2.3
SINGULARITY_IMAGE does not exist anymore, is that right?

[Stefan Kombrink]

Nope. Still works. I needed to put

SINGULARITYENV_SINGULARITY_IMAGE="$SINGULARITY_IMAGE"
export SINGULARITYENV_SINGULARITY_IMAGE

into /etc/singularity/init

[Guohua Li]

Thank you so much! the wrapper script is working well with small modification! 

Thank you so much! 

영원히 살 것 처럼 꿈꾸고 내일 죽을 것 처럼 살자!  

This too shall pass away!

※ 이국화 LI GUOHUA
※ 건국대학교 대학원 신기술융합학과 박사과정

※ 융합 컴퓨팅 연구실 (IC Lab)
※ iIT, Department of Advanced Technology Fusion, Konkuk University
※ Industry-University Cooperation Bldg. 805

※ KonKuk University, Hwayang-dong, Gwangjin-Gu, Seoul 143-701 South Korea
※ Cell Phone: +82) 10-3666-8263

※ E-mail: heavenkong@gmail.com

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[Gregory M. Kurtzer]

We need help with recipes and documentation! Can one of you guys post a PR including how this is done to our website GitHub?

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

[vanessa s]

Or just copy paste into an email here with good description, etc, and I'd be happy to do the docs part.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

--

Gregory M. Kurtzer

CEO, SingularityWare, LLC.

Senior Architect, RStor

Computational Science Advisor, Lawrence Berkeley National Laboratory

--
You received this message because you are subscribed to the Google Groups "singularity" group.

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.

--

Vanessa Villamia Sochat
Stanford University '16

(603) 321-0676

[Stefan Kombrink]

Hi Vanessa,

 there's a recipe and you can glance at the code.
Should serve well as starting point for documentation:

https://github.com/katakombi/multinode_sshwrapper

Stefan

Am Samstag, 29. Juli 2017 04:22:35 UTC+2 schrieb vanessa:

Or just copy paste into an email here with good description, etc, and I'd be happy to do the docs part.

On Fri, Jul 28, 2017 at 8:05 PM, Gregory M. Kurtzer <gmku...@gmail.com> wrote:

We need help with recipes and documentation! Can one of you guys post a PR including how this is done to our website GitHub?

Thanks!

On Mon, Jul 17, 2017 at 6:32 PM, Guohua Li <heave...@gmail.com> wrote:

Thank you so much! the wrapper script is working well with small modification! 

Thank you so much! 

영원히 살 것 처럼 꿈꾸고 내일 죽을 것 처럼 살자!  

This too shall pass away!

※ 이국화 LI GUOHUA
※ 건국대학교 대학원 신기술융합학과 박사과정

※ 융합 컴퓨팅 연구실 (IC Lab)
※ iIT, Department of Advanced Technology Fusion, Konkuk University
※ Industry-University Cooperation Bldg. 805

※ KonKuk University, Hwayang-dong, Gwangjin-Gu, Seoul 143-701 South Korea
※ Cell Phone: +82) 10-3666-8263

※ E-mail: heavenkong...@gmail.com

To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[vanessa s]

ah yes, will do! I know this container well :) Thanks Stefan!

To unsubscribe from this group and stop receiving emails from it, send an email to singularity+unsubscribe@lbl.gov.