[slurm-users] container on slurm cluster
GHui
So I use the tom account submit a jack's job.
Any help will be appreciated.
--GHui
GHui
Ole Holm Nielsen
http://slurm.schedmd.com/quickstart_admin.html
/Ole
Hermann Schwärzler
I have a few questions regarding your mail:
* What kind of container are you using?
* How exactly do you switch to a different user inside the container?
Regards,
Hermann
Marcus Wagner
If on bare metal I do a sudo -u <newuser> and then submit a job, I would expect that to be submitted as newuser, and not as the old one.
Best
Marcus
Dipl.-Inf. Marcus Wagner
IT Center
Gruppe: Server, Storage, HPC
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wag...@itc.rwth-aachen.de
www.itc.rwth-aachen.de
Social Media Kanäle des IT Centers:
https://blog.rwth-aachen.de/itc/
https://www.facebook.com/itcenterrwth
https://www.linkedin.com/company/itcenterrwth
https://twitter.com/ITCenterRWTH
https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ
GHui
I run container on my host with username rsync. And it only has itself privilege.
I create the same username, UID and GID in container with the host.
I run "podman exec -it <container> /bin/bash" to login with host user rsync. And the user is root on container.
Now I submit job with root in container. And job is running on host. And the job's owner is root.
So I submit a job with user rsync, but it runs as root privilege.
On 5/16/22 7:53 AM, GHui wrote:
> I fount a serious problem. If I run a container on a common user, eg. tom. In container I switch user to jack, now, if I submit a job to slurm cluster, the job owner is jack.
> So I use the tom account submit a jack's job.
>
> Any help will be appreciated.
> --GHui</container>
Brian Andrus
I suggest you check out Singularity, which was built from the ground up
to address most issues. And it can run other container types (eg: docker).
Brian Andrus
Hermann Schwärzler
fyi: I am not a podman-expert so my questions might be stupid. :-)
From what you told us so far you are running the podman-command as
non-root but you are root inside the container, right?
What is the output of "podman info | grep root" in your case?
How are you submitting a job from inside the container?
You have to have Slurm and munge installed and correctly configured
inside the container for that, right?
Why are you using such a container in the first place?
Regards,
Hermann
Timo Rothenpieler
> You are starting to understand a major issue with most containers.
>
> I suggest you check out Singularity, which was built from the ground up
> to address most issues. And it can run other container types (eg: docker).
>
> Brian Andrus
the same thing.
Documentation and Google results are wildly split between the two.
Stephan Roth
still maintained. Sylabs forked Singularity and kept the name.
So there are effectively 3 versions around for the time being:
https://github.com/apptainer/singularity (3.8 Series)
https://github.com/apptainer/apptainer (Successor to the original)
https://github.com/sylabs/singularity (SingularityCE, fork from Sylabs)
Also see: https://apptainer.org/news/community-announcement-20211130/
Personal note: I'm not sure what I'd choose as a successor to
Singularity 3.8, yet. Thoughts are welcome.
Stephan
GHui
Yes, I run podman as non-root, and use the root inside the container.
But the root in container has only privilege non-root which I run podman-command.
[]$ podman info | grep root
rootless: true
I exec "sbatch sub.sh" to submit inside the container.
I had config the right slurm and munge inside the container.
I just want to run some software in container, in order to get out of host OS environment.
On a whim, I config the slurm on container and submit a job.
On 5/17/22 16:10:55, Hermann wrote:
> fyi: I am not a podman-expert so my questions might be stupid. :-)
> From what you told us so far you are running the podman-command as
> non-root but you are root inside the container, right?
> What is the output of "podman info | grep root" in your case?
> How are you submitting a job from inside the container?
> You have to have Slurm and munge installed and correctly configured
> inside the container for that, right?
> Why are you using such a container in the first place?
>
>> On 5/17/22 7:49 AM, GHui wrote:
>> I use podman 4.0.2. And slurm 21.08.8-2.
>> I run container on my host with username rsync. And it only has itself privilege.
>> I create the same username, UID and GID in container with the host.
>> I run "podman exec -it <container>> /bin/bash" to login with host user rsync. And the user is root on container.
>> Now I submit job with root in container. And job is running on host. And the job's owner is root.
>> So I submit a job with user rsync, but it runs as root privilege.
>>
>> On 5/16/22 7:53 AM, GHui wrote:
>>> I fount a serious problem. If I run a container on a common user, eg. tom. In container I switch user to jack, now, if I submit a job to slurm cluster, the job owner is jack.
>>> So I use the tom account submit a jack's job.
>>>
>>> Any help will be appreciated.
>>> --GHui</container>
GHui
I think the main poblem is that container can cheat Slurm.
On 5/17/22 06:58:20, Brian Andrus wrote:
> You are starting to understand a major issue with most containers.
> I suggest you check out Singularity, which was built from the ground up
> to address most issues. And it can run other container types (eg: docker).
> Brian Andrus
>
>> On 5/17/22 7:49 AM, GHui wrote:
>> I use podman 4.0.2. And slurm 21.08.8-2.
>> I run container on my host with username rsync. And it only has itself privilege.
>> I create the same username, UID and GID in container with the host.
>> I run "podman exec -it container-id /bin/bash" to login with host user rsync. And the user is root on container.
Markus Kötter
On 18.05.22 08:25, Stephan Roth wrote:
> Personal note: I'm not sure what I'd choose as a successor to
> Singularity 3.8, yet. Thoughts are welcome.
enroot does unprivileged sandboxes/containers, pyxis is the slurm SPANK
glue.
https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf
https://github.com/NVIDIA/enroot
https://github.com/NVIDIA/pyxis
Notes from operation:
I recommend using nvme for the container storage, default configuration
uses tmpfs.
> …/enroot.conf
…
> ENROOT_RUNTIME_PATH /tmp/enroot/user-$(id -u)
> ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)
tmpfiles.d/…
> d /tmp/enroot/ 0777 root root - -
> d /tmp/enroot-data/ 0777 root root - -
> d /tmp/pyaxis-runtime/ 0777 root root - -
> dest: /etc/slurm/plugstack.conf.d/pyxis.conf
> content: |
> required /usr/lib/x86_64-linux-gnu/slurm/spank_pyxis.so runtime_path=/tmp/pyxis-runtime/
Time savers:
The container url formatting is … unexpected, # is used as seperator for
the path in host:port#path/file - and may need to be escaped to avoid
getting interpreted as comment.
It uses a netrc formatted .credentials file for container registries
with authentication.
Insert the credentials twice - with and without port.
Can do more than documented. (e.g. #SBATCH --container-image)
MfG
--
Markus Kötter, +49 681 870832434
30159 Hannover, Lange Laube 6
Helmholtz Center for Information Security
Josef Dvoracek
Who has access to munge.key can effectively became root at slurm cluster.
you should not disclose munge.key to containers.
cheers
josef
On 18. 05. 22 9:13, GHui wrote:
> ...I had config the right slurm and munge inside the container.
>
...
Brian Andrus
It seems that things are doing what they should.
You are allowing an account to become root inside the pod and the pod is
considered a trusted environment by slurm (you are running munge inside it).
So as far as slurm is concerned, 'root' from a trusted environment is
submitting a job.
Which circles back to the big issue with most container environments:
security.
You should not allow full access to root inside your container (or other
environments). Use sudoers and be highly restrictive with what commands
are allowed.
If that is not possible, you need to trust the people you give the
permission to that they will not abuse it.
Brian Andrus