[slurm-users] Exclude Slurm packages from the EPEL yum repository

[Ole Holm Nielsen]

We use the EPEL yum repository on our CentOS 7 nodes. Today EPEL
surprisingly delivers Slurm 20.11.2 RPMs, and the daily yum updates
(luckily) fail with some errors:

--> Running transaction check
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package:
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package:
slurm-libpmi-20.02.6-1.el7.x86_64
---> Package slurm.x86_64 0:20.11.2-2.el7 will be an update
--> Processing Dependency: pmix for package: slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libfreeipmi.so.17()(64bit) for package:
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libipmimonitoring.so.6()(64bit) for package:
slurm-20.11.2-2.el7.x86_64
--> Processing Dependency: libslurmfull-20.11.2.so()(64bit) for package:
slurm-20.11.2-2.el7.x86_64
---> Package slurm-contribs.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-contribs.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-devel.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-devel.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-perlapi.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-perlapi.x86_64 0:20.11.2-2.el7 will be an update
---> Package slurm-slurmdbd.x86_64 0:20.02.6-1.el7 will be updated
---> Package slurm-slurmdbd.x86_64 0:20.11.2-2.el7 will be an update
--> Running transaction check
---> Package freeipmi.x86_64 0:1.5.7-3.el7 will be installed
---> Package pmix.x86_64 0:1.1.3-1.el7 will be installed
---> Package slurm.x86_64 0:20.02.6-1.el7 will be updated
--> Processing Dependency: slurm(x86-64) = 20.02.6-1.el7 for package:
slurm-libpmi-20.02.6-1.el7.x86_64
--> Processing Dependency: libslurmfull.so()(64bit) for package:
slurm-libpmi-20.02.6-1.el7.x86_64
---> Package slurm-libs.x86_64 0:20.11.2-2.el7 will be installed
--> Finished Dependency Resolution
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64
(@/slurm-libpmi-20.02.6-1.el7.x86_64)
Requires: libslurmfull.so()(64bit)
Removing: slurm-20.02.6-1.el7.x86_64
(@/slurm-20.02.6-1.el7.x86_64)
libslurmfull.so()(64bit)
Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
Not found
Error: Package: slurm-libpmi-20.02.6-1.el7.x86_64
(@/slurm-libpmi-20.02.6-1.el7.x86_64)
Requires: slurm(x86-64) = 20.02.6-1.el7
Removing: slurm-20.02.6-1.el7.x86_64
(@/slurm-20.02.6-1.el7.x86_64)
slurm(x86-64) = 20.02.6-1.el7
Updated By: slurm-20.11.2-2.el7.x86_64 (epel)
slurm(x86-64) = 20.11.2-2.el7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest


We still run Slurm 20.02 and don't want EPEL to introduce any Slurm
updates!! Slurm must be upgraded with some care, see for example
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

Therefore we must disable EPEL's slurm RPMs permanently. The fix is to
add to the file /etc/yum.repos.d/epel.repo an "exclude=slurm*" line like
the last line in:

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
exclude=slurm*

/Ole

[Philip Kovacs]

I can assure you it was easier for you to filter slurm from your repos than it was for me to make them available to both epel7 and epel8.

No good deed goes unpunished I guess.

[Ian Mortimer]

On Sat, 2021-01-23 at 20:43 +0000, Philip Kovacs wrote:

> I can assure you it was easier for you to filter slurm from your
> repos than it was for me to make them available to both epel7 and
> epel8.

Thanks for your efforts. Much appreciated.


--
Ian

[Ryan Novosielski]

I agree that by and large it’s no big deal, but a suggestion might be to provide the SLURM as slurm-*-<version> being the set of packages you install, so that updating between major versions wouldn’t happen by surprise, given how careful one needs to be with SLURM upgrades — ordering, timing, etc. VirtualBox does something like that, and their upgrades aren’t even as disruptive. 

More work still, though, and I install SLURM via OpenHPC so I’m not a constituent necessarily. Just an idea. 

Thanks for your effort!

--

#BlackLivesMatter
____
|| \\UTGERS,       |---------------------------*O*---------------------------
||_// the State     |         Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ     | Office of Advanced Research Computing - MSB C630, Newark
    `'

On Jan 23, 2021, at 15:44, Philip Kovacs <pkd...@yahoo.com> wrote:



[Bjørn-Helge Mevik]

Thanks for the heads-up, Ole!

--
B/H

[Ole Holm Nielsen]

On 1/23/21 9:43 PM, Philip Kovacs wrote:
> I can assure you it was easier for you to filter slurm from your repos
> than it was for me to make them available to both epel7 and epel8.
>
> No good deed goes unpunished I guess.

I do sympathize with your desire to make the Slurm installation a bit
easier by providing RPMs via the EPEL repo. I do not underestimate the
amount of work it takes to add software to EPEL.

However, I have several issues with your approach:

1. Breaking existing Slurm installations could cause big time problems at
a lot of sites! The combined work to repair broken installations at many
sites might be substantial. Sites who are more than two releases behind
20.11 could end up with dysfunctional clusters. You are undoubtedly aware
that 20.11.3 fixes a major problem in 20.11.2 wrt. OpenMPI, so the upgrade
from 20.02 to 20.11.2 may cause problems.

2. Your EPEL RPMs *must not* upgrade between major Slurm releases, like
the 20.02 to 20.11 upgrade that almost happened at our site! I refer
again to the delicate upgrade procedure described in
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

3. You could have solicited advice from the slurm-users list before
planning your EPEL Slurm packages.

4. How do you plan to keep updating future Slurm minor versions on EPEL in
a timely fashion?

5. How did you build your RPM packages? The built-in options may be
important, for example, this might be recommended:
$ rpmbuild -ta slurm-xxx.tar.bz2 --with mysql --with slurmrestd

6. Building Slurm RPM packages is actually a tiny part of what it takes to
install Slurm from scratch. There are quite a number of prerequisites and
other things to set up besides the RPMs, see
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation
plus configuration of Slurm itself and its database.

In conclusion, I would urge you to ensure that your EPEL packages won't
mess up existing Slurm installations! I agree with Ryan Novosielski that
you should rename your RPMs so that they don't overwrite packages built by
SchedMD's rpmbuild system.

I propose that you add the major version 20.11 right after the "slurm"
name so that your EPEL RPMs would be named "slurm-20.11-*" like in:

slurm-20.11-20.11.2-2.el7.x86_64

People with more knowledge of RPM than I have could help you ensure that
no unwarranted upgrades or double Slurm installations can take place.

Thanks,
Ole

> <http://download.fedoraproject.org/pub/epel/7/$basearch>
> metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir

[Andy Riebs]

Several things to keep in mind...

1. Slurm, as a product undergoing frequent, incompatible revisions, is not well-suited for provisioning from a stable public repository! On the other hand, it's just not that hard to build a stable version where you can directly control the upgrade timing.
2. If you really want a closely managed source for your Slurm RPMs, get them from the SchedMD website.
   
3. "You could have solicited advice..." -- while this is certainly true, for many of us in the open source world, the standard is "release something quickly, and then improve it, based in part on feedback, over time."
   
4. Slurm packages (and other contributions, including suggestions on this mailing list) that haven't been provided by SchedMD have probably been provisioned and tested by a volunteer -- be sure to keep the conversation civil!

Andy Riebs

[mercan]

Hi;

We are using the yumlock feature of the yum to protect unwanted upgrade
of the some packages. Also, Ole mentioned "exclude=slurm" option of the
repo file. It is not a solutionless problem. But, the package maintainer
is a valued resource which hard to find.

Regards,

Ahmet M.


25.01.2021 16:59 tarihinde Andy Riebs yazdı:

>
> Several things to keep in mind...
>

> 1. Slurm, as a product undergoing frequent, incompatible revisions,

> is not well-suited for provisioning from a stable public
> repository! On the other hand, it's just not that hard to build a
> stable version where you can directly control the upgrade timing.

> 2. If you really want a closely managed source for your Slurm RPMs,

> get them from the SchedMD website.

> 3. "You could have solicited advice..." -- while this is certainly

> true, for many of us in the open source world, the standard is
> "release something quickly, and then improve it, based in part on
> feedback, over time."

> 4. Slurm packages (and other contributions, including suggestions on

[Ole Holm Nielsen]

On 1/25/21 2:59 PM, Andy Riebs wrote:
> Several things to keep in mind...
>

> 1. Slurm, as a product undergoing frequent, incompatible revisions, is

> not well-suited for provisioning from a stable public repository! On
> the other hand, it's just not that hard to build a stable version
> where you can directly control the upgrade timing.

I agree that Slurm is probably not well suited for a public repository
because of the special care that *must* be taken when upgrading between
major versions.

When I use both EPEL for a lot of nice software (Munge, Lmod, ...), AND I
build my own Slurm RPMs, now suddenly slurm RPMs from EPEL upsets this
stable scenario.

> 2. If you really want a closely managed source for your Slurm RPMs, get

> them from the SchedMD website.

All of us get the Slurm source from the SchedMD website. And all of us
have to build our own RPMs from that source (a simple one-liner). SchedMD
doesn't provide any RPMs.

> 3. "You could have solicited advice..." -- while this is certainly true,

> for many of us in the open source world, the standard is "release
> something quickly, and then improve it, based in part on feedback,
> over time."

I don't think this trial-and-error-like approach is suitable for Slurm.
We're running production HPC clusters that need to stay very stable.

> 4. Slurm packages (and other contributions, including suggestions on this

> mailing list) that haven't been provided by SchedMD have probably been
> provisioned and tested by a volunteer -- be sure to keep the
> conversation civil!

We all have to build our own Slurm RPMs, and we should not get them from a
volunteer. IMHO, building Slurm RPMs is very simple. It's the deployment
and upgrading which is the hard part of the equation.

I think my points quoted below deserve careful consideration by the EPEL
volunteer, because the results could be potentially harmful.

Thanks,
Ole

[Tina Friedrich]

Sorry. I really don't want this to be a flame war, but...

...I would say having SLURM rpms in EPEL could be very helpful for a lot
of people.

I get that this took you by surprise, but that's not a reason to not
have them in the repository. I, for one, will happily test if they work
for me, and if they do, that means that I can stop having to build them.
I agree it's not hard to do, but if I don't have to do it I'll be very
happy about that.

There are surely other packages in public repositories where 'this would
break things if upgraded' applies. For example, nvidia driver RPMs and
I'm immensely grateful to the people maintaining those packages. It
makes my live setting up GPU node much easier. However, it would break
my production clusters if I pulled upgrades to those without proper
planning. That does not make me say that nvidia drivers ought not to be
in a public repository, it makes me configure version locking for them,
so I control what version is installed & when they are upgraded.

I mean sure you do test your upgrade sets before you apply them to your
production machines?

Along the same lines, if I (and a lot of other sites) where to upgrade
to the latest kernel version in the Red Hat repos, that'd break my file
system. Doesn't mean I think Red Hat should stop packaging kernels.

I would very much say the same thing applies to SLURM packages.

Tina

--
Tina Friedrich, Advanced Research Computing Snr HPC Systems Administrator

Research Computing and Support Services
IT Services, University of Oxford
http://www.arc.ox.ac.uk http://www.it.ox.ac.uk

[Andy Riebs]

See below

On 1/25/2021 9:36 AM, Ole Holm Nielsen wrote:
> On 1/25/21 2:59 PM, Andy Riebs wrote:
>> Several things to keep in mind...
>>
>>  1. Slurm, as a product undergoing frequent, incompatible revisions, is
>>     not well-suited for provisioning from a stable public repository! On
>>     the other hand, it's just not that hard to build a stable version
>>     where you can directly control the upgrade timing.
>
> I agree that Slurm is probably not well suited for a public repository
> because of the special care that *must* be taken when upgrading
> between major versions.
>
> When I use both EPEL for a lot of nice software (Munge, Lmod, ...),
> AND I build my own Slurm RPMs, now suddenly slurm RPMs from EPEL
> upsets this stable scenario.
>

Interesting; this does sound like a problem.

>>  2. If you really want a closely managed source for your Slurm RPMs, get
>>     them from the SchedMD website.
>
> All of us get the Slurm source from the SchedMD website.  And all of
> us have to build our own RPMs from that source (a simple one-liner). 
> SchedMD doesn't provide any RPMs.

Yeah, I was a little embarrassed after sending the previous note when I
checked to see what they have on the site. I would have sworn that they
once provided RPMs, as well, but I could be mistaken. (With multiple
versions of Slurm for multiple versions of RedHat/Fedora, SLES,
Debian/Ubuntu/..., this would have become a very difficult problem to
handle.)

>
>>  3. "You could have solicited advice..." -- while this is certainly
>> true,
>>     for many of us in the open source world, the standard is "release
>>     something quickly, and then improve it, based in part on feedback,
>>     over time."
>
> I don't think this trial-and-error-like approach is suitable for
> Slurm. We're running production HPC clusters that need to stay very
> stable.
>
>>  4. Slurm packages (and other contributions, including suggestions on
>> this
>>     mailing list) that haven't been provided by SchedMD have probably
>> been
>>     provisioned and tested by a volunteer -- be sure to keep the
>>     conversation civil!
>
> We all have to build our own Slurm RPMs, and we should not get them
> from a volunteer. IMHO, building Slurm RPMs is very simple. It's the
> deployment and upgrading which is the hard part of the equation.
> I think my points quoted below deserve careful consideration by the
> EPEL volunteer, because the results could be potentially harmful.

I think you've raised some good points, but keep in mind that you're in
a community of thousands of people with multiple diverging requirements
who are taking advantage of free software. Moe and Danny set up SchedMD
precisely for users who need more consistent, reliable, and specific
support than might be available for free.

I'm serious about suggesting a contract with SchedMD. Though I've been
working with Slurm for nearly 20 years, I've always enjoyed the
technical challenges, and have managed to avoid needing a contract for
my own work. Though I'll confess, there have been several times that
I've taken advantage of the fact that my customer du jour had both a
problem and a contract with SchedMD. In those cases, SchedMD was always
responsive and helpful.

Regards,
Andy

[Jeffrey T Frey]

> ...I would say having SLURM rpms in EPEL could be very helpful for a lot of people.
>
> I get that this took you by surprise, but that's not a reason to not have them in the repository. I, for one, will happily test if they work for me, and if they do, that means that I can stop having to build them. I agree it's not hard to do, but if I don't have to do it I'll be very happy about that.

There have been plenty of arguments for why having them in EPEL isn't necessarily the best option. Many open source products (e.g. Postgres, Docker) maintain their own YUM repository online -- probably to exercise greater control over what's published, but also to avoid overlap with mainstream package repositories. If there is value perceived in having pre-built packages available, then perhaps the best solution for all parties is to publish the packages to a unique repository: those who want the pre-built packages explicitly configure their YUM to pull from that repository, those who have EPEL configured (which is a LOT of us) don't get overlapping Slurm packages interfering with their local builds.


::::::::::::::::::::::::::::::::::::::::::::::::::::::
Jeffrey T. Frey, Ph.D.
Systems Programmer V & Cluster Management
IT Research Cyberinfrastructure
& College of Engineering
University of Delaware, Newark DE 19716
Office: (302) 831-6034 Mobile: (302) 419-4976
::::::::::::::::::::::::::::::::::::::::::::::::::::::

[Brian Andrus]

That is basically how I do it.

I created a local repository for the packages I build (slurm and any
other, like openmpi). This provides as much control as I could possibly
need/want. I can name them how I like to avoid conflict with any
external repositories.

I think it is a good idea to have them in EPEL for so many folks that
just want to try the basic setup. This is how we get adoption by more
people. As they learn and want more, they can start building their own
with any options they desire.

Also, a plug for support contracts. I have been doing slurm for a very
long while, but always encourage my clients to get a support contract.
That is how SchedMD stays alive and we are able to have such a good
piece of software. I see the cloud providers starting to build tools
that will eventually obsolesce slurm for the cloud. I worry that there
won't be enough paying customers for Tim to keep things running as well
as he has. I'm pretty sure most folks that use slurm for any period of
time has received more value that a small support contract would be.

Brian Andrus

[gil...@rist.or.jp]

Folks,


Tina made excellent points on why EPEL packaging SLURM is a good thing
and I am not going to re-iterate them.
Instead, I acknowledge Philip Kovacs positive contribution to those who
simply hoped for a "hassle free single node SLURM cluster to start with"


For some reasons [I do not understand], some chose to translate "I do
not want to use EPEL SLURM
packages [on my cluster]" into "EPEL should not package SLURM".
Do not get me wrong, there are many legit reasons why one would not want
to use SLURM from EPEL, and the points have already been made. That
being said, the two previous statements are not logically equivalent,
and using local vs EPEL packages should in my opinion remain a site
policy and should not impact EPEL volunteers/packagers/maintainers.


As far as I am concerned, I believe using yum-plugin-priorities (
replaced by dnf from RHEL8) is a superior solution if one wants to
prefer using site packages (and this is not limited to SLURM) instead of
those provided by third party repositories.



Running "daily yum update" on "production HPC clusters that need to stay
very stable" is in my not so humble opinion a step in the opposite
direction.

A workflow that did not anticipate third party repositories (such as
EPEL) could start providing packages that conflict with packages built
on site (such as SLURM) is a workflow that was flawed since the
beginning.
This was recently evidenced by EPEL, so let's avoid blaming the
messenger:
[SLURM RPMs from] EPEL did not "upset" any "stable scenario".

Asking EPEL to rename the SLURM packages would cause some different
issues, for example when EPEL starts providing packages that depend on
SLURM.
But if one believes this is the best option, eating own dog food should
always be on the table: rename local SLURM packages (since SchedMD does
not provide any RPMs).
Regardless the technical aspects, and from a pure OSS philosophical
point of view, asking EPEL to make such changes for one's self
convenience seems pretty wrong to me.



Cheers,

Gilles

[Prentice Bisbal]

I think the crux of this issue isn't necessarily that EPEL is now
providing Slurm packages, but that when they were made available it took
an EPEL user by surprise.

EPEL maintains multiple mailing lists:

https://fedoraproject.org/wiki/EPEL#Communicating_with_the_EPEL_SIG

Searching through that list, I quickly found announcements that Slurm
202.11.2 was added to EPEL 7 and EPEL 8 on Friday evening:

https://lists.fedoraproject.org/archives/list/epel-packa...@lists.fedoraproject.org/message/2NOE6TYZKUCRYEY4Z754IGHAORRZG6SC/

https://lists.fedoraproject.org/archives/list/epel-packa...@lists.fedoraproject.org/message/KPO7IONLWK627CTAG2JWBEIOE7LCK7BQ/

These announcements indicate that this addition to EPEL was in response
to a user request:

https://bugzilla.redhat.com/show_bug.cgi?id=1912491

While I have to admit the timing of the announcements was very
unfortunate (it was the weekend, when no one would probably see the
announcements until Monday, when  it might have been too late to
intervene), I think we as system administrators need to take some
responsibility for the software we install on our systems, and if we use
EPEL, we should probably subscribe to EPEL's announcement lists to
reduce/minimize surprises like this. I admit I don't subscribe to EPEL's
mailing lists myself, but I'm about to subscribe right now as a result
of this discussion.

Also, I have to echo the comments from others in this discussion that
automatic updates on production systems is not a good idea. Yes, our
Cybersecurity departments want us to do it, but they're not the ones who
have to deal with applications that suddenly stop working when an
automatic update breaks something.

Also, I'm glad someone was able to find this problem before it caused an
issue for them or anyone else and they were able to warn the rest of us,
so thanks for the warning!

Personally, I think it's good that Slurm RPMs are now available through
EPEL, although I won't be able to use them, and I'm sure many people on
the list won't be able to either, since licensing issues prevent them
from providing support for NVIDIA drivers, so those of us with GPUs on
our clusters will still have to compile Slurm from source to include
NVIDIA GPU support.

Prentice

[Ole Holm Nielsen]

In another thread, On 26-01-2021 17:44, Prentice Bisbal wrote:
> Personally, I think it's good that Slurm RPMs are now available through
> EPEL, although I won't be able to use them, and I'm sure many people on
> the list won't be able to either, since licensing issues prevent them
> from providing support for NVIDIA drivers, so those of us with GPUs on
> our clusters will still have to compile Slurm from source to include
> NVIDIA GPU support.

We're running Slurm 20.02.6 and recently added some NVIDIA GPU nodes.
The Slurm GPU documentation seems to be
https://slurm.schedmd.com/gres.html
We don't seem to have any problems scheduling jobs on GPUs, even though
our Slurm RPM build host doesn't have any NVIDIA software installed, as
shown by the command:
$ ldconfig -p | grep libnvidia-ml

I'm curious about Prentice's statement about needing NVIDIA libraries to
be installed when building Slurm RPMs, and I read the discussion in bug
9525,
https://bugs.schedmd.com/show_bug.cgi?id=9525
from which it seems that the problem was fixed in 20.02.6 and 20.11.

Question: Is there anything special that needs to be done when building
Slurm RPMs with NVIDIA GPU support?

Thanks,
Ole

[Robert Kudyba]

On Mon, Jan 25, 2021 at 6:36 PM Brian Andrus <toom...@gmail.com> wrote:

Also, a plug for support contracts. I have been doing slurm for a very
long while, but always encourage my clients to get a support contract.
That is how SchedMD stays alive and we are able to have such a good
piece of software. I see the cloud providers starting to build tools
that will eventually obsolesce slurm for the cloud. I worry that there
won't be enough paying customers for Tim to keep things running as well
as he has. I'm pretty sure most folks that use slurm for any period of
time has received more value that a small support contract would be.

We considered this but we have a very small cluster. And when I reached out for a quote, I was told "SchedMD has a MIN node count of 256 for $10K/yr".

Since we're using Bright Computing we've always had to ignore Slurm updates from yum and have to compile our own version.

Curious, which cloud provider scheduling tools do you see gaining traction?

[Paul Edmon]

In our RPM spec we use to build slurm we do the following additional
things for GPU's:

BuildRequires: cuda-nvml-devel-11-1

the in the %build section we do:

export CFLAGS="$CFLAGS
-L/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs/
-I/usr/local/cuda-11.1/targets/x86_64-linux/include/"

That ensures the cuda libs are installed and it directs slurm to where
they are.  After that configure should detect the nvml libs and link
against them.

I've attached our full spec that we use to build.

-Paul Edmon-

[Jason Simms]

We’re in the same boat. Extremely small cluster. $10k for support. We don’t need nearly that level of engagement, but there ya go. We’ve passed for now, but I’d like to have a support contract ideally. 

Jason

--

Jason L. Simms, Ph.D., M.P.H.

Manager of Research and High-Performance Computing

XSEDE Campus Champion
Lafayette College
Information Technology Services
710 Sullivan Rd | Easton, PA 18042
Office: 112 Skillman Library
p: (610) 330-5632

[Ole Holm Nielsen]

Thanks Paul!

On 26-01-2021 20:50, Paul Edmon wrote:
> In our RPM spec we use to build slurm we do the following additional
> things for GPU's:
>
> BuildRequires: cuda-nvml-devel-11-1
>
> the in the %build section we do:
>
> export CFLAGS="$CFLAGS
> -L/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs/
> -I/usr/local/cuda-11.1/targets/x86_64-linux/include/"
>
> That ensures the cuda libs are installed and it directs slurm to where
> they are.  After that configure should detect the nvml libs and link
> against them.
>
> I've attached our full spec that we use to build.

What I don't understand is, is it actually *required* to make the NVIDIA
libraries available to Slurm? I didn't do that, and I'm not aware of
any problems with our GPU nodes so far. Of course, our GPU nodes have
the libraries installed and the /dev/nvidia? devices are present.

Are some of Slurm's GPU features missing or broken without the
libraries? SchedMD's slurm.spec file doesn't mention any "--with
nvidia" (or similar) build options, so I'm really puzzled.

Most of our nodes don't have GPUs, so I wouldn't like to install
libraries on those nodes needlessly.

Thanks,
Ole

[Paul Raines]

You should check your jobs that allocated GPUs and make sure
CUDA_VISIBLE_DEVICES is being set in the environment. This is a sign
you GPU support is not really there but SLURM is just doing "generic"
resource assignment.

I have both GPU and non-GPU nodes. I build SLURM rpms twice. Once on a
non-GPU node and use those RPMs to install on the non-GPU nodes. Then build
again on the GPU node where CUDA is installed via the NVIDIA CUDA YUM repo
rpms so the NVML lib is at /lib64/libnvidia-ml.so.1 (from rpm
nvidia-driver-NVML-455.45.01-1.el8.x86_64) and no special mods to the default
RPM SPEC is needed. I just run

rpmbuild --tb slurm-20.11.3.tar.bz2

You can run 'rpm -qlp slurm-20.11.3-1.el8.x86_64.rpm | grep nvml' and see
that /usr/lib64/slurm/gpu_nvml.so only exists on the one built on the
GPU node.

-- Paul Raines (http://help.nmr.mgh.harvard.edu)

[Robert Kudyba]

You all might be interested in a patch to the SPEC file, to not make the slurm RPMs depend on libnvidia-ml.so, even if it's been enabled at configure time. See https://bugs.schedmd.com/show_bug.cgi?id=7919#c3

On Tue, Jan 26, 2021 at 3:17 PM Paul Raines <rai...@nmr.mgh.harvard.edu> wrote:

You should check your jobs that allocated GPUs and make sure
CUDA_VISIBLE_DEVICES is being set in the environment.  This is a sign
you GPU support is not really there but SLURM is just doing "generic"
resource assignment.

I have both GPU and non-GPU nodes.  I build SLURM rpms twice. Once on a
non-GPU node and use those RPMs to install on the non-GPU nodes. Then build
again on the GPU node where CUDA is installed via the NVIDIA CUDA YUM repo
rpms so the NVML lib is at /lib64/libnvidia-ml.so.1 (from rpm
nvidia-driver-NVML-455.45.01-1.el8.x86_64) and no special mods to the default
RPM SPEC is needed.  I just run

   rpmbuild --tb slurm-20.11.3.tar.bz2

You can run 'rpm -qlp slurm-20.11.3-1.el8.x86_64.rpm | grep nvml' and see
that /usr/lib64/slurm/gpu_nvml.so only exists on the one built on the
GPU node.

-- Paul Raines (https://urldefense.proofpoint.com/v2/url?u=http-3A__help.nmr.mgh.harvard.edu&d=DwIBAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=GNGEhyc3F2bEZxbHK93tumXk56f37DOl99aYsOeUVOE&s=ZuCDM15RrOpv2t-j8DywWrwpn86qa79eBuSPEs96SFo&e= )

On Tue, 26 Jan 2021 2:29pm, Ole Holm Nielsen wrote:

> In another thread, On 26-01-2021 17:44, Prentice Bisbal wrote:
>>  Personally, I think it's good that Slurm RPMs are now available through
>>  EPEL, although I won't be able to use them, and I'm sure many people on
>>  the list won't be able to either, since licensing issues prevent them from
>>  providing support for NVIDIA drivers, so those of us with GPUs on our
>>  clusters will still have to compile Slurm from source to include NVIDIA
>>  GPU support.
>
> We're running Slurm 20.02.6 and recently added some NVIDIA GPU nodes.
> The Slurm GPU documentation seems to be

> https://urldefense.proofpoint.com/v2/url?u=https-3A__slurm.schedmd.com_gres.html&d=DwIBAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=GNGEhyc3F2bEZxbHK93tumXk56f37DOl99aYsOeUVOE&s=GxF9VoynMmgS3BBrWWsmPM1Itt0hshTIkGh3x4Xy3hA&e=

> We don't seem to have any problems scheduling jobs on GPUs, even though our
> Slurm RPM build host doesn't have any NVIDIA software installed, as shown by
> the command:
> $ ldconfig -p | grep libnvidia-ml
>
> I'm curious about Prentice's statement about needing NVIDIA libraries to be
> installed when building Slurm RPMs, and I read the discussion in bug 9525,

> https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.schedmd.com_show-5Fbug.cgi-3Fid-3D9525&d=DwIBAg&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=GNGEhyc3F2bEZxbHK93tumXk56f37DOl99aYsOeUVOE&s=6GDTIFa-spnv8ZMtKsdwJaLreyZMX4T5EW3MnAX54iI&e=

[Paul Edmon]

You can include gpu's as gres in slurm with out compiling specifically against nvml.  You only really need to do that if you want to use the autodetection features that have been built into the slurm.  We don't really use any of those features at our site, we only started building against nvml to future proof ourselves for when/if those features become relevant to us.

To me at least it would be nicer if there was a less hacky way of getting it to do that.  Arguably Slurm should dynamically link against the libs it needs or not depending on the node.  We hit this issue with Lustre/IB as well where you have to roll a separate slurm for each type of node you have if you want these which is hardly ideal.

-Paul Edmon-

[Ole Holm Nielsen]

Thanks Paul!

On 26-01-2021 21:11, Paul Raines wrote:
> You should check your jobs that allocated GPUs and make sure
> CUDA_VISIBLE_DEVICES is being set in the environment.  This is a sign
> you GPU support is not really there but SLURM is just doing "generic"
> resource assignment.

Could you elaborate a bit on this remark? Are you saying that I need to
check if CUDA_VISIBLE_DEVICES is defined automatically by Slurm inside
the batch job as described in https://slurm.schedmd.com/gres.html?

What do you mean by "your GPU support is not really there" and Slurm
doing "generic" resource assignment? I'm just not understanding this...

With my Slurm 20.02.6 built without NVIDIA libraries, Slurm nevertheless
seems to be scheduling multiple jobs so that different jobs are assigned
to different GPUs. The GRES=gpu* values point to distinct IDX values
(GPU indexes). The nvidia-smi command shows individual processes
running on distinct GPUs. All seems to be fine - or am I completely
mistaken?

Thanks,
Ole

[Ole Holm Nielsen]

On 26-01-2021 21:36, Paul Edmon wrote:
> You can include gpu's as gres in slurm with out compiling specifically
> against nvml.  You only really need to do that if you want to use the
> autodetection features that have been built into the slurm.  We don't
> really use any of those features at our site, we only started building
> against nvml to future proof ourselves for when/if those features become
> relevant to us.

Thanks for this clarification about not actually *requiring* the NVIDIA
NVML library in the Slurm build!

Now I'm seeing this description in https://slurm.schedmd.com/gres.html
about automatic GPU configuration by Slurm:

> If AutoDetect=nvml is set in gres.conf, and the NVIDIA Management Library (NVML) is installed on the node and was found during Slurm configuration, configuration details will automatically be filled in for any system-detected NVIDIA GPU. This removes the need to explicitly configure GPUs in gres.conf, though the Gres= line in slurm.conf is still required in order to tell slurmctld how many GRES to expect.

I have defined our GPUs manually in gres.conf with File=/dev/nvidia?
lines, so it would seem that this obviates the need for NVML. Is this
the correct conclusion?

/Ole

[Paul Edmon]

That is correct.  I think NVML has some additional features but in terms
of actually scheduling them what you have should work. They will just be
treated as normal gres resources.

-Paul Edmon-

[Paul Raines]

Yes, you need to check inside the job.

This is a while ago now, but I am pretty sure I remember though from
the SLURM accounting aspect the jobs were being assigned GPUs fine as you
would see in 'scontrol show job' or 'sacct --job', the CUDA_VISIBLE_DEVICES
environment variable was not being set so all jobs on the node saw
(an possibly used) all GPUs and would get into conflicts.

YOu could see this by doing something like

login$ srun --ntasks-per-node=1 --cpus-per-task=4 --gpus 2 --pty /bin/bash
node$ echo $CUDA_VISIBLE_DEVICES

and get nothing. After installing the proper RPMs on the GPU node with
the NVML support and doing the test again I get

login$ srun --ntasks-per-node=1 --cpus-per-task=4 --gpus 2 --pty /bin/bash
node$ echo $CUDA_VISIBLE_DEVICES
0,1

If CUDA_VISIBLE_DEVICES is not being set, also search for error messages
in your slurmd log file on the node

This also probably requires you to have

ProctrackType=proctrack/cgroup
TaskPlugin=task/affinity,task/cgroup
GresTypes=gpu

like I do

-- Paul Raines (http://help.nmr.mgh.harvard.edu)

[Christopher Samuel]

On 1/26/21 12:10 pm, Ole Holm Nielsen wrote:

> What I don't understand is, is it actually *required* to make the NVIDIA
> libraries available to Slurm?  I didn't do that, and I'm not aware of
> any problems with our GPU nodes so far.  Of course, our GPU nodes have
> the libraries installed and the /dev/nvidia? devices are present.

You only need it if you want to use NVML autodetection of GPUs, we don't
have any nvidia software in the OS image we use to build our vast array
of RPMs and they work just fine on our GPU nodes.

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA

[Loris Bennett]

Same here - $10k for less than 200 nodes. That's an order of magnitude
which makes the finance people ask what we are getting for the money.
As we don't have any special requirements which would require
customisation, that's not easy to answer, so currently we don't have a
support contract. However, I feel we should really contribute to
SchedMD's continued existence.

Cheers,

Loris

Jason Simms <sim...@lafayette.edu> writes:

> We’re in the same boat. Extremely small cluster. $10k for support. We
> don’t need nearly that level of engagement, but there ya go. We’ve
> passed for now, but I’d like to have a support contract ideally.
>
> Jason
>
> On Tue, Jan 26, 2021 at 2:49 PM Robert Kudyba <rku...@fordham.edu> wrote:
>
> On Mon, Jan 25, 2021 at 6:36 PM Brian Andrus <toom...@gmail.com> wrote:
>
> Also, a plug for support contracts. I have been doing slurm for a very
> long while, but always encourage my clients to get a support contract.
> That is how SchedMD stays alive and we are able to have such a good
> piece of software. I see the cloud providers starting to build tools
> that will eventually obsolesce slurm for the cloud. I worry that there
> won't be enough paying customers for Tim to keep things running as well
> as he has. I'm pretty sure most folks that use slurm for any period of
> time has received more value that a small support contract would be.
>
> We considered this but we have a very small cluster. And when I reached out for a quote, I was told "SchedMD has a MIN node count of 256 for $10K/yr".
>
> Since we're using Bright Computing we've always had to ignore Slurm updates from yum and have to compile our own version.
>
> Curious, which cloud provider scheduling tools do you see gaining traction?
--

Dr. Loris Bennett (Hr./Mr.)
ZEDAT, Freie Universität Berlin Email loris....@fu-berlin.de

[Brian Andrus]

I've definitely been there with the minimum cost issue. One thing I have done personally is start attending SLUG. Now I can give back and learn more in the process. That may be an option to pitch, iterating the value you receive from open source software as part of the ROI.

Interestingly, I have been able to deploy completely to cloud using only slurm. It has the ability to integrate into any cloud cli, so nothing else has been needed. Just for the heck of it, I am thinking of integrating it into Terraform, although not necessary.

Brian Andrus

[Tina Friedrich]

Yeah, I don't build against NVML either at the moment (it's filed under
'try when you've got some spare time'). I'm pretty much 'autodetecting'
what my gres.conf file needs to look like on nodes via my config
management, and that all seems to work just fine.

CUDA_VISIBLE_DEVIZES and cgroup device restrictions works, as well.

Tina

[Philip Kovacs]

Lots of mixed reactions here, many in favor (and grateful) for the add to EPEL, many much less enthusiastic.

I cannot rename an EPEL package that is now in the wild without providing an upgrade path to the new name. 

Such an upgrade path would defeat the purpose of the rename and won't help at all.

The best option, in my opinion, would be to use one of the following yum plugins:

yum-plugin-versionlock

yum-plugin-priorities

yum-plugin-protectbase

With one or more of these, you can lock slurm on a particular version (versionlock), prioritize one repo over another

(priorities) or protect certain repos from any upgrade (protectbase).  Using these plugins also closes the general problem

of not wanting locally built packages ever to be upgraded until you deem otherwise.  Name collisions become irrelevant.

I can also do one of two other things:

1. Remove slurm from EPEL

2. Downgrade slurm in EPEL to a stable but older version that likely won't interfere with most installations.

Phil

[Jürgen Salk]

Hi Phil,

assuming that all sites maintaining their own Slurm rpm packages must
now somehow ensure that these are not replaced by the EPEL packages
anyway, why wouldn't it be possible, in the long run, to follow the
Fedora packaging guidelines for renaming existing packages?

https://docs.fedoraproject.org/en-US/packaging-guidelines/#renaming-or-replacing-existing-packages

Best regards
Jürgen

[Brian Andrus]

Wow,

This is getting so ridiculous that my email program has started putting
this thread in junk...

The spirit of Linux is to give you the tools so you can do things how
you want. You have the tools, do not expect someone else to come over
and plant/maintain your garden. Just because they can do a thing doesn't
mean they should do a thing.

There are many ways to achieve what is desired, most of which do not
require anyone other than the system admin.

If your issue can be solved without affecting others, leave them alone
and fix your issue.

Brian Andrus

[Relu Patrascu]

We've long observed that various distributions make decisions we may not
necessarily agree with. As we regard the slurm installation critical to
our function, we choose to build from source and control its version
(and contents) without any doubts.

Relu

[Philip Kovacs]

I am familiar with the package rename process and it would not have the effect you might think it would.

If I provide an upgrade path to a new package name, e.g. slurm-xxx, the net effect would be to tell yum or

dnf-managed systems that the new package name for slurm is slurm-xxx.  That would make the problem

worse by not only upgrading slurm from EPEL but also renaming it.

The way to handle the problem properly is to use any of the other methods I described earlier that make

the name collision moot, i.e. tell your systems that your local repo has a higher priority or that packages

in that repo are protected from upgrade or lock slurm onto a designated version.

If you accidentally got pulled into an upgrade that you didn't want, those are the steps to prevent it in the future.

You can easily create test cases for yum by placing (artificially) higher versions of the packages you want

to protect into a test repo.  Then use yum protectbase or yum priorities to tell your system that your local repo

is protected and/or that the local repo has a higher priority than the test repo, depending on which yum plugins

you use. Then verify that that `yum check-update` does not report an upgrade for your test packages.

Phil

[Ryan Novosielski]

My main point here is that essentially upgrading someone from, for example, SLURM 20.02 to SLURM 20.11 is not desirable, and that’s why upgrades between major versions, IMO, should not happen automatically. There’s a whole section of the documentation about how to do this properly, and I’m not sure that one should ever count on a package to do this properly, at least without advance planning (which I guess you could argue someone should do anyway). That doesn’t have much to do with the initial complaint of them just appearing, which is a one-time problem as now it’s in the repo. At any rate, not wanting to upgrade automatically is at least true of slurmdbd and slurmctld, maybe less true of the client versions. Sure, someone should know what they’re upgrading to, but I don’t imagine anyone’s goal here is to make it easier to shoot oneself in the foot. I imagine no one releases packages with the express goal of having people version lock them out. :-D

None of this affects me, as I’m not installing SLURM this way, but if someone /were/ asking me what I thought, I’d say that the current “slurm” package should be renamed “slurm-20.11,” which can follow an upgrade path with no issue as it’s safe to upgrade within a SLURM release, but that should only ever upgrade to another minor release of SLURM 20.11. The next one should be called slurm-21.02 or whatever version it ultimately becomes (I don’t think it’s even been pre-released yet). I don’t know enough about EPEL to know if that’s an allowable solution, but it’s one I’ve seen used for other software packages where one needs to manually upgrade to a new major version (I’m thinking on Ubuntu where the VirtualBox package has an embedded major/minor version number in it so you only automatically upgrade to a new point release).

To give an example of why we don’t just upgrade via packages, our SlurmDBD upgrade has at times taken more than 24 hours to do, and if something else gets upgraded sometime before that, it could cause problems.

--
#BlackLivesMatter
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'

[Michael Jennings]

On Wednesday, 03 February 2021, at 18:06:27 (+0000),

Philip Kovacs wrote:

> I am familiar with the package rename process and it would not have

> the effect you might think it would.If I provide an upgrade path to

> a new package name, e.g. slurm-xxx, the net effect would be to tell

> yum ordnf-managed systems that the new package name for slurm is
> slurm-xxx.  That would make the problemworse by not only upgrading

> slurm from EPEL but also renaming it.
>
> The way to handle the problem properly is to use any of the other

> methods I described earlier that makethe name collision moot,

> i.e. tell your systems that your local repo has a higher priority or

> that packagesin that repo are protected from upgrade or lock slurm

> onto a designated version.
>
> If you accidentally got pulled into an upgrade that you didn't want,
> those are the steps to prevent it in the future.
>
> You can easily create test cases for yum by placing (artificially)

> higher versions of the packages you wantto protect into a test

> repo.  Then use yum protectbase or yum priorities to tell your

> system that your local repois protected and/or that the local repo

> has a higher priority than the test repo, depending on which yum

> pluginsyou use. Then verify that that `yum check-update` does not

> report an upgrade for your test packages.

Phil's not wrong here. Package renames must use "Obsoletes:" and
"Provides:" to facilitate the seamless migration from the old name to
the new name, and as a consequence of that, the end result will not
change; YUM/DNF will replace the site's Slurm RPMs with Phil's renamed
RPMs from EPEL.

In addition to the plugins Phil (and others) mentioned previously
(which really *are* the best ways to address this!), I'll share *my*
super-spiffy top secret way of handling this exact type of situation
(i.e., preventing site-maintained RPMs from being unintentionally
"upgraded"): Epoch. For any RPMs I personally build/maintain that
could possibly show up somewhere externally (i.e., stuff that's not
"LANL Internal Use Only/OUO"), I set the Epoch: header value to a
YYYYMMDD datestamp.

For example, I maintain my own RPM for Mutt, my e-mail client. To
make sure my RPM always wins, my mutt.spec file has this:

Epoch: %{expand:%(date +%%Y%%m%%d)}

This provides a build-date-stamped value for the Epoch header field.
(For anyone not familiar with Epoch, it's basically a package
versioning "trump card:" Epoch comparison has the highest precedence
of all, overriding both Version and Release. Whichever RPM has the
highest Epoch value wins, full stop.) No matter what version of Mutt
might appear in EPEL or anywhere else, my RPM won't be touched because
of its extremely high Epoch. (Typically even upstream RPMs that *do*
have Epochs set are done incrementally starting at 1, so they'd have
to mess up the versioning a spectacular number of times (specifically,
20,210,203 times and counting) to catch up to my RPM's Epoch! :-D

HTH,
Michael

--
Michael E. Jennings <m...@lanl.gov> - [PGPH: he/him/his/Mr] -- hpc.lanl.gov
HPC Systems Engineer -- Platforms Team -- HPC Systems Group (HPC-SYS)
Strategic Computing Complex, Bldg. 03-2327, Rm. 2341 W: +1 (505) 606-0605
Los Alamos National Laboratory, P.O. Box 1663, Los Alamos, NM 87545-0001

[Philip Kovacs]

What you are describing is implemented in the EL ecosystems as module streams.  Streams are setup by maintainers

to correspond to major version trains of software.  Users can then select the stream they want and can change streams

as desired (unfortunately this is often not trivial), without upgrading or downgrading their systems.  It's a lot of extra work

and misery for maintainers to manage several independent working streams, however, so projects such as "Modularity"

have received a fair amount of criticism for that and other reasons.