[slurm-users] Upgrading a slurm on a cluster, 17.02 --> 18.08
Baker D.J.
Hello,
I wondered if I could compare notes with other community members who have upgraded slurm on their cluster. We are currently running slurm v17.02 and I understand that the rpm mix/structure changed at v17.11. We are, by the way, planning to upgrade to v18.08.
I gather that I should upgrade slurmdbd first and ideally the slurmctld should not be touched/killed at point. On our cluster we run both the slurmdbd and the slurmctld on a single host. I have been advised to just upgrade slurmdbd initially -- upgrading the rpms in a piecemeal fashion. That is, just installing the slurm and slurmdbd rpms in the first instance. Due to the rpm structure changes at v17.11 that strategy doesn't work -- the yum updates fail with dependency issues.
I guess that the only solution is to upgrade all the slurm at once. That means that the slurmctld will be killed (unless it has been stopped first). Is there anyone who has done an upgrade who would be willing to share their experiences, please? In other words, is it valid to kill both the slurmdbd and slurmcltd processes at the start of the upgrade and, if so, how does the loss of the slurmctld affect the cluster re running jobs, user activity, etc?
Best regards,
David
Chris Samuel
> I guess that the only solution is to upgrade all the slurm at once. That
> means that the slurmctld will be killed (unless it has been stopped first).
option to (allegedly, I've never used it) suppress the execution of pre/post
install scripts.
A big warning would be do not use systemctl to start the new slurmdbd for the
first time when upgrading!
Stop the older one first (and then take a database dump) and then run the new
slurmdbd process with the "-Dvvv" options (inside screen, just in case) so
that you can watch its progress and systemd won't decide it's been taking too
long to start and try and kill it part way through the database upgrades).
Once that's completed successfully then you can ^C it and start it up via the
systemctl once more.
Hope that's useful!
All the best,
Chris
[1] - I've always installed into ${shared_local_area}/slurm/${version} and had
a symlink called "latest" that points at the currently blessed version of
Slurm. Then I stop slurmdbd, upgrade that as above, then I can do slurmctld
(with partitions marked down, just in case). Once those are done I can
restart slurmd's around the cluster.
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
Baker D.J.
Thank you for your comments. I could potentially force the upgrade of the slurm and slurm-slumdbd rpms using something like:
I would have thought that someone would have crossed this bridge, but maybe most admins don't use rpms...
Sent: 25 September 2018 13:00
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Upgrading a slurm on a cluster, 17.02 --> 18.08
Paul Edmon
I use rpm's for our installs here. I usually pause all the jobs prior to the upgrade, then I follow the guide here:
https://slurm.schedmd.com/quickstart_admin.html
I haven't done the upgrade to 18.08 though yet, and so I haven't had to contend with the automatic restart that seems to be the case with the new rpm spec script (we went to 17.11 prior to the rpm spec reorg). Frankly I wish that they didn't do the automatic restart as I like to manage that myself.
As Chris said though you definitely want to do the slurmdbd upgrade from the commandline. I've had it where when just restarting the service it times out and the database only gets partially update. In which case I had to restore from the mysqldump I had made and tried again. Also highly recommend doing mysqldumps prior to major version updates.
-Paul Edmon-
Chris Samuel
> That will certainly work, however the slurmctld (or in the case of my test
> node, the slurmd) will be killed. The logic is that at v17.02 the slurm rpm
> provides slurmctld and slurmd. So upgrading that rpm will destroy/kill the
> existing slurmctld or slurmd processes.
Nothing should invoke the systemd commands with that, should it? Or do you
mean taking the libraries, etc, away out underneath of the running process
will cause it to crash?
Might be worth testing that on on a VM to see if it will happen.
Best of luck!
Chris
Baker D.J.
Thank you for your reply. You're correct, the systemd commands aren't invoked, however upgrading the slurm rpm effectively pulls the rug from under /usr/sbin/slurmctld. The v17.02 slurm rpm provides /usr/sbin/slurmctld, but from v17.11 that executable is provided by the slurm-slurmctld rpm.
In other words, doing a minimal install of just the slurm and the slurmdbd rpms deletes the slurmctld executable. I haven't explicitly tested this, however I tested the upgrade on a compute node and experimented with the slurmd -- the logic should be the same.
I guess that the question that comes to mind is.. Is it a really big deal if the slurmctld process is down whilst the slurmdbd is being upgraded? Bearing in mind that I will probably opt to suspend all run jobs and stop the partitions during the upgrade.
Best regards,
David
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Upgrading a slurm on a cluster, 17.02 --> 18.08
Bjørn-Helge Mevik
> I guess that the question that comes to mind is.. Is it a really big deal if
> the slurmctld process is down whilst the slurmdbd is being upgraded?
never noticed any problems with that.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Ole Holm Nielsen
> Baker D.J. <D.J....@soton.ac.uk> writes:
>
>> I guess that the question that comes to mind is.. Is it a really big deal if
>> the slurmctld process is down whilst the slurmdbd is being upgraded?
>
> I tend to always stop slurmctld before upgrading slurmdbd, and have
> never noticed any problems with that.
in https://slurm.schedmd.com/quickstart_admin.html#upgrade
It's OK to stop slurmctld for some minutes, but you may want to increase
some timeouts in slurm.conf. See also my Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrade-slurmctld
/Ole
Christopher Benjamin Coffey
I'd recommend the following that I've learned from bad experiences upgrading between the last major version.
1. Consider upgrading to mysql-server 5.5 or greater
2. Purge/archive unneeded jobs/steps before the upgrade, to make the upgrade as quick as possible:
slurmdbd.conf:
ArchiveDir=/common/adm/slurmdb_archive
ArchiveEvents=yes
ArchiveJobs=yes
ArchiveSteps=no
ArchiveResvs=no
ArchiveSuspend=no
PurgeEventAfter=1month
PurgeJobAfter=6months
PurgeResvAfter=2month
PurgeStepAfter=6months
PurgeSuspendAfter=2month
3. Take a fresh mysql dump after the archives occur:
mysqldump --all-databases > slurm_db.sql
4. Testing the update on another machine, or vm that has a representation of your environment (same rpms, configs, etc). Just take your newly created dump from production and load it into the test system:
mysql -u root < slurm_db.db
Once you take care of any connection issues in mysql, allowing a different host to connect, then you can fire up slumdbd to perform the upgrade. And see how long it takes, and what hiccups you will run into. Now you know, and can plan your maintenance window accordingly.
Hope that helps! Good luck!
Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
On 9/26/18, 8:57 AM, "slurm-users on behalf of Baker D.J." <slurm-use...@lists.schedmd.com on behalf of D.J....@soton.ac.uk> wrote:
Thank you for your reply. You're correct, the systemd commands aren't invoked, however upgrading the slurm rpm effectively pulls the rug from under /usr/sbin/slurmctld. The v17.02 slurm rpm provides /usr/sbin/slurmctld,
but from v17.11 that executable is provided by the slurm-slurmctld rpm.
In other words, doing a minimal install of just the slurm and the slurmdbd rpms deletes the slurmctld executable. I haven't explicitly tested this, however I tested the upgrade on a compute node and experimented with
the slurmd -- the logic should be the same.
I guess that the question that comes to mind is.. Is it a really big deal if the slurmctld process is down whilst the slurmdbd is being upgraded? Bearing in mind that I will probably opt to suspend all run jobs and stop
the partitions during the upgrade.
Best regards,
David
Sent: 26 September 2018 11:26
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Upgrading a slurm on a cluster, 17.02 --> 18.08
On Tuesday, 25 September 2018 11:54:31 PM AEST Baker D. J. wrote:
> That will certainly work, however the slurmctld (or in the case of my test
> node, the slurmd) will be killed. The logic is that at v17.02 the slurm rpm
> provides slurmctld and slurmd. So upgrading that rpm will destroy/kill the
> existing slurmctld or slurmd processes.
If you do that with the --noscripts then will it really kill the process?
Nothing should invoke the systemd commands with that, should it? Or do you
mean taking the libraries, etc, away out underneath of the running process
will cause it to crash?
Might be worth testing that on on a VM to see if it will happen.
Best of luck!
Chris
--
Chris Samuel :
: Melbourne, VIC
Ole Holm Nielsen
upgrade as quick as possible:
>
> slurmdbd.conf:
>
> ArchiveDir=/common/adm/slurmdb_archive
> ArchiveEvents=yes
> ArchiveJobs=yes
> ArchiveSteps=no
> ArchiveResvs=no
> ArchiveSuspend=no
> PurgeEventAfter=1month
> PurgeJobAfter=6months
> PurgeResvAfter=2month
> PurgeStepAfter=6months
> PurgeSuspendAfter=2month
for the Archive* and Purge* parameters in slurmdbd.conf. The
description and example (at the end) in
https://slurm.schedmd.com/slurmdbd.conf.html is really terse.
Does anyone have a good explanation of usage of the Archive and Purge
features for the Slurm database? For example, how can the archived data
be used for accounting etc.?
Thanks,
Ole
Bjørn-Helge Mevik
> Does anyone have a good explanation of usage of the Archive and Purge features
> for the Slurm database? For example, how can the archived data be used for
> accounting etc.?
that is used by default to create the archive files to figure out their
format. Alternatively, you would have to write your own ArchiveScript
to do the archiving.
Chris Samuel
> Does anyone have a good explanation of usage of the Archive and Purge
> features for the Slurm database? For example, how can the archived data
> be used for accounting etc.?
about usage (even after systems had been decommissioned) that we needed to go
back to get data out of Slurm for. Luckily we had some beefy Percona MySQL
servers in a cluster!
All the best,
Paul Edmon
Formerly we kept it all in there but we found that slurm upgrades were
taking too long and the database was getting very large. Data that is
dumped can be read in by sacct using the flags documented in there.
It's very rare though that we need to look back at that data.
-Paul Edmon-