[slurm-users] Upgrade from 17.02.11 to 21.08.2 and state information

109 views
Skip to first unread message

Nathan Smith

unread,
Feb 2, 2022, 4:38:44 PMFeb 2
to slurm...@schedmd.com

The "Upgrades" section of the quick-start guide [0] warns:

> Slurm permits upgrades to a new major release from the past two major
> releases, which happen every nine months (e.g. 20.02.x or 20.11.x to
> 21.08.x) without loss of jobs or other state information. State
> information from older versions will not be recognized and will be
> discarded, resulting in loss of all running and pending jobs.

We are planning for an upgrade from 17.02.11 to 21.08.2. As a part of
our upgrade procedure we'd be bringing the scheduler to full stop, so
the loss of running and pending jobs would not be a concern. Is there
anything more to state information than running and pending jobs? For
example, would the JobID count revert to 1 in the case of such an
upgrade?

[0] https://slurm.schedmd.com/quickstart_admin.html#upgrade

--
Nathan Smith
Research Systems Engineer
Advanced Computing Center
Oregon Health & Science University

Brian Haymore

unread,
Feb 2, 2022, 4:50:59 PMFeb 2
to slurm...@schedmd.com, Slurm User Community List
Are you running slurmdbd in your current setup?  If you are then the upgrade path there might have additional considerations moving this far in versions.

--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150, Fax: 801-585-5366
http://bit.ly/1HO1N2C

From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Nathan Smith <smin...@ohsu.edu>
Sent: Wednesday, February 2, 2022 2:38 PM
To: slurm...@schedmd.com <slurm...@schedmd.com>
Subject: [slurm-users] Upgrade from 17.02.11 to 21.08.2 and state information
 

Brian Andrus

unread,
Feb 2, 2022, 6:50:02 PMFeb 2
to slurm...@lists.schedmd.com
I actually just did that path for a children's hospital.

It was fairly straight-forward. Running jobs were not affected.

You do need to go 17->18->19->20->21

This is because there were changes in the db schema.

If you plan on bringing everything to a stop (no running jobs), you
should be good. You will still need to do the incremental for the db
changes, but no worries about state files either way.

Brian Andrus

Nathan Smith

unread,
Feb 3, 2022, 10:38:08 AMFeb 3
to Slurm User Community List, slurm...@schedmd.com

Yes, we are running slurmdbd. We could arrange enough downtime to do an incremental upgrade of major versions as Brian Andrus suggested, at least on the slurmctld and slurmdbd systems. The slurmds I would just do a direct upgrade once the scheduler work was completed.

 

--

Nathan Smith

Research Systems Engineer

Advanced Computing Center

Oregon Health & Science University

 

Ole Holm Nielsen

unread,
Feb 3, 2022, 2:56:25 PMFeb 3
to slurm...@lists.schedmd.com
On 03-02-2022 16:37, Nathan Smith wrote:
> Yes, we are running slurmdbd. We could arrange enough downtime to do an
> incremental upgrade of major versions as Brian Andrus suggested, at
> least on the slurmctld and slurmdbd systems. The slurmds I would just do
> a direct upgrade once the scheduler work was completed.

As Brian Andrus said, you must upgrade Slurm by at most 2 major
versions, and that includes slurmd's as well! Don't do a "direct
upgrade" of slurmd by more than 2 versions!

I recommend separate physical servers for slurmdbd and slurmctld. Then
you can upgrade slurmdbd without taking the cluster offline. It's OK
for slurmdbd to be down for many hours, since slurmctld caches the state
information in the meantime.

I've described the Slurm upgrade process in detail in my Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

Since you start from 17.02, you have to be extremely cautious when
upgrading the database! See the Wiki page for details. Make sure to
test the database upgrade on a test server, using a database dump in
stead of the real slurmdbd server.

I hope this helps.

/Ole

> *From:* slurm-users <slurm-use...@lists.schedmd.com> *On Behalf
> Of *Brian Haymore
> *Sent:* Wednesday, February 2, 2022 1:51 PM
> *To:* slurm...@schedmd.com; Slurm User Community List
> <slurm...@lists.schedmd.com>
> *Subject:* [EXTERNAL] Re: [slurm-users] Upgrade from 17.02.11 to 21.08.2
> and state information
>
> Are you running slurmdbd in your current setup?  If you are then the
> upgrade path there might have additional considerations moving this far
> in versions.
>
> --
> Brian D. Haymore
> University of Utah
> Center for High Performance Computing
> 155 South 1452 East RM 405
> Salt Lake City, Ut 84112
> Phone: 801-558-1150, Fax: 801-585-5366
> http://bit.ly/1HO1N2C
> <https://urldefense.com/v3/__http:/bit.ly/1HO1N2C__;!!Mi0JBg!eqxyactyQJqJ7Bwy-LEQT4WeJrmjDkqZxfwNtCBk_zliQifvEt1RQj4RYjUwe98$>
>
> ------------------------------------------------------------------------
>
> *From:*slurm-users <slurm-use...@lists.schedmd.com
> <mailto:slurm-use...@lists.schedmd.com>> on behalf of Nathan
> Smith <smin...@ohsu.edu <mailto:smin...@ohsu.edu>>
> *Sent:* Wednesday, February 2, 2022 2:38 PM
> *To:* slurm...@schedmd.com <mailto:slurm...@schedmd.com>
> <slurm...@schedmd.com <mailto:slurm...@schedmd.com>>
> *Subject:* [slurm-users] Upgrade from 17.02.11 to 21.08.2 and state
> information
>
>
> The "Upgrades" section of the quick-start guide [0] warns:
>
> > Slurm permits upgrades to a new major release from the past two major
> > releases, which happen every nine months (e.g. 20.02.x or 20.11.x to
> > 21.08.x) without loss of jobs or other state information. State
> > information from older versions will not be recognized and will be
> > discarded, resulting in loss of all running and pending jobs.
>
> We are planning for an upgrade from 17.02.11 to 21.08.2. As a part of
> our upgrade procedure we'd be bringing the scheduler to full stop, so
> the loss of running and pending jobs would not be a concern. Is there
> anything more to state information than running and pending jobs? For
> example, would the JobID count revert to 1 in the case of such an
> upgrade?
>
> [0] https://slurm.schedmd.com/quickstart_admin.html#upgrade
> <https://urldefense.com/v3/__https:/slurm.schedmd.com/quickstart_admin.html*upgrade__;Iw!!Mi0JBg!eqxyactyQJqJ7Bwy-LEQT4WeJrmjDkqZxfwNtCBk_zliQifvEt1RQj4RNExvAfw$>

Ryan Novosielski

unread,
Feb 3, 2022, 4:00:11 PMFeb 3
to Ole.H....@fysik.dtu.dk, Slurm User Community List


> On Feb 3, 2022, at 2:55 PM, Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> wrote:
>
> On 03-02-2022 16:37, Nathan Smith wrote:
>> Yes, we are running slurmdbd. We could arrange enough downtime to do an incremental upgrade of major versions as Brian Andrus suggested, at least on the slurmctld and slurmdbd systems. The slurmds I would just do a direct upgrade once the scheduler work was completed.
>
> As Brian Andrus said, you must upgrade Slurm by at most 2 major versions, and that includes slurmd's as well! Don't do a "direct upgrade" of slurmd by more than 2 versions!
>
> I recommend separate physical servers for slurmdbd and slurmctld. Then you can upgrade slurmdbd without taking the cluster offline. It's OK for slurmdbd to be down for many hours, since slurmctld caches the state information in the meantime.

The one thing you want to watch out for here – maybe more so if you are using a VM than a physical server as you may have sized the RAM for how much slurmctld appears to need, as we did – is that that caching that takes place on the slurmctld uses memory (I guess obviously, when you think about it). The result there can be that eventually if you have slurmd down for a long time (we had someone who was hitting a bug that would start running jobs right after everyone went to sleep for example), your slurmctld can run out of memory, crash, and then that cache is lost. You don’t normally see that memory being used like that, because slurmdbd is normally up/accepting the accounting data.

--
#BlackLivesMatter
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'

Bjørn-Helge Mevik

unread,
Feb 4, 2022, 2:59:49 AMFeb 4
to slurm...@schedmd.com
Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> writes:

> As Brian Andrus said, you must upgrade Slurm by at most 2 major
> versions, and that includes slurmd's as well! Don't do a "direct
> upgrade" of slurmd by more than 2 versions!

That should only be an issue if you have running jobs during the
upgrade, shouldn't it? As I understand it, without any running jobs,
you can do pretty much what you want on the compute nodes. Or am I
missing something here?

--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

signature.asc

Ole Holm Nielsen

unread,
Feb 4, 2022, 9:28:19 AMFeb 4
to slurm...@lists.schedmd.com
On 04-02-2022 08:59, Bjørn-Helge Mevik wrote:
> Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> writes:
>
>> As Brian Andrus said, you must upgrade Slurm by at most 2 major
>> versions, and that includes slurmd's as well! Don't do a "direct
>> upgrade" of slurmd by more than 2 versions!
>
> That should only be an issue if you have running jobs during the
> upgrade, shouldn't it? As I understand it, without any running jobs,
> you can do pretty much what you want on the compute nodes. Or am I
> missing something here?

I think that Slurm's communication protocol is incompatible when
versions differ by more than 2. So the slurmd daemons may possibly lose
contact with the slurmctld in that case.

In my experience, it's not a problem to upgrade slurmd while the nodes
are running jobs: Upgrade the slurmd RPM, and slurmd will restart itself
and attach to the running jobs. There are probably cases where this
will cause job crashes, so please heed the information collected in the
Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-on-centos-7
There may be some issues with MPI applications as mentioned in the Wiki.

/Ole

Ole Holm Nielsen

unread,
Feb 4, 2022, 9:37:20 AMFeb 4
to Slurm User Community List
On 03-02-2022 21:59, Ryan Novosielski wrote:
>> On Feb 3, 2022, at 2:55 PM, Ole Holm Nielsen <Ole.H....@fysik.dtu.dk> wrote:
>>
>> On 03-02-2022 16:37, Nathan Smith wrote:
>>> Yes, we are running slurmdbd. We could arrange enough downtime to do an incremental upgrade of major versions as Brian Andrus suggested, at least on the slurmctld and slurmdbd systems. The slurmds I would just do a direct upgrade once the scheduler work was completed.
>>
>> As Brian Andrus said, you must upgrade Slurm by at most 2 major versions, and that includes slurmd's as well! Don't do a "direct upgrade" of slurmd by more than 2 versions!
>>
>> I recommend separate physical servers for slurmdbd and slurmctld. Then you can upgrade slurmdbd without taking the cluster offline. It's OK for slurmdbd to be down for many hours, since slurmctld caches the state information in the meantime.
>
> The one thing you want to watch out for here – maybe more so if you are using a VM than a physical server as you may have sized the RAM for how much slurmctld appears to need, as we did – is that that caching that takes place on the slurmctld uses memory (I guess obviously, when you think about it). The result there can be that eventually if you have slurmd down for a long time (we had someone who was hitting a bug that would start running jobs right after everyone went to sleep for example), your slurmctld can run out of memory, crash, and then that cache is lost. You don’t normally see that memory being used like that, because slurmdbd is normally up/accepting the accounting data.

The slurmctld caches job state information in:
# scontrol show config | grep StateSaveLocation
StateSaveLocation = /var/spool/slurmctld

The StateSaveLocation should retain job information even if slurmctld
crashes (at least the data which have been committed to disk).

The StateSaveLocation file system must not fill up, of course! There
are also some upper limits to the number of records in
StateSaveLocation, but I can't find the numbers right now.

/Ole

Reply all
Reply to author
Forward
0 new messages