[slurm-users] Extreme long db upgrade 16.05.6 -> 17.11.3

[Christopher Benjamin Coffey]

Hello,

We have been trying to upgrade slurm on our cluster from 16.05.6 to 17.11.3. I'm thinking this should be doable? Past upgrades have been a breeze, and I believe during the last one, the db upgrade took like 25 minutes. Well now, the db upgrade process is taking far too long. We previously attempted the upgrade during a maintenance window and the upgrade process did not complete after 24 hrs. I gave up on the upgrade and reverted the slurm version back by restoring a backup db.

Since the failed attempt at the upgrade, I've archived a bunch of jobs as we had 4 years of jobs in the database. Now only keeping last 1.5 years worth. This reduced our db size down from 3.7GB to 1.1GB. We are now archiving jobs regularly through slurm.

I've finally had time to look at this a bit more and we've restored the reduced database onto another system to test the upgrade process in a dev environment, hoping to prove that the slimmed down db will upgrade within a reasonable amount of time. Yet, the current upgrade on this dev system has already taken 20 hrs. The database has 1.8M jobs. That doesn't seem like that many jobs!

The conversion is stuck on this command:

update "monsoon_job_table" as job left outer join ( select job_db_inx, SUM(consumed_energy) 'sum_energy' from "monsoon_step_table" where id_step >= 0 and consumed_energy != 18446744073709551614 group by job_db_inx ) step on job.job_db_inx=step.job_db_inx set job.tres_alloc=concat(job.tres_alloc, concat(',3=', case when step.sum_energy then step.sum_energy else 18446744073709551614 END)) where job.tres_alloc != '' && job.tres_alloc not like '%,3=%':

The system is no slouch:

28 core, E5-2680 v4 2.4GHz
SSD
128GB memory

Anyone have this issue? Anyone have a suggestion? This seems like a ridiculous amount of time needed to perform the upgrade! The database is healthy as far as I see. No errors in the slurmdbd log, etc.

Let me know if you need more info!

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

[Kurt H Maier]

On Wed, Feb 21, 2018 at 11:56:38PM +0000, Christopher Benjamin Coffey wrote:
> Hello,
>
> We have been trying to upgrade slurm on our cluster from 16.05.6 to 17.11.3. I'm thinking this should be doable? Past upgrades have been a breeze, and I believe during the last one, the db upgrade took like 25 minutes. Well now, the db upgrade process is taking far too long. We previously attempted the upgrade during a maintenance window and the upgrade process did not complete after 24 hrs. I gave up on the upgrade and reverted the slurm version back by restoring a backup db.

We hit this on our try as well: upgrading from 17.02.9 to 17.11.3. We
truncated our job history for the upgrade, and then did the rest of the
conversion out-of-band and re-imported it after the fact. It took us
almost sixteen hours to convert a 1.5 million-job store.

We got hung up on precisely the same query you did, on a similarly hefty
machine. It caused us to roll back an upgrade and try again during our
subsequent maintenance window with the above approach.

khm

[Christopher Benjamin Coffey]

This is great to know Kurt. We can't be the only folks running into this.. I wonder if the mysql update code gets into a deadlock or something. I'm hoping a slurm dev will chime in ...

Kurt, out of band if need be, I'd be interested in the details of what you ended up doing.

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

[Hendryk Bockelmann]

Hi Chris,

we were faced with exactly the same problem - update of 16.05.11 to
17.11.3 took more than 24 hours without finalizing the conversion of job
table. Finally, we cancelled the process, went back to "old" version
16.05.11 and restored the database. At that time we had 10.5 million
jobs in db with a total db size of 11 GB.

Our "solution":
* Activate purge/archive in slurmdbd.conf keeping just 1months of data.
This reduced the db to 250000 jobs and 600 MB (needed roughly 2 hours
for initial purge)
* Update to slurm 17.11.3 with reduced db; it took just 15 minutes!

Best,
Hendryk

--
Dr. Hendryk Bockelmann
Wissenschaftliches Rechnen
Abteilung Anwendungen

Deutsches Klimarechenzentrum GmbH (DKRZ)
Bundesstraße 45 a, D-20146 Hamburg, Germany

[Jessica Nettelblad]

We experienced the same problem. On our two new clusters with smaller databases (<1 million jobs), the upgrade from 17.02.9 to 17.11.2 and 17.11.3 was quick and smooth. On the third, older cluster, where we have a larger database (>30 million jobs) the upgrade was a mess, both in mysql and mariadb. It got stuck on that exact query, energy consumption in one job table. I did some tricks to get around it, only to get stuck on other queries instead.

I put some time on it without figuring out exactly why the conversion got stuck all the time. Then I decided to install 17.11 with a fresh database, and add necessary info to it.

Basically, all our policy information is regularly imported from an external infrastructure, so we could rerun those scripts to recreate the data. Keeping our historical accounting data "hot" in the database was also not needed, although it has been convenient at times -- hence we had not been actively purging it before. All things considered, I decided not to dig deeper into the conversion issue.

We're very happy with the performance of 17.11 now that it's up and running, they've cleaned up a bunch of unnecessary locks that have caused bottlenecks for us in the past. Good luck with the conversion!

On Thu, Feb 22, 2018 at 1:30 AM, Christopher Benjamin Coffey <Chris....@nau.edu> wrote:

This is great to know Kurt. We can't be the only folks running into this.. I wonder if the mysql update code gets into a deadlock or something. I'm hoping a slurm dev will chime in ...

Kurt, out of band if need be, I'd be interested in the details of what you ended up doing.

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

[Malte Thoma]

FYI:
* We broke our upgrade from 17.02.1-2 to 17.11.2 after about 18 h.
* Dropped the job table ("truncate xyz_job_table;")
* Executed the everlasting sql command by hand on a back-up database
* Meanwhile we did the slurm upgrade (fast&easy)
* Reset the First-Job-ID to a high number
* Inserted the converted datatable in the real database again.

It took two experts for this task and we would appreciate a better upgrade-concept very much!
I fact, we hesitate to upgrade from 17.11.2 to 17.11.3, because we are afraid of similar problems. Does anyone has experience with
this?

It would be good to know if there is ANY chance if future upgrades will cause the same problems or if this will become better?

Regards,
Malte

--
Malte Thoma Tel. +49-471-4831-1828
HSM Documentation: https://goo.gl/R4drbb (User)
http://goo.gl/c4A5iE (Admin)
HPC Documentation: https://goo.gl/o435rT (User)
https://goo.gl/GMssqe (Admin)
AWI, Geb.E (3125)
Am Handelshafen 12
27570 Bremerhaven
Tel. +49-471-4831-1828

[Paul Edmon]

Typically the long db upgrades are only for major version upgrades. 
Most of the time minor versions don't take nearly as long.

At least with our upgrade from 17.02.9 to 17.11.3 the upgrade only took
1.5 hours with 6 months worth of jobs (about 10 million jobs).  We don't
track energy usage though so perhaps we avoided that particular query
due to that.

From past experience these major upgrades can take quite a bit of time
as they typically change a lot about the DB structure in between major
versions.

-Paul Edmon-

[Christopher Benjamin Coffey]

Thanks Paul. I didn't realize we were tracking energy ( . Looks like the best way to stop tracking energy is to specify what you want to track with AccountingStorageTRES ? I'll give that a try.

Best,
Chris

—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

[Miguel Gila]

We recently ran a similar exercise: when updating from 17.02.7 to 17.11.03-2, we had to stop the upgrade on our production DB (shared with other databases) after nearly half-day into it. It had reached a job table for a system with 6 million jobs and still had to go thru another one with >7 million jobs and several smaller ones...

Interestingly enough, a poor vmare VM (2CPUs, 3GB/RAM) with MariaDB 5.5.56 outperformed our central MySQL 5.5.59 (128GB, 14core, SAN) by a factor of at least 3 on every table conversion. As a result, since both DBs were upgraded in parallel with the same source production data, instead of rolling back, we decided to move all our Slurm DB queries to the VM. After giving it more RAM and more CPUs, the VM is performing quite well and are evaluating how to move forward.

Needless to say, we were all shocked by this performance difference and are still trying to figure why. CPU on the MySQL DB was nearly 80-90% busy all the time, no iowait. The "left outer join" query was the place where everything to stuck. In my eyes, all this doesn't make a lot of sense :-S

Cheers,
Miguel

--
Miguel Gila
CSCS Swiss National Supercomputing Centre
HPC Operations

[Ole Holm Nielsen]

On 22-02-2018 21:27, Christopher Benjamin Coffey wrote:
> Thanks Paul. I didn't realize we were tracking energy ( . Looks like the best way to stop tracking energy is to specify what you want to track with AccountingStorageTRES ? I'll give that a try.

Perhaps it's a good idea for a lot of sites to configure a non-default
AccountingStorageTRES in slurm.conf omitting the energy resource?

Question: Is it as simple as defining in slurm.conf:
AccountingStorageTRES=Billing,CPU,Memory,Node
and restarting slurmctld?

It seems that a "scontrol reconfig" should suffice, according to the
scontrol man-page which lists the parameters requiring a restart.

Or is the process more complicated since changes to the database are
involved? The slurm.conf man-page reads:
> AccountingStorageTRES
> Comma separated list of resources you wish to track on the cluster. These are the resources requested by the sbatch/srun job when it is submitted. Currently this consists of any GRES, BB (burst buffer) or license along with CPU, Memory, Node, and Energy. By default Billing, CPU, Energy, Memory, and Node are tracked.
>
> Whenever these resources are used on the cluster they are recorded. The TRES are automatically set up in the database on the start of the slurmctld.

/Ole

[Chris Samuel]

On Friday, 23 February 2018 8:04:50 PM AEDT Miguel Gila wrote:

> Interestingly enough, a poor vmare VM (2CPUs, 3GB/RAM) with MariaDB 5.5.56
> outperformed our central MySQL 5.5.59 (128GB, 14core, SAN) by a factor of
> at least 3 on every table conversion.

Wild idea completely out of left field..

Does the production system have the updates for Meltdown and Spectre applied,
whereas the VM setup does not?

There are meant to be large impacts from those fixes for syscall heavy
applications and databases are one of those nightmare cases...

cheers,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

[Christopher Benjamin Coffey]

Good thought Chris. Yet in our case our system does not have the spectre/meltdown kernel fix.

Just to update everyone, we performed the upgrade successfully after we purged more data jobs/steps first. We did the following to ensure the purge happened right away per Hendryk's recommendation:

ArchiveDir=/common/adm/slurmdb_archive
ArchiveEvents=yes
ArchiveJobs=yes
ArchiveSteps=no
ArchiveResvs=no
ArchiveSuspend=no
PurgeEventAfter=1month
PurgeJobAfter=2880hours # <- changing from 18months
PurgeResvAfter=2month
PurgeStepAfter=2880hours # <- changing from 18 months
PurgeSuspendAfter=2month

Specifying hours so that the purge kicked in quickly.

After having only 4 months of jobs/steps, the update took ~1.5 hrs. This was for a 334MB db with 1.1M jobs.

We've also made this change to stop tracking energy and other things we don't care about right now:

AccountingStorageTRES=cpu,mem

Hope that will help for the future.

Thanks Hendryk, and Paul :)

Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167

Chris Samuel : https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=02%7C01%7Cchris.coffey%40nau.edu%7C392a2dc2bbde477e503208d57b0aa473%7C27d49e9f89e14aa099a3d35b57b2ba03%7C0%7C0%7C636550207997139489&sdata=EdGJVxEIu5K%2Bi2yE2pxKFx7t%2BWmiwNtr6ufchjeHzPc%3D&reserved=0 : Melbourne, VIC

[Miguel Gila]

Microcode patches were not applied to the physical system, only the kernel was upgraded, so I'm not sure whether the performance hit could come from that or not.

<rant>Reducing the size of the DB to make the upgrade process complete in a reasonable time is like shooting a mosquito with a shotgun. Yeah, it works, but there must be better/easier/smarter ways of doing it. IMHO the DB upgrade process should be cleaner/safer.</rant>

For now we're staying with our VM and will see how it behaves over time.

M.

[Chris Samuel]

On Wednesday, 28 February 2018 2:13:41 AM AEDT Miguel Gila wrote:

> Microcode patches were not applied to the physical system, only the kernel
> was upgraded, so I'm not sure whether the performance hit could come from
> that or not.

Yes it would, it's the kernel changes that cause the impact. My understanding
is tha the microcode update had features that were intended to mitigate that.

Also note Intel later withdrew the microcode update due to instability on
earlier CPUs (Linux distros reverted their firmware updates at that time):

https://newsroom.intel.com/news/root-cause-of-reboot-issue-identified-updated-guidance-for-customers-and-partners/

and it appears the most recent update is intended to be pushed out via firmware
updates rather than a microcode file loaded from the OS.

https://newsroom.intel.com/news/latest-intel-security-news-updated-firmware-available/

Best of luck!

[Peter Kjellström]

On Wed, 28 Feb 2018 06:51:15 +1100
Chris Samuel <ch...@csamuel.org> wrote:

> On Wednesday, 28 February 2018 2:13:41 AM AEDT Miguel Gila wrote:
>
> > Microcode patches were not applied to the physical system, only the
> > kernel was upgraded, so I'm not sure whether the performance hit
> > could come from that or not.
>
> Yes it would, it's the kernel changes that cause the impact. My

> understanding is that the microcode update had features that were
> intended to mitigate that.

Yes and no.

The kernel has page table isolation (ie meltdown protection) regardless
of microcode level.

The microcode was half of the fix for most of spectre (together with
other kernel patches). If the microcode is unavailable these kernel
patches will not be used/activate. Look for ibrs and ibpb
in /sys/kernel/deubg/x86.

The latter part is somewhat redhat specific.

In our tests (before the microcode was reverted and ibrs/ibpb disabled)
this caused more performance impact than the page table isolation
(YMMV).

> Also note Intel later withdrew the microcode update due to
> instability on earlier CPUs (Linux distros reverted their firmware
> updates at that time):
>
> https://newsroom.intel.com/news/root-cause-of-reboot-issue-identified-updated-guidance-for-customers-and-partners/
>
> and it appears the most recent update is intended to be pushed out
> via firmware updates rather than a microcode file loaded from the OS.

Or manual downloads for admin -> micrcodectl load in linux.

/Peter

[Roshan Thomas Mathew]

We ran into this issue trying to move from 16.05.3 -> 17.11.7 with 1.5M records in job table.

In our first attempt, MySQL reported "ERROR 1206 The total number of locks exceeds the lock table size" after about 7 hours. 

Increased InnoDB Buffer Pool size - https://dba.stackexchange.com/questions/27328/how-large-should-be-mysql-innodb-buffer-pool-size - to 12G (the machine hosting mysql has 128GB) and restarted the conversion and which then completed successfully in 6.5 hours.

I am sure there are other MySQL tweaks that can be applied catered towards SLURM, will be useful if we can pool them together into the documentation.

Cheers,

Roshan 

[Ole Holm Nielsen]

On 07/18/2018 10:56 AM, Roshan Thomas Mathew wrote:
> We ran into this issue trying to move from 16.05.3 -> 17.11.7 with 1.5M
> records in job table.
>
> In our first attempt, MySQL reported "ERROR 1206 The total number of
> locks exceeds the lock table size" after about 7 hours.
>
> Increased InnoDB Buffer Pool size -
> https://dba.stackexchange.com/questions/27328/how-large-should-be-mysql-innodb-buffer-pool-size
> - to 12G (the machine hosting mysql has 128GB) and restarted the
> conversion and which then completed successfully in 6.5 hours.
>
> I am sure there are other MySQL tweaks that can be applied catered
> towards SLURM, will be useful if we can pool them together into the
> documentation.

I think this is a needle-in-haystack documentation problem :-)

The MySQL optimization has already been documented in
https://slurm.schedmd.com/accounting.html

I've summarized the information in my Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_database#mysql-configuration

/Ole

[Lech Nieroda]

We’ve run into exactly the same problem, i.e. an extremely long upgrade process to the 17.11.x major release. Luckily, we’ve found a solution.

The first approach was to tune various innodb options, like increasing the buffer pool size (8G), the log file size (64M) or the lock wait timeout (900) but that didn’t really help. Even extreme values like 40G for the buffer pool size on a 500G machine didn’t speed things up.
The conversion process still hung on the „sending data“ stage of the aforementioned query, which updates the tres_alloc attribute by using a join operation of the job and step tables. We’ve let the process run on the 500G machine for more than 90 hours and had to kill it.
Keep in mind that our slurm-database contained ca. 11 million jobs in the job table and 13 million job steps in the step table. This may seem like much but mysql should be able to handle such numbers easily.

In order to finish the slurm upgrade within the allotted maintenance window, we’ve had to purge all jobs older than a few months via the slurmdbd-purge options. Although this worked fairly quickly, it didn’t solve the problem of having to convert 11 million jobs.

Further analysis of the query has shown that the mysql optimizer has choosen the wrong execution plan. This may depend on the mysql version, ours was 5.1.69.
Apparently, the "right table" of the join operation wasn’t created first, which resulted in a massive performance loss.
The solution was straightforward - extract the creation of the "right table“ from the query and perform it first by creating a temporary table.
The result: the job table conversion was over in 17 minutes. The entire database upgrade operation was done in 43 minutes.
The jobs were re-inserted into the main database afterwards.

In case this helps, the patch is attached.

Kind regards,
Lech

[Chris Samuel]

On Monday, 1 April 2019 7:55:09 AM PDT Lech Nieroda wrote:

> Further analysis of the query has shown that the mysql optimizer has choosen
> the wrong execution plan. This may depend on the mysql version, ours was
> 5.1.69.

I suspect this is the issue documented in the release notes for 17.11:

https://github.com/SchedMD/slurm/blob/slurm-17.11/RELEASE_NOTES

NOTE FOR THOSE UPGRADING SLURMDBD: The database conversion process from
SlurmDBD 16.05 or 17.02 may not work properly with MySQL 5.1 (as was the
default version for RHEL 6). Upgrading to a newer version of MariaDB or
MySQL is strongly encouraged to prevent this problem.

--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA

[Lech Nieroda]

That’s probably it.
Sub-queries are known for potential performance issues, so one wonders why the devs didn’t extract it accordingly and made the code more robust or at least compatible with RHEL/CentOS 6 rather than including that remark in the release notes.

[Ole Holm Nielsen]

Hi Lech,

IMHO, the Slurm user community would benefit the most from your
interesting work on MySQL/MariaDB performance, if your patch could be
made against the current 18.08 and the coming 19.05 releases. This
would ensure that your work is carried forward.

Would you be able to make patches against 18.08 and 19.05? If you
submit the patches to SchedMD, my guess is that they'd be very
interested. A site with a SchedMD support contract (such as our site)
could also submit a bug report including your patch.

/Ole

[Lech Nieroda]

Hello Chris,

I’ve submitted the bug report together with a patch.
We don’t have a support contract but I suppose they’ll at least read it ;)
The code is identical for 18.08.x and 19.05.x, it’s just a different offset.

Kind regards,
Lech

[Ole Holm Nielsen]

Hi Lech,

Thanks for submitting the patch to SchedMD:
https://bugs.schedmd.com/show_bug.cgi?id=6796

SchedMD already decided that they won't fix the problem:

> Thank you for the submission, but I will not be merging this upstream at this time.
>
> Support for the 17.11 release is nearly ended, and we do not expect to make any further maintenance releases on that branch.
>
> For 18.08, we're rather late in to the lifecycle to make a change this significant to the MySQL conversion code, and I will not be passing this along in to our review process for further evaluation. Our recommendation to sites to run a modern MySQL/MariaDB version stands as our primary means of avoiding this class of issues.

Can you confirm that your patch is only relevant for an old MySQL 5.1?

On our CentOS 7 systems we run the OS's MariaDB server 5.5. Would
MySQL/MariaDB version 5.5 be affected by your patch or not?

Best regards,
Ole

On 4/3/19 12:30 PM, Lech Nieroda wrote:
> Hello Chris,
>
> I’ve submitted the bug report together with a patch.
> We don’t have a support contract but I suppose they’ll at least read it ;)
> The code is identical for 18.08.x and 19.05.x, it’s just a different offset.
>
> Kind regards,
> Lech
>
>> Am 02.04.2019 um 15:18 schrieb Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>:
>>
>> Hi Lech,
>>

>> IMHO, the Slurm user community would benefit the most from your interesting work on MySQL/MariaDB performance, if https://bugs.schedmd.com/show_bug.cgi?id=6796your patch could be made against the current 18.08 and the coming 19.05 releases. This would ensure that your work is carried forward.

[Lech Nieroda]

Hi Ole,

> Am 03.04.2019 um 12:53 schrieb Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>:
> SchedMD already decided that they won't fix the problem:

Yes, I guess it’s a bit late in the release lifecycles. Nevertheless it’s a pity, as there are certainly a lot of users around who’d rather not upgrade their distribution default mysql-servers just for the sake of a conversion.

> Can you confirm that your patch is only relevant for an old MySQL 5.1?
>
> On our CentOS 7 systems we run the OS's MariaDB server 5.5. Would MySQL/MariaDB version 5.5 be affected by your patch or not?

The patch will work with any mysql version >= 5.1, since all it does is simplify the query by changing an implicit derived table to an explicit temporary table.
This way the query complexity is reduced and its execution order doesn’t depend on the „intelligence“ of the mysql optimizer while presenting exactly the same end results.

We haven’t tested mysql 5.5 whether its optimizer chooses the right execution plan with this query.
As I’ve said, it took roughly 17 minutes with 11 million jobs, 18 million steps and a innodb buffer pool size of 8G.
If the table conversion takes more than half an hour and you don’t have tens of millions of jobs then the optimizer has a problem and the patch would help you.


Kind regards,
Lech

[Ole Holm Nielsen]

Hi Lech,

Maybe you could add your arguments to the bug report
https://bugs.schedmd.com/show_bug.cgi?id=6796 hoping that SchedMD may be
convinced that this is a useful patch for future versions of Slurm, also
for MySQL/MariaDB versions 5.5 and newer.

Best regards,
Ole

[Lech Nieroda]

Hi Ole,

since we aren’t using RHEL7/CentOS7 we haven’t tested it with mysql 5.5 and it’d probably carry more weight if someone running that OS would test it and add an appropriate comment. You are welcome to try it out.
That being said, the release notes explicitly mention that versions 5.1 and 5.5 should be avoided for the conversion process (probably due to this bug as I haven’t encountered any other issues) and the dev stated that they’d rather keep that warning than fixing the issue, so I’m not sure if that’ll be enough to convince them.

Kind regards,
Lech

[Prentice Bisbal]

> the dev stated that they’d rather keep that warning than fixing the issue, so I’m not sure if that’ll be enough to convince them.

Anyone else as disappointed by this as I am? I get that it's too late to
add something like this to 17.11 or 18.08, but it seems like SchedMD
isn't even interested in looking at this for 19.x If a Linux distro
comes with a particular version of a MySQL or MariaDB, and SchedMD says
they support that version of the distro, then they should support the
version of the DB that comes with that distro. From what I gather from
these discussion so far, SchedMD is basically saying we support Linux
distro X, but not the MySQL/MariaDB version that comes with that distro.
Is that a correct reading of this situation?

--
Prentice

[Chris Samuel]

On Wednesday, 3 April 2019 6:33:17 AM PDT Prentice Bisbal wrote:

> Anyone else as disappointed by this as I am? I get that it's too late to
> add something like this to 17.11 or 18.08, but it seems like SchedMD
> isn't even interested in looking at this for 19.x

Not really surprising, it's not applicable to 19.05 (which is coming up
quickly now and so they'll be busy trying to get ready for that).

"Release dates in calendar are closer than they appear"

All the best,
Chris

[Lech Nieroda]

That’s correct but let’s keep in mind that it only concerns the upgrade process and not production runtime which has certain implications.
The affected database structures have been introduced in 17.11 and an upgrade affects only versions 17.02 or prior, it wouldn’t be a problem for users who have made a fresh install of 17.11 or newer.
Furthermore, upgrades shouldn’t skip more than one release, as that would lead to loss of state files and other important information, so users probably won’t upgrade from 17.02 to 19.05 directly. If they’d do that then yes, the patch would be applicable for 19.x, it’s just less likely to occur.
It’s most needed in 17.11 and 18.08 but won’t be included due to late stage in their lifecycle. An understandable decision.

To sum it up, the issue affects those users who still have 17.02 or prior versions, use their distribution defaults for mysql/mariadb from RHEL6/CentOS6 and RHEL7/CentOS7, have millions of jobs in their database *and* would like to upgrade slurm without upgrading mysql.
Those few unfortunate souls will perhaps find this thread and use the patch ;-)

Kind regards,
Lech

[Chris Samuel]

On 4/4/19 4:07 am, Lech Nieroda wrote:

> Furthermore, upgrades shouldn’t skip more than one release, as that would lead to loss of state files and other important information, so users probably won’t upgrade from 17.02 to 19.05 directly. If they’d do that then yes, the patch would be applicable for 19.x, it’s just less likely to occur.

Upgrading more than 2 releases isn't supported, so I don't believe the
19.05 slurmdbd will have the code in it to upgrade tables from earlier
than 17.11.

People still on 17.02 and earlier will have to go to an intermediate
release to get their tables properly updated (and as you say, either
apply your patch or migrate their MySQL server to a box running a more
recent version of MySQL - it doesn't have to be on the same system
running slurmdbd).

All the best,
Chris

[Lech Nieroda]

> Upgrading more than 2 releases isn't supported, so I don't believe the 19.05 slurmdbd will have the code in it to upgrade tables from earlier than 17.11.

I haven’t found any mention of this in the upgrade section of the QuickStart guide (see https://slurm.schedmd.com/quickstart_admin.html#upgrade ), is the lack of support described somewhere else?
The guide states:

"Slurm permits upgrades between any two versions whose major release numbers differ by two or less (e.g. 16.05.x or 17.02.x to 17.11.x) without loss of jobs or other state information. State information from older versions will not be recognized and will be discarded, resulting in loss of all running and pending jobs."

Upgrades from older versions are possible but result in the loss of state files. While it is possible, it is not very practical. One would have to make an offline upgrade where the servers and clients are upgraded at the same time, otherwise they wouldn’t be able to communicate.
The code for converting 17.02 and prior versions is present in 19.05pre.

Kind regards,
Lech

[Prentice Bisbal]

Lech,

Thanks for the explanation. Now that you explained it like that, I
understand SchedMD's decision. I was misreading the situation. I was
under the impression that this affected *all* db upgrades, not just
those from one old version a slightly less older version.

Prentice

[Ole Holm Nielsen]

Hi Lech,

I've tried to summarize your work on the Slurm database upgrade patch in
my Slurm Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#database-upgrade-from-slurm-17-02-and-older

Could you kindly check if my notes are correct and complete? Hopefully
this Wiki will also help others.

Best regards,
Ole

[Lech Nieroda]

Hi Ole,

your summary is correct as far as I can tell and will hopefully help some users.
One thing I’d add is the remark from the 18.08 Release Notes ( https://github.com/SchedMD/slurm/blob/slurm-18.08/RELEASE_NOTES ), which adds mysql 5.5 to the list.
They’ve mentioned that mysql 5.5 is the default for RHEL6 but it’s the default for RHEL7, isn’t it? Assuming that you use RHEL7/CentOS7 with mysql 5.5, have you checked how long your upgrade would take with the patch?

Kind regards,
Lech

[Ole Holm Nielsen]

Hi Lech,

Thanks! I added the 18.08 Release Notes reference to
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#database-upgrade-from-slurm-17-02-and-older

I've already upgraded from 17.11 to 18.08 without your patch, and this
went smoothly as expected. We upgraded from 17.02 to 17.11 last summer,
also without any problems. Testing the database upgrade on a test
server is crucial as always, see suggestions in my Wiki page section
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#make-a-dry-run-database-upgrade

Best regards,
Ole

On 4/5/19 10:27 AM, Lech Nieroda wrote:
> Hi Ole,
>
> your summary is correct as far as I can tell and will hopefully help some users.
> One thing I’d add is the remark from the 18.08 Release Notes ( https://github.com/SchedMD/slurm/blob/slurm-18.08/RELEASE_NOTES ), which adds mysql 5.5 to the list.
> They’ve mentioned that mysql 5.5 is the default for RHEL6 but it’s the default for RHEL7, isn’t it? Assuming that you use RHEL7/CentOS7 with mysql 5.5, have you checked how long your upgrade would take with the patch?
>
> Kind regards,
> Lech
> >>
>>
>
>

--
Ole Holm Nielsen
PhD, Senior HPC Officer
Department of Physics, Technical University of Denmark,
Building 307, DK-2800 Kongens Lyngby, Denmark
E-mail: Ole.H....@fysik.dtu.dk
Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/
Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620