[slurm-users] SLURM on Ubuntu 16.04

[Eric F. Alemany]

Greetings,

New to the forum and new to SLURM. I have 5 servers. 1 can be the master/headnode and 4 can be the compute node. My goal is to help a post-doc run “jobs” on the cluster and utilizing all the CPU’s and RAM from the 4 compute nodes. The post-doc runs radiation Monte Carlo simulation, RNA/DNA sequencing, DESEQ..etc

I understand this is a very general question and i am sure there is a lot to take in consideration but if someone could guide through simple steps on how to install and configure SLURM on ubuntu for 1 master/headnode and 4 compute nodes I would be very grateful.

Thank you for your time and help.

Best,

Eric

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Andy Riebs]

Let me recommend to you the excellent "Quick Start" documentation at SchedMD.com -- yours should be a very easy cluster to set up.

Andy

-- 
Andy Riebs
andy....@hpe.com
Hewlett-Packard Enterprise
High Performance Computing Software Engineering
+1 404 648 9024
My opinions are not necessarily those of HPE
    May the source be with you!

[evan clark]

I would also recommend this guide, it helped quite a bit when getting my personal cluster online. https://wiki.fysik.dtu.dk/niflheim/SLURM

One recommendation is to have ntp, shared storage for home directories, and LDAP so user ids are synced across machines.

Eric F. Alemany

April 25, 2018 3:27 PM

[John Kelly]

Hi Eric

I found this site very useful 

https://github.com/mknoxnv/ubuntu-slurm

-jfk

[Eric F. Alemany]

Hi John, 

Thank you for your reply.

I followed the steps on the same site and am careful with all the steps but i still cannot make it work. Some people say that Ubuntu has packages for SLURM and i dont have to deal with downloading tar ball and make sure that all the files go to the right place.

Thanks 

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Eric F. Alemany]

Hi Evan,

Thank you for your message. I saw that website which is well done but i am not familiar with CentOS and RedHat. I get the idea but i dont seem to be able to install and configure SLURM on Ubuntu.

Thanks 

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[John Kelly]

Hi Eric

I installed on ubuntu 14.04 and I did use the tar ball.  What error did you get ?   

-jfk

[Evan Clark]

What issues are you encountering? Building the binaries or getting it running?

—

Regards,

Evan Clark

[Patrick Goetz]

Hi Eric -

Did you follow my suggestion of -- on 18.04, mind you; the packages on
16.04 are too old --

- Install the slurmctld package on the SMS (the master)
- Install the slurmd package on the nodes?

You'll still need to do some configuration, but my guess is this will
pull in the necessary dependencies and set everything up for you.

On 04/25/2018 04:15 PM, Eric F. Alemany wrote:
> Hi John,
>
> Thank you for your reply.
>
> I followed the steps on the same site and am careful with all the steps
> but i still cannot make it work. Some people say that Ubuntu has
> packages for SLURM and i dont have to deal with downloading tar ball and
> make sure that all the files go to the right place.
>
> Thanks
> _____________________________________________________________________________________________________
>

> *
> *Eric F.  Alemany*
> *
> /System Administrator for Research/

>
> Division of Radiation & Cancer  Biology
> Department of Radiation Oncology
>
> Stanford University School of Medicine
> Stanford, California 94305
>

> Tel:1-650-498-7969 <tel:1-650-498-7969>No Texting
> Fax:1-650-723-7382 <tel:1-650-723-7382>

>
>
>
>> On Apr 25, 2018, at 1:52 PM, John Kelly <john....@broadcom.com

>> <mailto:john....@broadcom.com>> wrote:
>>
>> Hi Eric
>>
>> I found this site very useful
>>
>> https://github.com/mknoxnv/ubuntu-slurm
>>
>> -jfk
>>
>> On Wed, Apr 25, 2018 at 1:01 PM, evan clark <ecla...@fau.edu
>> <mailto:ecla...@fau.edu>> wrote:
>>
>> I would also recommend this guide, it helped quite a bit when
>> getting my personal cluster online.
>> https://wiki.fysik.dtu.dk/niflheim/SLURM
>> <https://wiki.fysik.dtu.dk/niflheim/SLURM>
>>
>> One recommendation is to have ntp, shared storage for home
>> directories, and LDAP so user ids are synced across machines.
>>

>>> Eric F. Alemany <mailto:eale...@stanford.edu>

>>> April 25, 2018 3:27 PM
>>> Greetings,
>>>
>>> New to the forum and new to SLURM. I have 5 servers. 1 can be the
>>> master/headnode and 4 can be the compute node. My goal is to help
>>> a post-doc run “jobs” on the cluster and utilizing all the CPU’s
>>> and RAM from the 4 compute nodes. The post-doc runs radiation
>>> Monte Carlo simulation, RNA/DNA sequencing, DESEQ..etc
>>> I understand this is a very general question and i am sure there
>>> is a lot to take in consideration but if someone could guide
>>> through simple steps on how to install and configure SLURM on
>>> ubuntu for 1 master/headnode and 4 compute nodes I would be very
>>> grateful.
>>> Thank you for your time and help.
>>>
>>> Best,
>>> Eric
>>> _____________________________________________________________________________________________________
>>>

>>> *
>>> *Eric F.  Alemany*
>>> *
>>> /System Administrator for Research/

>>>
>>> Division of Radiation & Cancer  Biology
>>> Department of Radiation Oncology
>>>
>>> Stanford University School of Medicine
>>> Stanford, California 94305
>>>

>>> Tel:1-650-498-7969 <tel:1-650-498-7969>No Texting
>>> Fax:1-650-723-7382 <tel:1-650-723-7382>
>>>
>>>
>>>
>>
>>
>

[Eric F. Alemany]

HI John,

Each time i enabled/started slurmctld the error was that the slurmctld.service could not be find or did not exist when in fact it existed in the correct directory /etc/systemd/system.

Same with slurmdbd.service.

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[John Kelly]

Hi Eric

Did you try starting the slurmctl by itself with slurmctl -Dcv to see what errors it gives ? 

-jfk

[Eric F. Alemany]

Hi Patrick,

I am going to follow your suggestion to install slurm via ubuntu 18.04 package. I am waiting for the final release of 18.04 which i believe is tomorrow.

As you mentioned I will install:

the slurmctld package on the SMS (the master)

the slurmd package on the nodes

I know there will be some other configuration i would still need to do like the slurm.conf file. The slurm.conf needs to be the same on all nodes including the master.

And then there is the database configuration as well.

 

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Christopher Samuel]

On 26/04/18 09:49, Eric F. Alemany wrote:

> I am going to follow your suggestion to install slurm via ubuntu
> 18.04 package.

Just be aware that the version in bionic is outdated, it's 17.11.2.

Most importantly you will want to be sure that they have backported the
patch to close CVE-2018-7033 (fixed in 17.11.5).

Good luck,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

[Christopher Samuel]

On 26/04/18 09:58, Christopher Samuel wrote:

> Most importantly you will want to be sure that they have backported the
> patch to close CVE-2018-7033 (fixed in 17.11.5).

Went and found their sources, there is no mention of this being fixed
in the proposed version, so it seems that bionic will ship Slurm with
this CVE unpatched. :-(

According to:
https://people.canonical.com/~ubuntu-security/cve/2018/CVE-2018-7033.html

it is listed as still "needs triage".

Also unfixed in any Debian release too.

https://security-tracker.debian.org/tracker/CVE-2018-7033

cheers,

[Eric F. Alemany]

Hi Chris,

Thank you for the update(s). It is what it is - right ? 

_____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Patrick Goetz]

Hi Chris -

He has 4 nodes and one master. I'm pretty sure he's not going to be
using slurmdbd? Of course something to keep in mind if things work out
so well that his organization is commanding him to order an additional
thousand nodes in 6 months.

[Patrick Goetz]

I don't think the problem Chris is referring to (a SQL injection attack)
is going to apply to you because you're way too small to need to worry
about Slurm accounting, but if it is a concern, install the distro
packages; confirm that things are roughly working and then just take
note of how things are set up. You can use

dpkg -L <package_name>

to see what files are installed where, and the initial install will tell
you what package dependencies are being pulled in. This will help you
do a setup from the tarball.

Oh, and it is possible that the Ubuntu package will helpfully turn on
the slurmdbd for you. Find out what service it's running as
(slurmdbd.service?) and mask it:

systemctl stop slurmdbd
systemctl mask slurmdbd

Disclaimer: I don't currently use the Slurm accounting system and have
no idea how it is set up, so the instructions above are likely entirely
incorrect.

On 04/25/2018 07:11 PM, Eric F. Alemany wrote:
> Hi Chris,
>
> Thank you for the update(s). It is what it is - right ?
>
> _____________________________________________________________________________________________________
>

> *
> *Eric F.  Alemany*
> *
> /System Administrator for Research/
>

> Division of Radiation & Cancer  Biology
> Department of Radiation Oncology
>
> Stanford University School of Medicine
> Stanford, California 94305
>

> Tel:1-650-498-7969 <tel:1-650-498-7969>No Texting
> Fax:1-650-723-7382 <tel:1-650-723-7382>

>
>
>
>> On Apr 25, 2018, at 5:03 PM, Christopher Samuel <ch...@csamuel.org

[Eric F. Alemany]

Good morning 

As you mentioned right now I have 1 master and 4 nodes and couple post-docs. I think we can manage it without the SQL ( MariaDB) accounting feature.

I read in the schedmd website that I can use a text file instead of the sql database.

I will still need to found out how to configure that. But today, I'll be upgrading my master and nodes to ubuntu 18.04.

._____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Chris Samuel]

On Thursday, 26 April 2018 11:27:35 PM AEST Patrick Goetz wrote:

> He has 4 nodes and one master. I'm pretty sure he's not going to be
> using slurmdbd?

No, he may have a greater need for it to be able to do fair-share (or some
other sort of balancing or hard limits) over a smaller amount of resources.

A lot depends on the number of users, number of jobs, size of jobs and
walltimes of them.

All the best,

[Eric F. Alemany]

Hi Chris

Thank you for your message. At this point I want to get familiar with SLURM the easiest way. I have two post-docs I'd like to test SLURM with. Their jobs are pretty big, they run Monte Carlo simulation. Again, this would be a test environment and can always be changed, re-configured later to answer the post-doc needs.

Cheers

._____________________________________________________________________________________________________

Eric F.  Alemany

System Administrator for Research

Division of Radiation & Cancer  Biology

Department of Radiation Oncology

Stanford University School of Medicine

Stanford, California 94305

Tel:1-650-498-7969  No Texting

Fax:1-650-723-7382

[Roland Fehrenbacher]

>>>>> "E" == Eric F Alemany <eale...@stanford.edu> writes:

Hi Eric,

sorry, I'm a bit late to this thread. You might want to have a look at
Qlustar as an easy cluster solution based on Ubuntu 16.04, version 10.0
just released recently. Slurm is fully setup after installation, nice
management GUI, with Slurm component, and a huge lot more. It's now
100% Open Source and free.

Best,

Roland

-------
https://www.q-leap.com / https://qlustar.com
--- HPC / Storage / Cloud Linux Cluster OS ---

E> Hi Chris Thank you for your message. At this point I want to get
E> familiar with SLURM the easiest way. I have two post-docs I'd
E> like to test SLURM with. Their jobs are pretty big, they run
E> Monte Carlo simulation. Again, this would be a test environment
E> and can always be changed, re-configured later to answer the
E> post-doc needs.

E> Cheers

E> ._____________________________________________________________________________________________________

E> Eric F. Alemany System Administrator for Research

E> Division of Radiation & Cancer Biology Department of Radiation
E> Oncology

E> Stanford University School of Medicine Stanford, California 94305

E> Tel:1-650-498-7969 No Texting Fax:1-650-723-7382

E> On Apr 26, 2018, at 21:55, Chris Samuel <ch...@csamuel.org>
E> wrote:

E> On Thursday, 26 April 2018 11:27:35 PM AEST Patrick Goetz
E> wrote:

E> He has 4 nodes and one master. I'm pretty sure he's not
E> going to be

E> using slurmdbd?

E> No, he may have a greater need for it to be able to do
E> fair-share (or some other sort of balancing or hard limits)
E> over a smaller amount of resources.

E> A lot depends on the number of users, number of jobs, size of
E> jobs and walltimes of them.

E> All the best, Chris -- Chris Samuel :
E> http://www.csamuel.org/ : Melbourne, VIC