Installation on Slurm cluster

[Boonthum Kunyangyuen]

I installed and tested basic materials on my laptop (using linux-ubuntu). But I wonder how should I do If I want to install on Slurm Cluster? Should I use git clone and compile same in my computer?

[gabo...@gmail.com]

I don't have a Slurm cluster to try it, but there is a good chance that the same steps seen on slides 12-13 in [1] will work such as:

git clone https://github.com/richard-evans/vampire.git

cd vampire

make parallel

However, it looks like the "qsub job.sh" step would need to change.  In [2], it has that changing to "sbatch job.sh".

[1] https://vampire.york.ac.uk/resources/workshop2017_day5.pdf

[2] https://www.nrel.gov/hpc/assets/pdfs/pbs-to-slurm-translation-sheet.pdf

Kind Regards,

Gavin

VAMPIRE user

[milton...@gmail.com]

I think you need to use "make parallel-archer", but it's easiest to just contact the support at the cluster you're using. Either you ask how you can install it locally or if they can install it globally.

Here are the instructions for how to do it on the slurm cluster I work on: (Thanks to Tor at the SNIC technical support!)
cd <path to your home folder on the cluster>
mkdir vampire_local_install
cd vampire_local_install
git clone git://github.com/richard-evans/vampire.git
cd vampire

module swap PrgEnv-cray PrgEnv-gnu
make parallel-archer

This should produce the binary you need: vampire-parallel
If you want to leave it there, you can use this one in any
computation with the batch script:

#!/bin/bash -l
#SBATCH -J vampire-test
#SBATCH -t 03:59:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
# load the vampire module

module swap PrgEnv-cray PrgEnv-gnu
export VAMP_LOCAL=<path to your home folder on the cluster>/vampire_local_installation/vampire/

srun $VAMP_LOCAL/vampire-parallel

Hope that works for you!
/Milton

[p201...@hyderabad.bits-pilani.ac.in]

Dear Vampire Users,

I am a new user. We have an HPC facility in our institute.  I think I successfully installed vampire software in HPC using the command  ''git clone https://github.com/richard-evans/vampire.git''.  After watching vampire workshop videos and going through the forum, I understand that two files are required to run the simulation. One is input, and another is material file. I need your help running the sample program provided in the vampire directory, i.e. Co.mat. I have gone through Gavin's suggestions for creating a job.sh file, but I am still unclear. Could you please help me to create the job file? 

One of Gavin's suggestions mentions that GPU-based simulations are "Limited feature set and some persistent bugs… but around 20x faster than serial code". Could you please elaborate on this? Is it good to use GPU-based computations?

For running mumax3 simulations, I used the following job.sh file. 

#!/bin/bash
#SBATCH -p gpu_v1
#SBATCH --gres=gpu:1
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mem 1500M
#SBATCH -t 0-01:00 # time (D-HH:MM)
#SBATCH --job-name="CoPt"
#SBATCH -o slurm.%j.out
#SBATCH -e slurm.%j.err
#SBATCH --mail-user=p201...@hyderabad.bits-pilani.ac.in
#SBATCH --mail-type=ALL
srun ./mumax3 CoPt.txt

I am attaching a screenshot of the installation process for your reference.

Best regards,

Sateesh

[p201...@hyderabad.bits-pilani.ac.in]

Would anyone be able to help me with running vampire simulations on HPC? I appreciate any help you can provide.

Best regards,

Sateesh

[gabo...@gmail.com]

If you're going to use the GPU executable to run VAMPIRE calculations, then you would first need to know what NVIDIA graphics cards your HPC has.  You would also need to know if CUDA is installed on the HPC.  VAMPIRE 6.0 currently only supports NVIDIA graphics cards that have compute capability of 3.0, 3.5, 5.0, and 6.1.  So, you would most likely have to contact your HPC help desk to find out what that is unless you have a website for your HPC that has that information listed.

For users new to VAMPIRE, I would recommend learning to use a CPU executable (which would be either vampire-serial or vampire-parallel) for running a VAMPIRE calculation first.

You would have to build the vampire-serial and/or vampire-parallel executables to be able to do that.  However, I'm not seeing that you have done that in the screenshot you sent.  You would have to navigate into the folder where you installed vampire and run the make command there.

After the executable has been successfully generated by make, you will most likely want to add that to your PATH.  That way you would only have to specify the executable name and not have to give the full path to the exectuable every time you want to use it.

In your job.sh file, it looks like you have GPU lines:

#SBATCH -p gpu_v1
#SBATCH --gres=gpu:1

You would need to consult your HPC documentation and change those lines to use a CPU instead.

The srun command you have for mumax3:

#SBATCH --job-name="CoPt"

...

srun ./mumax3 CoPt.txt

For the job.sh in the same folder as your input and Co.mat files, you should just need to change that use for example the serial calculation executable for VAMPIRE:

#SBATCH --job-name="Co"

...

srun vampire-serial

For vampire-parallel, you may want to see an example given by a VAMPIRE user at the following link:

https://groups.google.com/g/vampire-users/c/MuXsxtSLw2c/m/xFz_BUXkAQAJ

Kind Regards,

Gavin

VAMPIRE user