Running jobs on buckethead

[Lukasz Kaczmraczyk]

This is alternative to rdb server. We will still use rdb for development and buckethead could be used to run longer jobs. If you have an account on buckethead you can install mofem and run jobs.

Part 1: Installation

To install mofem, follow steps:

1. Edit .bashrc and add following lines at the end

export PATH=/home/staff/lk58p/lk58p/mofem_installation/local/bin:/home/staff/lk58p/mofem_installation/petsc/arch-linux2-c-opt/bin:$PATH

module load gcc/6.4.0
module load gridengine

export MOFEM_INSTALL_DIR=$HOME/mofem_installation

Logout and log in to your account.

2. Create directory and clone source

mkdir $MOFEM_INSTALL_DIR

cd $MOFEM_INSTALL_DIR
git clone https://bitbucket.org/likask/mofem-cephas.git mofem-cephas

3. Configure core library

mkdir $MOFEM_INSTALL_DIR/lib
cd  $MOFEM_INSTALL_DIR/lib

cmake -DCMAKE_BUILD_TYPE=Release -DPETSC_DIR=/home/staff/lk58p/mofem_installation/petsc/ -DPETSC_ARCH=arch-linux2-c-opt -DMOAB_DIR=/home/staff/lk58p/mofem_installation/petsc/arch-linux2-c-opt/ -DWITH_ADOL-C=1 -DWITH_TETGEN=1  -DWITH_MED=1 -DCMAKE_INSTALL_PREFIX=$MOFEM_INSTALL_DIR/users_modules $MOFEM_INSTALL_DIR/mofem-cephas/mofem

4.  Make, install and test

make -j4 install
ctest -D Experimental

5. Configure user modules

cd $MOFEM_INSTALL_DIR/users_modules
cmake -DCMAKE_BUILD_TYPE=Release users_modules

* Clone your users modules

6. Make users modules

make -j4
ctest -D Experimental

Part 2: Running jobs

1. Submitting job

If you are login into buckethead you are in head node. You CAN NOT run jobs on the head node, but you have to submit job into queue. If you do not know how it works, please first contact me.

#! /bin/bash


# The job's name
#$ -N MoFEM


# The queue in which to run the job
#$ -q test


# File to which standard error should be directed
#$ -e ./stderr


# File to which standard output should be directed
#$ -o ./stdout


# E-mail address to which status updates should be sent
# N.B.: in an array job, a separate e-mail will be sent for each task!
#$ -M my_e...@glasgow.ac.uk


# Events on which to send a status update
#$ -m beas


# Request for 1.0 GB of memory per task (needed on Miffy and Dusty)
#$ -l mem_tokens=1.0G


#$ -pe mpi 1 # where N is the number of processors required


# List of commands which do the actual work
echo "$NSLOTS received"
cat $PE_HOSTFILE


# List of commands which do the actual work

echo "Hellow world"

# WORKIND_DIR=/home/staff/lk58p/mofem_installation/users_moules/fracture_mechanics/for_francios


# cd $WORKIND_DIR
# /home/staff/lk58p/mofem_installation/petsc/arch-linux2-c-#opt/bin/mpirun -np $NSLOTS ./crack_propagation -my_file #out.h5m  2>&1 | tee -a log

2. Queues 

Above example is for gcec.q queue, this is Glasgow Computational Engineering Centre (GCEC) where on our new nodes. However, you can submit job to queues, 

LUSTER QUEUE                   CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE  
--------------------------------------------------------------------------------
gcec.q                            0.41     24      0    320    376      0     32
materials.q                       0.41     16      0    272    376      0     88
pg-long.q                         0.41    257      0      3    376      0    116
postgrad.q                        0.40      2      0    262    380      0    116
staff.q                           0.41      0      0    260    376      0    116
test                              0.43      0      0    104    112      0      8
ug-long.q                         0.41      0      0    260    376      0    116
undergrad.q                       0.41      0      0    260    376      0    116

3. Note I do not give you information how to submit a job. if you don't know this contact me. 

4. More information:

http://malmsteen.eng.gla.ac.uk/wiki/hpc-wiki/index.php/Main_Page

[ignat...@gmail.com]

Hello,

I am currently going through the process presented in this post and a found two points that might need to be changed.

The process run smoothly until make in Part 1/step 3.

Instead of typing cmake command, cmake3 should be used since cmake is at an older version than the one required.

After that, all runs smoothly until cmake  in Part 1/step 5 where this error appears:

CMake Error: The source directory "/home/staff/ia45r/mofem_installation/users_modules/users_modules" does not appear to contain CMakeLists.txt.

Specify --help for usage, or press the help button on the CMake GUI.

This is because the clone command in Part 1/step 2   should be:

git cc git clone --recurse-submodules https://bitbucket.org/likask/mofem-cephas.git mofem-cephas

instead of:

git clone https://bitbucket.org/likask/mofem-cephas.git mofem-cephas

Thanks,

Ignatios

[Lukasz Kaczmraczyk]

Ignatios,

Thanks for verifying this, and correcting. 

I drop using direct installation and only using spack. I recommend following this instruction, 

http://mofem.eng.gla.ac.uk/mofem/html/install_spack.html#spack_buckedhead

Kind regards,

Lukasz

[ignat...@gmail.com]

Hello Lukasz,

Thank you for your reply.

I will switch to spack as suggested.

Kind regards,

Ignatios