INLA on HPC server causes core dumps

David Dayi Li

unread,

Jul 7, 2021, 8:09:56 PM7/7/21

to R-inla discussion group

Hi,

I am running LGCP simulation with inlabru and SPDE on a HPC server. But I kept on getting core dump for every simulation that is run. I have contacted support staff of the HPC server about this issue, and they said the problem is from INLA. Below is the result I got after trying to view one of the core files:

$ gdb core core.224843

[New LWP 224843]

Core was generated by `grep ^export'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0x0000564fdff492a8 in ?? ()

The support staff does not really know what is going on, and I have no idea what is happening here. The code I am running is outputting the result I wanted, so there should not be any program crashing.

Although I can ignore these core dumps, they become a problem if I want to do tens of thousands of simulation. Them being outputted to my remote server directory is likely to cause storage and access problem. Does anyone know what is happening here?

Thanks,

David

Helpdesk

unread,

Jul 8, 2021, 3:21:07 AM7/8/21

to David Dayi Li, R-inla discussion group

can you do

Sys.setenv(INLA_DEBUG=1)
inla(y~1,data=data.frame(y=0), verbose=TRUE,debug=TRUE)

and send me the output? hopefully, it should be clear from this what is
going on.

> --
> You received this message because you are subscribed to the Google
> Groups "R-inla discussion group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to r-inla-discussion...@googlegroups.com.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/r-inla-discussion-group/beef4b96-4714-4025-b651-19a68871b8a2n%40googlegroups.com
> .

--
Håvard Rue
he...@r-inla.org

David Dayi Li

unread,

Jul 8, 2021, 1:37:38 PM7/8/21

to R-inla discussion group

Hi Havard,

I tried your code and I found the part where it outputted segmentation fault. The result is here:

Run inla...*** /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run *** LD_PRELOAD=/scratch/dli346/R_locals/INLA/bin/linux/64bit/first/libjemalloc.so.2

/scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run: line 48: 24897 Segmentation fault (core dumped) ldd -r "$DIR/$prog"

*** /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run *** exec /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl -b -s -v -t32:1 -B0 /tmp/RtmpJ66xAR/file60a3525beaf8/Model.ini

Is there any fix for this?

Thanks,

David

INLA help

unread,

Jul 8, 2021, 2:22:00 PM7/8/21

to R-inla discussion group, David Dayi Li

And this the centos7 build? And it’s supposed to be the correct one?

—

Haavard Rue

HelpDesk

help@r-inla. org

To view this discussion on the web, visit https://groups.google.com/d/msgid/r-inla-discussion-group/199b22b9-019b-48fe-8bfb-6a1266ce4b96n%40googlegroups.com.

David Dayi Li

unread,

Jul 8, 2021, 2:39:17 PM7/8/21

to R-inla discussion group

Yes, this is the centOS7 build and I checked my HPC system version, it is centOS7. The support staff of HPC server just emailed me that the error is a relocation error. But it does not affect the program from running. Hope this helps.

David

Helpdesk

unread,

Jul 9, 2021, 12:58:49 PM7/9/21

to David Dayi Li, R-inla discussion group

and this fails as well I guess (do in shell)

export INLA_DEBUG=1
bash -vx /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run

???

David Dayi Li

unread,

Jul 9, 2021, 1:03:53 PM7/9/21

to R-inla discussion group

Hi Harvard,

This is the output I got:

#!/bin/bash

# -*- shell-script -*-

#

########################################################################

# Start Lmod BASHENV

########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then

case "$-" in

*v*x*) __lmod_vx='vx' ;;

*v*) __lmod_vx='v' ;;

*x*) __lmod_vx='x' ;;

esac;

fi

+ '[' -z '' ']'

+ case "$-" in

+ __lmod_vx=vx

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx

+ '[' -n vx ']'

+ set +vx

Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/init/bash)

Shell debugging restarted

+ unset __lmod_vx

########################################################################

# End Lmod BASHENV

########################################################################

#

# Local Variables:

# mode: shell-script

# indent-tabs-mode: nil

# End:

#!/bin/bash

cmd=$(readlink -e "$0")

++ readlink -e /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run

+ cmd=/scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.run

DIR=$(readlink -e $(dirname "$cmd"))

+++ dirname /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.run

++ readlink -e /scratch/dli346/R_locals/INLA/bin/linux/64bit

+ DIR=/scratch/dli346/R_locals/INLA/bin/linux/64bit

tmp=$(basename "$0")

++ basename /scratch/dli346/R_locals/INLA/bin/linux/64bit/inla.mkl.run

+ tmp=inla.mkl.run

prog=${tmp%%.run}

+ prog=inla.mkl

D=""

+ D=

if [ ! -z ${R_HOME+x} ]; then

d="$R_HOME/lib"

if [ -d "$d" ]; then

D=$d

fi

+ '[' '!' -z ']'

for d in {,/usr}/lib64 /usr/lib64/R/lib {,/usr}/lib/x86_64-linux-gnu {,/usr}/lib; do

if [ -d "$d" ]; then

if [ -z "$D" ]; then

D="$d"

else

D="$D:$d"

fi

done