Getting forrtl: severe (174): SIGSEGV, segmentation fault occurred when running FDS v6.4.0

2,146 views

Skip to first unread message

Michael Abboud

unread,

Aug 12, 2016, 2:56:26 AM8/12/16

to FDS and Smokeview Discussions

Hello,

I'm attempting to run an FDS6 file generated by PyroSim Version 2016.1.0425

I have a node running CentOS Linux v7, OpenMPI v1.8.4 precompiled binaries and FDS_6.4.0-SMV_6.3.6_linux64 precompiled binaries all installed.

The PyroSimVersion 2016.1.0425 Release Notes explicitly state: This version of PyroSim is designed for FDS version 6.4.0 and Smokeview version 6.3.6

Hence the specific versions in use here.

However, if I execute fds against my PyroSim file, I get the following output and error:

Mesh 1 is assigned to MPI Process 0

Mesh 2 is assigned to MPI Process 1

Mesh 3 is assigned to MPI Process 2

Mesh 4 is assigned to MPI Process 3

Mesh 5 is assigned to MPI Process 4

Mesh 6 is assigned to MPI Process 5

OpenMP thread 3 of 3 assigned to MPI process 0 of 5

OpenMP thread 1 of 3 assigned to MPI process 0 of 5

OpenMP thread 2 of 3 assigned to MPI process 0 of 5

OpenMP thread 0 of 3 assigned to MPI process 0 of 5

OpenMP thread 2 of 3 assigned to MPI process 5 of 5

OpenMP thread 3 of 3 assigned to MPI process 1 of 5

OpenMP thread 1 of 3 assigned to MPI process 1 of 5

OpenMP thread 3 of 3 assigned to MPI process 4 of 5

OpenMP thread 1 of 3 assigned to MPI process 4 of 5

OpenMP thread 3 of 3 assigned to MPI process 5 of 5

OpenMP thread 1 of 3 assigned to MPI process 5 of 5

OpenMP thread 0 of 3 assigned to MPI process 5 of 5

OpenMP thread 2 of 3 assigned to MPI process 4 of 5

OpenMP thread 0 of 3 assigned to MPI process 4 of 5

OpenMP thread 3 of 3 assigned to MPI process 2 of 5

OpenMP thread 2 of 3 assigned to MPI process 1 of 5

OpenMP thread 0 of 3 assigned to MPI process 1 of 5

OpenMP thread 1 of 3 assigned to MPI process 2 of 5

OpenMP thread 2 of 3 assigned to MPI process 2 of 5

OpenMP thread 0 of 3 assigned to MPI process 2 of 5

OpenMP thread 3 of 3 assigned to MPI process 3 of 5

OpenMP thread 2 of 3 assigned to MPI process 3 of 5

OpenMP thread 1 of 3 assigned to MPI process 3 of 5

OpenMP thread 0 of 3 assigned to MPI process 3 of 5

Completed Initialization Step 1

Completed Initialization Step 2

Completed Initialization Step 3

Completed Initialization Step 4

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007156A0 Unknown Unknown Unknown

libpthread.so.0 00007FDD958BA100 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000B3DF36 radcompute_radiat 1275 radi.f90

fds6_From_Live_Cl 0000000000B30474 rad_mp_compute_ra 602 radi.f90

fds6_From_Live_Cl 00000000007FC17E MAIN__ 297 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007FDD9550AB15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

forrtl: error (78): process killed (SIGTERM)

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000071566E Unknown Unknown Unknown

libpthread.so.0 00007F05CBC29100 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000063E0E2 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000424DFB Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004E0F07 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000042D8BA Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000408933 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000875D94 fds_IP_mesh_excha 2633 main.f90

fds6_From_Live_Cl 00000000007F8F50 MAIN__ 300 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007F05CB879B15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

forrtl: error (78): process killed (SIGTERM)

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000071566E Unknown Unknown Unknown

libpthread.so.0 00007F9CB6B9A100 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000430F7E Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000409AE6 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007FD66B fds_IP_timeout_ 2651 main.f90

fds6_From_Live_Cl 0000000000877A34 fds_IP_mesh_excha 2407 main.f90

fds6_From_Live_Cl 00000000007F8F50 MAIN__ 300 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007F9CB67EAB15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

forrtl: error (78): process killed (SIGTERM)

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000071566E Unknown Unknown Unknown

libpthread.so.0 00007FA596CBB100 Unknown Unknown Unknown

libpthread.so.0 00007FA596CB9FEA Unknown Unknown Unknown

libpthread.so.0 00007FA596CB6C53 Unknown Unknown Unknown

libpthread.so.0 00007FA596CB6BDE Unknown Unknown Unknown

libiomp5.so 00007FA597D9EC1B Unknown Unknown Unknown

libiomp5.so 00007FA597D3C403 Unknown Unknown Unknown

libiomp5.so 00007FA597D3DD59 Unknown Unknown Unknown

libiomp5.so 00007FA597D69CCF Unknown Unknown Unknown

libiomp5.so 00007FA597D3E6C8 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000B444D8 radcompute_radiat 1108 radi.f90

fds6_From_Live_Cl 0000000000B30474 rad_mp_compute_ra 602 radi.f90

fds6_From_Live_Cl 00000000007FC17E MAIN__ 297 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007FA59690BB15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

forrtl: error (78): process killed (SIGTERM)

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000071566E Unknown Unknown Unknown

libpthread.so.0 00007F911E982100 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000B472BB radcompute_radiat 1255 radi.f90

fds6_From_Live_Cl 0000000000B30474 rad_mp_compute_ra 602 radi.f90

fds6_From_Live_Cl 00000000007FC17E MAIN__ 297 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007F911E5D2B15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

forrtl: error (78): process killed (SIGTERM)

Image PC Routine Line Source

fds6_From_Live_Cl 00000000007E4D45 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000007E2B07 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000794044 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000793E56 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000070E829 Unknown Unknown Unknown

fds6_From_Live_Cl 000000000071566E Unknown Unknown Unknown

libpthread.so.0 00007FD5A5CE0100 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000B3418B radcompute_radiat 969 radi.f90

libiomp5.so 00007FD5A6DBBBC3 Unknown Unknown Unknown

libiomp5.so 00007FD5A6D8D8C7 Unknown Unknown Unknown

libiomp5.so 00007FD5A6D8ED0C Unknown Unknown Unknown

libiomp5.so 00007FD5A6D636C8 Unknown Unknown Unknown

fds6_From_Live_Cl 0000000000B427B0 radcompute_radiat 944 radi.f90

fds6_From_Live_Cl 0000000000B30474 rad_mp_compute_ra 602 radi.f90

fds6_From_Live_Cl 00000000007FC17E MAIN__ 297 main.f90

fds6_From_Live_Cl 00000000004086CE Unknown Unknown Unknown

libc.so.6 00007FD5A5930B15 Unknown Unknown Unknown

fds6_From_Live_Cl 00000000004085E9 Unknown Unknown Unknown

I'm endeavouring to help a customer migrate from FDS v6.1.0 in use on a Linux Cluster running an earlier version of Linux, to a newer Cluster and Linux deployment, and these are preliminary test results.

I'm not sure what the sigsegv errors above mean, nor where to look.

In addition, if I grab the very latest versions of OpenMPI v1.10 for CentOS 7 and FDS v6.5.1 source which I compile using GNU Linux compilers, I get nowhere at all when running my fds file.

Any help or guidance would be appreciated.

Michael

Kevin

unread,

Aug 12, 2016, 8:25:01 AM8/12/16

to FDS and Smokeview Discussions

The latest version of FDS is 6.5.1. It is designed to run with OpenMPI 1.8.4. Have you tried this combination? If so, and it still fails, open up an issue on the Issue Tracker and include the simplest form of the input file that does not execute properly.

Sam J

unread,

Aug 29, 2016, 11:21:34 PM8/29/16

to FDS and Smokeview Discussions

I remember seeing this error while trying to get some large runs started on my Ubuntu cluster. Smaller runs started without any problem. In one instance, one of my nodes did not have sufficient RAM. The system monitor would show the ram utilization rising and in the absence of swap/virtual memory, the run would abort as soon as the RAM utilization hit 100%. If you have enough RAM, then look into stacksize. Changing the stack size to "unlimited" fixed this error for me when insufficient RAM was not a problem.

I followed the second approach on this website (editing /etc/bash.bashrc):

http://disbauxes.upc.es/code/applied-physics-department/changing-the-stacksize-ulimit-s-limit-whenever-running-mpirun/

Phillip Spellman

unread,

Sep 23, 2016, 12:57:29 AM9/23/16

to FDS and Smokeview Discussions

For the future googlers,

I had similar issues when running very large meshes and Sam Js response helped. I thought I had set the stack limit etc before, but I must have changed something over time and not realised.

I added the following to /etc/bash.bashrc for the relevant user on each node:

ulimit -s unlimited 
ulimit -v unlimited