Compiling and running turbChannel : "AMG files not found, dumping data ..."

314 views
Skip to first unread message

Nadeem Malik

unread,
Jan 28, 2020, 5:35:03 PM1/28/20
to Nek5000

Hi Neks,

I am starting work on turbulent channel flow, with the aim of investigating turbulence generated by different types of physical grids (regular, fractal bars, etc). Note this is not the numerical grid that I refer to. 

For now I was trying to run the base case turbChannel with the default settings, (this is an LES code for turbulent channel flow): 


(1)  The genbox and genmap are  fine, but a warning came up in the compiling:


quanah:/Nek5000/trash/turbChannel$ makenek
using source path: /home/nadmalik/Nek5000
setup build environment
generating makefile ... done
building 3rd-party dependencies ... done
‘turbChannel.usr’ -> ‘turbChannel.f’
mpif77  -O2 -cpp -fdefault-real-8 -fdefault-double-8 -std=legacy  -DMPI -DUNDERSCORE -DGLOBAL_LONG_LONG -DTIMER -I/home/nadmalik/Nek5000/trash/turbChannel -I/home/nadmalik/Nek5000/core -I./ -I /home/nadmalik/Nek5000/core/experimental  -c /home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f
/home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f:142:47:

         if (ldimt.ge.2) call lambda2(t(1,1,1,1,2))
                                               1
Warning: Array reference at (1) is out of bounds (2 > 1) in dimension 5
/home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f:143:50:

         if (ldimt.ge.3) call comp_vort3(t(1,1,1,1,3),wo1,wo2,vx,vy,vz)
                                                  1
Warning: Array reference at (1) is out of bounds (3 > 1) in dimension 5
building libnek5000.a ... done
mpif77  -O2 -cpp -fdefault-real-8 -fdefault-double-8 -std=legacy  -DMPI -DUNDERSCORE -DGLOBAL_LONG_LONG -DTIMER -I/home/nadmalik/Nek5000/trash/turbChannel -I/home/nadmalik/Nek5000/core -I./ -I /home/nadmalik/Nek5000/core/experimental -o nek5000 /home/nadmalik/Nek5000/core/drive.f /home/nadmalik/Nek5000/trash/turbChannel/turbChannel.o libnek5000.a -L/home/nadmalik/Nek5000/3rd_party/blasLapack -lblasLapack -L/home/nadmalik/Nek5000/3rd_party/gslib/lib -lgs -Wl,--allow-multiple-definition


#############################################################
#                  Compilation successful!                  #
#############################################################
   text    data     bss     dec     hex filename
1714109   16732 138484584       140215425       85b8481 nek5000



(2) The code nevertheless appears to have started to run, but then stopped with the following:

....
gs_setup: 144296 unique labels shared
   pairwise times (avg, min, max): 0.000135484 0.000128399 0.000142712
   crystal router                : 0.000222133 0.000218973 0.000224215
   used all_to_all method: pairwise
   handle bytes (avg, min, max): 362912 333916 393052
   buffer bytes (avg, min, max): 151636 128032 181920
   setupds time 6.3341E-02 seconds  10 10      343680        1536
 setup h1 coarse grid, nx_crs=           2
 call usrsetvert
 done :: usrsetvert

gs_setup: 1352 unique labels shared
   pairwise times (avg, min, max): 5.12267e-05 4.91719e-05 5.31856e-05
   crystal router                : 1.72507e-05 1.68691e-05 1.76493e-05
   all reduce                    : 0.000109527 0.000109063 0.000109887
   used all_to_all method: crystal router
   handle bytes (avg, min, max): 29647.8 25516 36932
   buffer bytes (avg, min, max): 4548.44 3792 5984
AMG files not found, dumping data ...
AMG writing data from proc 0
AMG writing data from proc 1
AMG writing data from proc 2
....
....
AMG dump successful
--------------------------------------------------------------------------
mpiexec has exited due to process rank 1 with PID 104527 on
node compute-2-25 exiting improperly. There are three reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

3. this process called "MPI_Abort" or "orte_abort" and the mca parameter
orte_create_session_dirs is set to false. In this case, the run-time cannot
detect that the abort call was an abnormal termination. Hence, the only
error message you will receive is this one.

This may have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).

You can avoid this message by specifying -quiet on the mpiexec command line.

--------------------------------------------------------------------------


Is there a quick fix to this? Maybe the problem is too small -- on the TTU HPC, I was running on a single node with 36 cores. This gives just 43 elements per rank ?

Thanks
Nadeem



Tanmoy Chatterjee

unread,
Jan 28, 2020, 5:54:13 PM1/28/20
to Nadeem Malik, Nek5000
Once you run, this , run amg_setup or amg_hypre and generate the necessary amg files. Then rerun the case. This should work.

Best Regards,
Tanmoy


--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/31963b53-3c13-447a-9660-8410aac9fbe4%40googlegroups.com.

nadeem...@cantab.net

unread,
Jan 28, 2020, 6:17:00 PM1/28/20
to Tanmoy Chatterjee, Nadeem Malik, Nek5000

When I do “./amg_setup” or “./amg_hypre” it says command not found?

 

Incidentally, after the attempted run, this is the list of my files in the directory.

 

-rw-r--r-- 1 nadmalik ME 1016072 Jan 28 16:47 amgdmp_i.dat

-rw-r--r-- 1 nadmalik ME 1016072 Jan 28 16:47 amgdmp_j.dat

-rw-r--r-- 1 nadmalik ME 1016072 Jan 28 16:47 amgdmp_p.dat

-rw-r--r-- 1 nadmalik ME  114012 Jan 28 16:45 build.log

-rw-r--r-- 1 nadmalik ME    1764 Jan 28 16:47 machinefile.1081778

-rw-r--r-- 1 nadmalik ME   11528 Jan 28 16:44 makefile

-rwxr-xr-x 1 nadmalik ME 1838384 Jan 28 16:45 nek5000

-rw-r--r-- 1 nadmalik ME       0 Jan 28 16:47 Nek5000.e1081778

-rw-r--r-- 1 nadmalik ME   16099 Jan 28 16:47 Nek5000.o1081778

drwxr-xr-x 2 nadmalik ME    4096 Jan 28 16:45 obj

-rw-r--r-- 1 nadmalik ME     463 Jan 28 16:09 README.md

drwxr-xr-x 2 nadmalik ME    4096 Jan 28 16:09 ref

-rw-r--r-- 1 nadmalik ME     414 Jan 28 16:43 run.sh

-rw-r--r-- 1 nadmalik ME      58 Jan 28 16:47 SESSION.NAME

-rw-r--r-- 1 nadmalik ME    2214 Jan 28 16:42 SIZE

-rw-r--r-- 1 nadmalik ME  465784 Jan 28 16:09 turbChannel.amg_Aff.dat

-rw-r--r-- 1 nadmalik ME  652280 Jan 28 16:09 turbChannel.amg_AfP.dat

-rw-r--r-- 1 nadmalik ME   80016 Jan 28 16:09 turbChannel.amg.dat

-rw-r--r-- 1 nadmalik ME   54584 Jan 28 16:09 turbChannel.amg_W.dat

-rw-r--r-- 1 nadmalik ME     521 Jan 28 16:41 turbChannel.box

-rw-r--r-- 1 nadmalik ME   11091 Jan 28 16:45 turbChannel.f

-rw-r--r-- 1 nadmalik ME  166024 Jan 28 16:44 turbChannel.ma2

-rw-r--r-- 1 nadmalik ME     733 Jan 28 16:09 turbChannel.par

-rw-r--r-- 1 nadmalik ME 1032292 Jan 28 16:43 turbChannel.re2

-rw-r--r-- 1 nadmalik ME   10828 Jan 28 16:09 turbChannel.usr

 

 

-nadeem

Tanmoy Chatterjee

unread,
Jan 28, 2020, 6:35:13 PM1/28/20
to nadeem...@cantab.net, Nadeem Malik, Nek5000
did you do build amg_setup /amg_hypre (depending on which version you use)? You must use maketools amg_setup or maketools amg_hypre and build the tool, first. 

Best Regards,
Tanmoy


To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/002d01d5d631%2408a99250%2419fcb6f0%24%40cantab.net.

nadeem...@cantab.net

unread,
Jan 28, 2020, 6:49:54 PM1/28/20
to Tanmoy Chatterjee, Nadeem Malik, Nek5000

I must be missing something, because in ~/Nek5000/tools when I do “./maketools amg_setup” it says,

 

quanah:/Nek5000/tools$ ./maketools amg_setup

building amg_setup ... failed

 

FATAL ERROR occured during compilation!

Check build.log for details

 

 

build.log is in my turbChannel directory, and simply contains the compilation messages:

 

ar -cr libblasLapack.a *.o

‘turbChannel.usr’ -> ‘turbChannel.f’

mpif77  -O2 -cpp -fdefault-real-8 -fdefault-double-8 -std=legacy  -DMPI -DUNDERSCORE -DGLOBAL_LONG_LONG -DTIMER -I/home/nadmalik/Nek5000/trash/turbChannel -I/home/nadmalik/Nek5000/core -I./ -I /home/nadmalik/Nek5000/core/experimental  -c /home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f

/home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f:142:47:

 

         if (ldimt.ge.2) call lambda2(t(1,1,1,1,2))

                                               1

Warning: Array reference at (1) is out of bounds (2 > 1) in dimension 5

/home/nadmalik/Nek5000/trash/turbChannel/turbChannel.f:143:50:

 

         if (ldimt.ge.3) call comp_vort3(t(1,1,1,1,3),wo1,wo2,vx,vy,vz)

                                                  1

Warning: Array reference at (1) is out of bounds (3 > 1) in dimension 5

building libnek5000.a ... done

mpif77  -O2 -cpp -fdefault-real-8 -fdefault-double-8 -std=legacy  -DMPI -DUNDERSCORE -DGLOBAL_LONG_LONG -DTIMER -I/home/nadmalik/Nek5000/trash/turbChannel -I/home/nadmalik/Nek5000/core -I./ -I /home/nadmalik/Nek5000/core/experimental -o nek5000 /home/nadmalik/Nek5000/core/drive.f /home/nadmalik/Nek5000/trash/turbChannel/turbChannel.o libnek5000.a -L/home/nadmalik/Nek5000/3rd_party/blasLapack -lblasLapack -L/home/nadmalik/Nek5000/3rd_party/gslib/lib -lgs -Wl,--allow-multiple-definition

 

 

#############################################################

#                  Compilation successful!                  #

#############################################################

   text    data     bss     dec     hex filename

1714109   16732 138484584       140215425       85b8481 nek5000

 

What is missing ?

 

Thanks

Nadeem

nadeem...@cantab.net

unread,
Jan 28, 2020, 6:53:20 PM1/28/20
to Tanmoy Chatterjee, Nadeem Malik, Nek5000

p.s. in the tools directory, I see the following sub-directories, including “nekamg_setup”:

 

drwxr-xr-x 4 nadmalik ME 4096 Apr 24  2019 cgns2nek

drwxr-xr-x 3 nadmalik ME 4096 Apr 24  2019 exo2nek

drwxr-xr-x 2 nadmalik ME 4096 Jan 28 16:52 genbox

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 gencon

drwxr-xr-x 2 nadmalik ME 4096 Jan 28 16:52 genmap

-rwxr-xr-x 1 nadmalik ME  254 Jan 20 10:27 maketools

-rw-r--r-- 1 nadmalik ME 4723 Apr 24  2019 maketools.inc

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 n2to3

drwxr-xr-x 3 nadmalik ME 4096 Apr 24  2019 nekamg_setup

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 nekmerge

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 postnek

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 prenek

drwxr-xr-x 2 nadmalik ME 4096 Apr 24  2019 reatore2

 

and nekamg_setup contains:

 

-rw-r--r-- 1 nadmalik ME 36590 Apr 24  2019 amg_setup.c

-rw-r--r-- 1 nadmalik ME  8784 Apr 24  2019 amg_setup.h

-rw-r--r-- 1 nadmalik ME 97029 Jan 28 17:51 build.log

drwxr-xr-x 3 nadmalik ME  4096 Jan 28 17:51 hypre

-rw-r--r-- 1 nadmalik ME   431 Apr 24  2019 makefile

 

Thanks

Nadeem

 

 

From: nek...@googlegroups.com <nek...@googlegroups.com> On Behalf Of Tanmoy Chatterjee

Sent: Tuesday, January 28, 2020 5:35 PM
To: nadeem...@cantab.net

Tanmoy Chatterjee

unread,
Jan 28, 2020, 7:10:31 PM1/28/20
to nadeem...@cantab.net, Nadeem Malik, Nek5000
Can you try building nekamg_setup using maketools nekamg_setup.

Best Regards,
Tanmoy


To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/005c01d5d636%241b1dbf10%2451593d30%24%40cantab.net.

nadeem...@cantab.net

unread,
Jan 28, 2020, 7:26:40 PM1/28/20
to Tanmoy Chatterjee, Nek5000

I have tried on the TTU hpc, and the same result.

 

I have also tried on the TACC stampede2, and the maketool at least works; but here I am getting errors in compiling all of a sudden? I maybe a older version, so let me try a few things, then I’ll get back you.

Nadeem Malik

unread,
Jan 29, 2020, 8:15:59 PM1/29/20
to Nek5000

Ok, it runs and works on TACC Stampede2. So I can move to the next stages.

However, on the TTU hpc, when I do "./maketool genbox genmap nekamg_setup", then genbox and genmap are generated ok,  but for nekamg_setup it finds a fault and says to look in build.log ?

Can you tell me the difference between amg_setup, nekamg_setup, and hypre versions of these (how do I generate those) ? 

Thanks
Nadeem


To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

Byron Guerrero

unread,
Jan 29, 2020, 8:19:44 PM1/29/20
to Nek5000
Hi Nadeem,

I also had some issues with the AMG setup. Probably what we discussed in the following link might help.


Best,
Byron

YuHsiang Lan

unread,
Jan 29, 2020, 8:28:33 PM1/29/20
to Nek5000
Hi Nadeem,

So, what error do you get from build.log?
(it should be under the folder Nek5000/tools/nekamg_setup).

Yu-Hsiang
--

nadeem...@cantab.net

unread,
Jan 29, 2020, 8:33:17 PM1/29/20
to YuHsiang Lan, Nek5000

I get:

 

building nekamg_setup ... failed

 

FATAL ERROR occured during compilation!

Check build.log for details

 

 

Same if for amg_setup

 

-nadeem

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/76004b60-3ec5-47f6-8e8e-0776ed49dd9f%40googlegroups.com.

YuHsiang Lan

unread,
Jan 29, 2020, 8:34:58 PM1/29/20
to Nek5000
Right. and what's inside the build.log?
I saw your build.log file exists in your previous reply. It's under Nek5000/tools/nekamg_setup.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+unsubscribe@googlegroups.com.

Nadeem Malik

unread,
Jan 29, 2020, 8:51:31 PM1/29/20
to Nek5000
I get:

building nekamg_setup ... failed

FATAL ERROR occured during compilation!
Check build.log for details
quanah:/Nek5000/tools$ ./maketools amg_setup

and

building amg_setup ... failed

FATAL ERROR occured during compilation!
Check build.log for details
quanah:/Nek5000/tools$


-nadeem

Tanmoy Chatterjee

unread,
Jan 29, 2020, 9:23:54 PM1/29/20
to Nadeem Malik, Nek5000
build.log is a file. What logs due you see inside build.log file?
Thats the question.

Best Regards,
Tanmoy


To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/ff20eda9-fb75-486b-aaba-4ed83f278c32%40googlegroups.com.

Nadeem Malik

unread,
Jan 29, 2020, 9:43:39 PM1/29/20
to Nek5000

In Nek5000/tools/nekamg_setup, I have :

-rw-r--r-- 1 nadmalik ME 36590 Apr 24  2019 amg_setup.c
-rw-r--r-- 1 nadmalik ME  8784 Apr 24  2019 amg_setup.h
-rw-r--r-- 1 nadmalik ME 86333 Jan 29 19:31 build.log
drwxr-xr-x 3 nadmalik ME  4096 Jan 29 19:31 hypre
-rw-r--r-- 1 nadmalik ME   431 Apr 24  2019 makefile

and build.log is:

make: Entering directory `/home/nadmalik/Nek5000/tools/nekamg_setup'
hypre-2.15.1/AUTOTEST/
hypre-2.15.1/AUTOTEST/README.txt
hypre-2.15.1/AUTOTEST/basic.sh
hypre-2.15.1/AUTOTEST/check-double.filters
hypre-2.15.1/AUTOTEST/check-double.sh
hypre-2.15.1/AUTOTEST/check-glob-symb.filters
...
...
hypre-2.15.1/src/utilities/timing.h
hypre-2.15.1/src/utilities/version
++ pwd
++ which /opt/ohpc/pub/compiler/gcc/7.3.0/bin/gcc
+ cmake -DHYPRE_BIGINT=OFF -DHYPRE_INSTALL_PREFIX=/home/nadmalik/Nek5000/tools/nekamg_setup/hypre/hypre/build/../.. -DBUILD_SHARED_LIBS=OFF -DHYPRE_SEQUENTIAL=ON -DMPI_C_COMPILER=/opt/ohpc/pub/compiler/gcc/7.3.0/bin/gcc -DHYPRE_USING_FEI=OFF ../src
./install: line 36: cmake: command not found
make: *** [lib] Error 127
make: Leaving directory `/home/nadmalik/Nek5000/tools/nekamg_setup'

-nadeem


On Wednesday, January 29, 2020 at 7:34:58 PM UTC-6, YuHsiang Lan wrote:

YuHsiang Lan

unread,
Jan 29, 2020, 11:25:21 PM1/29/20
to Nek5000
As the logfile says, you need to install cmake to compile hypre.

Yu-Hsiang
--
Reply all
Reply to author
Forward
0 new messages