AMG_SETUP Nek5000 V19

[Byron Guerrero]

Hello Neks,

I'm trying to run a large simulation on Nek. Then, I've tried to use the AMG pressure solver. However, I have not yet succeeded in that.

My steps have been the following.

1.   on /Nek5000/tools, I compiled the amg_setup by using ./maketools amg_setup

2.   Then, I changed the makenek file and added the following two lines of code

      PPLIST="MPIIO"
      AMG=true
      AMG_DUMP=true

3.   Compiled my <case> files using makenek <case>

4.   Ran amg_setup on my <case> directory, and I was asked the following:

Choose a coarsening method. Available options are:
 - 3: Ruege-Stuben,
 - 6: Falgout,
 - 10: HMIS (default),
Choice: 10
Choose an interpolation method. Available options are:
 - 0: classical modified interpolation,
 - 6: extended + i interpolation (default),
Choice: 6
Enter smoother tolerance [0.5]:
0.2
Reading AMG dump files...
ERROR: File amgdmp_i.dat not found!

What do I need to do te get the amgdmp_i.dat files?

Thank you in advance for your help.

Best,

Byron

[YuHsiang Lan]

Hi Byron,

In the latest repo version of Nek5000, we have two AMG options:

1. dump AMG files, solve files with tools and save it into new files, then read the dat files in the production run.

2. On-the-fly AMG. using HYPRE BOOMER_AMG

option1:

Set param(40)=1 (or solver=sem_amg in par file), run Nek5000 with lx1=3 (just saving your allocation for the real job), then it will dump dat files for the first run.

Use amg_setup tools. Be carefull about your mesh size. You might have to re-compile the tools with larger E, set in ./maketool script. This is your 4th step

You will find that the tool will generate a new series of files.

recompile and run Nek5000 with lx1=your_real_lx1, again param(40)=1.

note:

- Basically, Nek5000 will dump files if the desired files are not presented.

- The tools has limitation. It's serial, so it takes ages to run large E. The max I ran is about E30M and I failed to do so for E120M.

- The latest version of Nek5000 re-name dat files with the case name. This part is new. If you want tot use old Nek5000's tool (they are slightly different in terms of options). You have to re-name them yourself.

- modifying PPLIST requires a full clean: ./makenek clean

Option2:

Put AMG, HYPRE into your PPLIST. and make sure you have cmake in your environment.

Set param(40)=2 (or solver=sem_amg_hypre in par file), You should be able to run Nek5000 directly.

note:

- HYPRE will act differently using different numbers of processes. In this sense, dat files are more reliable.

This configuration is different than the one for v17, but the general idea is the same.

Hope this helps.

Yu-Hsiang

--

[Byron Guerrero]

 Hi again Neks,

I have done what Yu-Hsiang suggested, and it works. Followed those steps using the HPC facility at my university. However, to be able to run the simulations I'm now using a different supercomputer. Since the bigger HPC facility I'm using has different MPI compilers, I was wondering if it's required to dump again the amg files?

In the new supercomputer, I have compilation issues and also problems when trying to run that particular simulation.

Thanks in advance for your support.

Best,

Byron

[Tanmoy Chatterjee]

Hi Byron,

I had a similar situation like yours. You dont need to dump the amg files again, once you have it and your geometry does not change.

Best Regards,

Tanmoy

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nek5000/34ec1f56-5e97-47ca-9c7f-dbe24d0d7b58%40googlegroups.com.

[Sina Tahmooresi]

Hello Yu-Hsiang,

I was following the threads on the AMG pressure solver setup for the case turbCahnnel. I want to learn the workflow of how the existing files turbChannel.amg_Aff.dat, turbChannel.amg_AfP.dat, turbChannel.amg.dat, turbChannel.amg_W.dat are created. 

I am using version 19.0 and thought option (1) explained in your response is quite straightforward to follow. I complied the case and then ran it. It fails, while theses files were dumped:
turbChannel.amgdmp_i.dat, turbChannel.amgdmp_j.dat, turbChannel.amgdmp_p.dat. they are different than the expected original ones and in the log file I read:
AMG files not found, dumping data ...
AMG writing data from proc 0
AMG writing data from proc 1
AMG writing data from proc 2
AMG writing data from proc 3
AMG writing data from proc 4
AMG writing data from proc 5
AMG writing data from proc 6
AMG writing data from proc 7
AMG dump successful

an error occured: dying ...

in your explanation, you mention " ... then it will dump dat files for the first run. Use amg_setup tools.." What do you mean by " Use amg_setup tools" here? 
I should add, when I tried the ./maketools makeamg_setup in tools directory, it fails :
building amg_setup ... failed with the buil.log reads :
make: Entering directory '/home/stahm017/.local/easybuild/software/2020/Core/nek5000/19.0/tools/nekamg_setup'
/home/stahm017/.local/easybuild/software/2020/Core/nek5000/19.0/tools/nekamg_setup/hypre

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
make: *** [makefile:13: lib] Error 2
make: Leaving directory '/home/stahm017/.local/easybuild/software/2020/Core/nek5000/19.0/tools/nekamg_setup'

My understanding is that the problem is related to the hypre (that's why I decided to use the option (1)), or I am mistaken and the failure is for the whole makeamg_setup.

In tool/ nekamg_setup, I can see that these files exist:
amg_setup.c  amg_setup.h  build.log  hypre  makefile, while the hypre contains:
hypre  install  v2.15.1.tar.gz, and the latest hypre directory is empty.


Can I have your advises on the current situation?

Thank you,
Sina

[YuHsiang Lan]

Hi Sina,

That usually happen when downloading the Nek5000 via released tar,gz file and sometime the file is corrupted during download.

See the attempt of the solution at here:

   https://groups.google.com/g/nek5000/c/082gb4qK9tw/m/Q4M3gObVCwAJ

If it still fails, get Nek5000 via git and checkout v23 should work.

(See also the discussion above that the path to HYPRE might need to change.)

> What do you mean by " Use amg_setup tools" here?

The tool was renamed from amg_setup to nekamg_setup.

> that's why I decided to use the option (1)

Both option use HYPRE. One calls it via nekamg_setup tool. The other calls it at runtime with MPI.

Hope this helps,

Yu-Hsiang

--