Simulation crashes after partitioning elements to MPI ranks

[Nils Benedikt]

Hi Neks,

I am trying to run a simulation with 247968 Elements on 512 nodes (24576 cores, with 48 cores per node). 
However, the highest number of nodes I could get the simulation to run on is 416 (19968 cores), with 448 nodes (21504 cores) it crashes already. I attached the logfile, the first 36 lines of the errorfile, and my SIZE file since I expect it this might have something to do with the problem. After using the Nek5000 chatbot I tried the following things, none of which had any effect on the problem:

1. Executing genmap with lower tolerance. I went down to 0.001 and the problem still appeared.

2. I set lelt=lelg/lpmin+3 instead of lelt=lelg/lpmin+1 in the SIZE file to make sure that I don't exceed lelt. 

3. I tried compiling nek5000 with additional compiler flags by adding the line 

export FCFLAGS="-g -O0 -fcheck=all -fbacktrace"

to my compile_script (I'm using Nek5000 with KTH Framework). I wanted to at least get some more info about the error but it didn't change anything.

4. I saw that sometimes the pressure preconditioner (semg_amg_hypre) causes such problems so I tried without it. It still crashed.

Does anyone know what causes this error or does anyone have any ideas what might be going on? I would be very happy about any help since I'm out of ideas.

Best regards

Nils 

[Philipp Schlatter]

Hi,

the output seems to indicate that there is some issue with file IO where the issues happen. Are you reading an input file? Could you perhpas try to use a different reader (not sure that works any longer to not use MPI-IO). Didnt you get a better output when compiling with -g?

In the size file, perhaps just add a few more elements to lelt (+10 or so, should not matter).

On the other hand, calculating the number of elements per rank, I get about 11, which I believe is way too few elements per rank. Do you really need to run with so many cores for that case?

You can also experiment a bit with the order of course, in an effort to understand better what could happen. 

Best regards,
Philipp

--
You received this message because you are subscribed to the Google Groups "Nek5000" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nek5000+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/nek5000/7aa88da9-a430-4fc6-a284-2126f12f5183n%40googlegroups.com.

[Nils Benedikt]

Hi,

1. I think that the code crashes while it is reading something from the .re2 file. I don't know (yet) how to use a different reader for that but I can try.

2. I initally made a mistake when compiling with -g, so the compiler didn't actually get the flag. Now I did and the output in the errorfile is a little more verbose but it only shows two more columns "Line" and "Source" which are always "unknown".

3. I added +10 in the SIZE file, it didn't make a difference. 

4. The issue is that we will need to run the simulation for a very long time (for all statistics to converge) with comparatively small timesteps. So every reduction in the amount of runs I have to do is good. I thought 10 elements per rank is the recommended minimum, at least I read that somewhere. In the runs with 19968 cores, I have 12/13 elements per rank and the speed up compared to the test runs with 768 cores is pretty much linear. So I thought it shouldn't be a problem to go a little higher with the number of cores.

5. Which order do you mean that I can experiment with?

Best regards

Nils

[YuHsiang Lan]

Hi Nils,

I didn't find the actual error from logfiles you sent.

Given that you got "source is unknown", it's likely the location is in 3rd party library, which could be something in gslib to even an MPI internal error.

I'd still suggest trying to get the top level nek5000 subroutines to narrow down the location.

Here are some suggestions to move forward.

- Put "-g" to both CFLAGS and FFLAGS in makenek, if lucky, gslib will print the location.

- Set "ulimit -s unlimited" in case the code tries to dump the core file. If there is a core file, you can play with it in gdb:

  gdb ./nek5000 core

- Run with debugger like gdb. For example:

  mpiexec -n 19968 gdb -ex "run" -ex "bt" --args ./nek5000

. This will force all ranks print messages, which can be messy, but hopefully you can mine the error trace from it.

- Try the latest Nek5000, if I remember correctly, the partitioning and re2 reading has been refactored. So, less possible to have a IO issue.

- If no luck, I'd suggest starting to put prints statements and narrow down the location.

 

  call nekgsync()                  ! make sure all ranks reach here

  if (nio.eq.0) write(*,*)'dbg a'  ! grep "dbg" logfile later

  call flush(6)                    ! make sure IO is flushed to logfile

   Start from here: https://github.com/Nek5000/Nek5000/blob/5afb6179daf69c5f1d0e5a6eab49d6f07e0c622f/core/drive1.f#L74

   It should be near of inside here: https://github.com/Nek5000/Nek5000/blob/5afb6179daf69c5f1d0e5a6eab49d6f07e0c622f/core/connect2.f#L66C14-L66C27

Hope this helps,

Yu-Hsiang

--

[Nils Benedikt]

Hi Yu-Hsiang,

Thank you for your extensive answer, I will go through the steps one by one and see if I find an answer.

Best 

Nils

[Nils Benedikt]

Hi Yu-Hsiang,

Adding -g, trying to dump the core file and running in debug mode all didn't help. 

With the print statements however, I was able to narrow it down to this line:  Nek5000/core/byte_mpi.f at 5afb6179daf69c5f1d0e5a6eab49d6f07e0c622f · Nek5000/Nek5000 · GitHub

If I read it correctly, for some reason the .re2 file cannot be opened. I'm out of my depth here, so I would be very thankful for any hints of what this means or how I can fix it.

I'm not sure if moving to a newer Nek version is a good idea, since my statistics computation relies on the KTH Framework. I don't know if this would still work with a newer Nek version (maybe Philipp knows more about this) and if it won't it will cost me too much time to implement something else.  

Best

Nils

[Philipp Schlatter]

can't you just run without MPI_IO, just to test? Not sure this still works, but that would be an option.

Regarding KTH Framework, it should work with later versions, I think. If you are at APS in Houston this week, let me know, then we could chat personally (or anybody else from the Nek team :-) ).

best, Philipp

To view this discussion visit https://groups.google.com/d/msgid/nek5000/e8f0559a-07f5-4744-932b-eb5cc3378d45n%40googlegroups.com.

[Nils Benedikt]

Hi Philipp, 

Thanks for the suggestion. I'm not sure what running without MPI_IO means, but I will find out, try, and see what happens.

I won't be at APS, but I'll see if KTH Framework still works with a newer Nek Version.

Best

Nils

[Nils Benedikt]

Hi again, 

When I compile with the NOMPIIO flag, the original issue is resolved, but now it crashes while reading the checkpoint/restart file (rs6 file, from the KTH Framework checkpointing). I attached the logfile, it crashes at a point where it doesn't make sense to me: KTH_Toolbox/driver/frame/mntr/mntrlog.f at 61598341d09f98fbbabbecbca44238e0d6ad0e03 · KTH-Nek5000/KTH_Toolbox · GitHub (line 634 in Git is Line 611 in my version)

@Yu-Hsiang: you said with a newer Nek version it might work, which one exactly should I use? I see that the last release was v19 which is older than the one I am using. Is the one that is currently in the Git master branch stable? Because it said some build failed. 

Best 

Nils

[Nils Benedikt]

Hi everyone, 

Just in case anyone stumbles upon this thread in the future wondering how to resolve this issue. It works, if I run 

mpiexec --mca io romio321 -n 24576 ./nek5000

Apparently ROMIO is some older MPI backend that's more robust than the modern OMPIO but that's just what ChatGPT said.

Best

Nils