Step-69 : Parallel execution on 4 nodes

[Nicholas Yue]

Hi

I have build deal.ii on the Nvidia Jetson Nano cluster from Picocluster

It has 5 nodes (pc[0-4], I am using pc0 as the head/login node to launch the application

The executable is in an NFS filesystem to share between all the nodes

The following works

mpirun --host pc1 --mca btl_tcp_if_include 192.168.0.0/24 --mca btl tcp,self /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release

However, when attempting to run it on more than one hosts fails

mpirun --host pc1,pc2 --mca btl_tcp_if_include 192.168.0.0/24 --mca btl tcp,self /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release

It seems to be consistently failing when writing the checkpoint file(s)

Are there special flags I need to setup up for some form of parallel IO that may be happening ?

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    ####################################################

    #########                                  #########

    #########      Cycle  000040  (0.5%)       #########

    #########      at time t = 0.01975410      #########

    #########                                  #########

    ####################################################

    ####################################################

    #########                                  #########

    #########      checkpoint computation      #########

    #########                                  #########

    #########                                  #########

    ####################################################

[pc1:07535] mca_sharedfp_individual_file_open: Error during datafile file open

[pc2:07602] mca_sharedfp_individual_file_open: Error during datafile file open

----------------------------------------------------

Exception on processing: 

---------------------------------------------------------

TimerOutput objects finalize timed values printed to the

screen by communicating over MPI in their destructors.

Since an exception is currently uncaught, this

synchronization (and subsequent output) will be skipped

to avoid a possible deadlock.

---------------------------------------------------------

----------------------------------------------------

Exception on processing: 

--------------------------------------------------------

An error occurred in line <1412> of file </home/picocluster/projects/dealii/dealii/dealii_git/source/distributed/tria_base.cc> in function

    void dealii::parallel::DistributedTriangulationBase<dim, spacedim>::DataTransfer::save(unsigned int, unsigned int, const string&) const [with int dim = 2; int spacedim = 2; std::__cxx11::string = std::__cxx11::basic_string<char>]

The violated condition was: 

    ierr == MPI_SUCCESS

Additional information: 

    deal.II encountered an error while calling an MPI function.

    The description of the error provided by MPI is "MPI_ERR_FILE: invalid

    file".

    The numerical value of the original error code is 30.

Stacktrace:

-----------

#0  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::DistributedTriangulationBase<2, 2>::DataTransfer::save(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#1  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::DistributedTriangulationBase<2, 2>::save_attached_data(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#2  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::save(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#3  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: Step69::MainLoop<2>::checkpoint(std::array<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, 4ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double, unsigned int)

#4  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: Step69::MainLoop<2>::run()

#5  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: main

--------------------------------------------------------

Aborting!

----------------------------------------------------

--------------------------------------------------------

An error occurred in line <1412> of file </home/picocluster/projects/dealii/dealii/dealii_git/source/distributed/tria_base.cc> in function

    void dealii::parallel::DistributedTriangulationBase<dim, spacedim>::DataTransfer::save(unsigned int, unsigned int, const string&) const [with int dim = 2; int spacedim = 2; std::__cxx11::string = std::__cxx11::basic_string<char>]

The violated condition was: 

    ierr == MPI_SUCCESS

Additional information: 

    deal.II encountered an error while calling an MPI function.

    The description of the error provided by MPI is "MPI_ERR_FILE: invalid

    file".

    The numerical value of the original error code is 30.

Stacktrace:

-----------

#0  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::DistributedTriangulationBase<2, 2>::DataTransfer::save(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#1  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::DistributedTriangulationBase<2, 2>::save_attached_data(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#2  /nfs/systems/dealii/head-bost_1_70_0/lib/libdeal_II.so.10.0.0-pre: dealii::parallel::distributed::Triangulation<2, 2>::save(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const

#3  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: Step69::MainLoop<2>::checkpoint(std::array<dealii::LinearAlgebra::distributed::Vector<double, dealii::MemorySpace::Host>, 4ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double, unsigned int)

#4  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: Step69::MainLoop<2>::run()

#5  /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release: main

--------------------------------------------------------

Aborting!

----------------------------------------------------

-------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code.. Per user-direction, the job has been aborted.

-------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[8910,1],1]

  Exit code:    1

--------------------------------------------------------------------------

[Wolfgang Bangerth]

On 9/3/21 11:49 AM, Nicholas Yue wrote:
> An error occurred in line <1412> of file
> </home/picocluster/projects/dealii/dealii/dealii_git/source/distributed/tria_base.cc>
> in function
>     void dealii::parallel::DistributedTriangulationBase<dim,
> spacedim>::DataTransfer::save(unsigned int, unsigned int, const string&) const
> [with int dim = 2; int spacedim = 2; std::__cxx11::string =
> std::__cxx11::basic_string<char>]
> The violated condition was:
>     ierr == MPI_SUCCESS
> Additional information:
>     deal.II encountered an error while calling an MPI function.
>     The description of the error provided by MPI is "MPI_ERR_FILE: invalid
>     file".
>     The numerical value of the original error code is 30.

It's hard to tell what the reason may be, but let me take a guess: Are you
sure that your file system supports MPI I/O? For example, NFS-mounted file
systems do not support this, and it would lead to errors like this one.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Matthias Maier]

Hi Nicholas,

On Fri, Sep 3, 2021, at 12:49 CDT, Nicholas Yue <yue.ni...@gmail.com> wrote:

> Hi

>
> It seems to be consistently failing when writing the checkpoint file(s)
>
> Are there special flags I need to setup up for some form of parallel IO
> that may be happening ?

> [...]

> Additional information:
> deal.II encountered an error while calling an MPI function.
> The description of the error provided by MPI is "MPI_ERR_FILE: invalid
> file".
> The numerical value of the original error code is 30.

This is interesting. It seems that MPI IO is failing.

Do you write into a distributed file system that is replicated among nodes?

Would you mind testing running the code with checkpointing disabled,
something like:


diff --git a/examples/step-69/step-69.cc b/examples/step-69/step-69.cc
index 4a801f97ba..4b7c9a2f63 100644
--- a/examples/step-69/step-69.cc
+++ b/examples/step-69/step-69.cc
@@ -2595,7 +2595,7 @@ namespace Step69

if (t > output_cycle * output_granularity)
{
- checkpoint(U, base_name, t, output_cycle);
+ // checkpoint(U, base_name, t, output_cycle);
output(U, base_name, t, output_cycle);
++output_cycle;
}


I am interested in seeing whether the solution output (into vtu) works.


Best,
Matthias

[Nicholas Yue]

The file system I am mounting via NFS is an ordinary Linux file system, it is not a HPC parallel filesystems like Lustre or anything like them.

I tried commenting out the call to check-pointing as you suggested and was able to run the code on 4 node (each with 4 cores) and it finished very quickly.

My mpirun command line looks like this

mpirun --host pc1,pc2,pc3,pc4 --mca btl_tcp_if_include 192.168.0.0/24 --mca btl tcp,self /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release

It is unlikely that I will have the resource to spin up a Lustre like parallel filesystems, do you have additional suggestion that may allow me to enable check-pointing ?

Cheers

[Wolfgang Bangerth]

On 9/3/21 7:49 PM, Nicholas Yue wrote:
> The file system I am mounting via NFS is an ordinary Linux file system, it is
> not a HPC parallel filesystems like Lustre or anything like them.
>
> I tried commenting out the call to check-pointing as you suggested and was
> able to run the code on 4 node (each with 4 cores) and it finished very quickly.
>
> My mpirun command line looks like this
>
> mpirun --host pc1,pc2,pc3,pc4 --mca btl_tcp_if_include 192.168.0.0/24 --mca
> btl tcp,self /nfs/systems/dealii/head-bost_1_70_0/examples/step-69/step-69.release
>
> It is unlikely that I will have the resource to spin up a Lustre like parallel
> filesystems, do you have additional suggestion that may allow me to enable
> check-pointing ?

You don't need something like Lustre. There are a number of ways to make
things work with NFS, for example using this approach
https://docs.huihoo.com/mpich/mpichman-chp4/node60.htm
I bet you can also find other approaches if you search for "MPI I/O" and "NFS".

[Matthias Maier]

> I tried commenting out the call to check-pointing as you suggested and was
> able to run the code on 4 node (each with 4 cores) and it finished very
> quickly.

Did it write out vtu files during this run? (They are written using MPI-IO)

Best,
Matthias

[Nicholas Yue]

@Matthias yes, it wrote out all 200 *.pvtu and *.vtu file pair and they are non-empty.

Seems it is only the check pointing that is having problem via MPI-IO

FYI, I have now also created and mounted a different storage and followed the `noac` NFS setting and have been able to run the full simulation of step-69 with the check pointing enabled.

I am currently in the processing of adding a third variation of a storage solution via OrangeFS and compare the I/O performance between OrangeFS and NFS with `noac` option. I am currently stuck trying to build the kernel module of OrangeFS for the ARM64/aarch64

Cheers

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/87bl57dk70.fsf%4043-1.org.

--

Nicholas Yue
https://www.linkedin.com/in/nicholasyue/

[Nicholas Yue]

That is great news. I will create and mount a new partition with the settings suggested in the article and try again.

Cheers

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/0df71f12-0074-a34b-ec03-42db16579d92%40colostate.edu.