"locked" vtu outputs

[David Montiel Taboada]

Hello everyone,

We are using PRISMS-PF, a phase field framework finite element built upon the  deal.II library (with the MPI and p4est) dependencies. 

The code uses the deal.II DataOut class (see code snippet at the end of the email) to output simulation results into vtu files at certain time steps. 

We have observed that for parallel runs (especially on multiple nodes) the simulation gets stuck whenever a file of type (and extension) "lock" is created.

For example, for the initial conditions, the code is supposed to output a file with name:

solution-0000000.vtu

but then it also creates s file with name

solution-0000000.vtu-1364024289-50151.lock

This is what the support from the cluster told us about the issue:

This seems to be an issue between dealii and openmpi working together.
I looks like dealii will lock a file when it it working on it across multiple nodes so it doesn't get written to prematurely, but there might be an instance of communications in mpi failing.

We are complaining with gcc/8.2.0 and openmpi/3.1.4 but the issue still happens with openMPI/ 3.1.6. 

If we use openMPI/4.2.0 the "lock" doesn't happen but some of the files appear to be corrupt. 

We are using deal.II v 9.1.1

Has anyone encountered this issue? Any help is appreciated.

Thank you!

David

Output method (some sections have been ommited, output part in bold):

===

//outputResults() method for MatrixFreePDE class

#include "../../include/matrixFreePDE.h"

#include <deal.II/numerics/data_out.h>

//output results

template <int dim, int degree>

void MatrixFreePDE<dim,degree>::outputResults() {

  //log time

  computing_timer.enter_section("matrixFreePDE: output");

  //create DataOut object

  DataOut<dim> data_out;

  //loop over fields

  for(unsigned int fieldIndex=0; fieldIndex<fields.size(); fieldIndex++){

    //mark field as scalar/vector

    std::vector<DataComponentInterpretation::DataComponentInterpretation> dataType \

      (fields[fieldIndex].numComponents, \

       (fields[fieldIndex].type==SCALAR ? \

DataComponentInterpretation::component_is_scalar: \

DataComponentInterpretation::component_is_part_of_vector));

    //add field to data_out

    std::vector<std::string> solutionNames (fields[fieldIndex].numComponents, fields[fieldIndex].name.c_str());

    data_out.add_data_vector(*dofHandlersSet[fieldIndex], *solutionSet[fieldIndex], solutionNames, dataType);

  }

data_out.build_patches (degree);

  //write to results file

  //file name

  std::ostringstream cycleAsString;

  cycleAsString << std::setw(std::floor(std::log10(userInputs.totalIncrements))+1) << std::setfill('0') << currentIncrement;

  char baseFileName[100], vtuFileName[100];

  sprintf(baseFileName, "%s-%s", userInputs.output_file_name.c_str(), cycleAsString.str().c_str());

  sprintf(vtuFileName, "%s.%u.%s", baseFileName,Utilities::MPI::this_mpi_process(MPI_COMM_WORLD),userInputs.output_file_type.c_str());

  // Write to file in either vtu or vtk format

  if (userInputs.output_file_type == "vtu"){

      // Set flags to output the time and cycle number as part of the vtu file

      dealii::DataOutBase::VtkFlags flags;

      flags.time = currentTime;

      flags.cycle = currentIncrement;

      flags.print_date_and_time = true;

      data_out.set_flags(flags);

      // Write the results to a file shared between all processes

      char svtuFileName[100];

      sprintf(svtuFileName, "%s.%s", baseFileName ,userInputs.output_file_type.c_str());

      data_out.write_vtu_in_parallel(svtuFileName, MPI_COMM_WORLD);

      pcout << "Output written to:" << svtuFileName << "\n\n";

    }

  //log time

  computing_timer.exit_section("matrixFreePDE: output");

}

===

[Timo Heister]

Hi David,

> For example, for the initial conditions, the code is supposed to output a file with name:
>
> solution-0000000.vtu
>
> but then it also creates s file with name
>
> solution-0000000.vtu-1364024289-50151.lock
>
> This is what the support from the cluster told us about the issue:

The code you posted uses MPI I/O for parallel output. This lock file
is created by the operating system and to me it looks like the
filesystem you are writing to might be an NFS storage that is not set
up to work well with parallel IO. Generally, NSF is not a great option
for performance. It might also be that the file system is not
configured correctly.

I have a few suggestions:
1. Use a real parallel file system for IO (most clusters have support
for a fast parallel file system and a slow home directory on NSF)
2. Change the code to not use parallel IO (replace
write_vtu_in_parallel by a call to write_vtu_with_pvtu_record() and
set n_groups = 0). This is slower, especially for large computations
with 100+ cores. You might need to change a couple of lines of code.
3. Talk to the admins to change/update the MPI implementation
(assuming this actually is a parallel file system)

Best,
Timo


--
Timo Heister
http://www.math.clemson.edu/~heister/

[David Montiel Taboada]

Thank you, Timo

I will look into it!

David

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/CAMRj59E1v%2BN3%3DXfmqLPV3NX_hhNuG_zwoMsejs_2b_6CL1Grng%40mail.gmail.com.