HDF5 output on cluster

[Igal Tsarfis]

Hi

I'm having trouble outputting to HDF format on a cluster of 2 machines.
The cases I've tried:
a. running the same input deck with tecplot, it writes the files.
b. running the same input deck using parallel run of 2 CPU's on EITHER one of the machines, it outputs HDF & XMF files.
c. using the HDF group test, seems to be successful.

The errors are:
 --> creating hdf5 output file: /home/v/Desktop/work/Pflotran_b/b.h5
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:
  #000: H5F.c line 1500 in H5Fcreate(): unable to create file
    major: File accessability
    minor: Unable to open file

The output log is attached.
Any thoughts would be appreciated,

Igal

[Hammond, Glenn E]

Note sure what to tell you here.  I modified the non-legacy regression suite to attempt to write to an hdf5 files.  Can you pull the latest source and see if all regression tests pass?

 

cd $PFLOTRAN_DIR/src/pflotran

hg pull -u

make pflotran

make test

 

There should be 51 of 51 tests that pass.

Glenn

[Gautam Bisht]

Igal,

Did you installed PETSc with support for HDF5 (--download-hdf5=1 or --with-hdf5-dir=your-local-hdf5-dir)?

-Gautam.

[Igal Tsarfis]

At the moment I'm working on the legacy version compiled (because of the MPHASE support), so I'll make a new copy and compile as you suggested.

I'll accomplish that by tomorrow probably, and upload the output.

Thank you,

Igal

[Igal Tsarfis]

Yes, I did installed PETSc with support for HDF5 using --download-hdf5=1 and had no prior HDF5 installation.

[Igal Tsarfis]

Hi

I've compiled a non-legacy version on two machines, along to the legacy one.

The regression test logs are attached (51 tests, 2 of which are failed only about the tolerance).

Again, each machine writes HDF5 & XMF when run individually, but not when run together.

I've noticed that the h5 file and the XMF ones are created, however, stays empty due to the same errors as on the legacy run.

Igal

[Hammond, Glenn E]

Igal,

 

Yes, clearly this is not an issue with HDF5 on a single parallel run.  After a google search on the HDF5 error messages sent yesterday (i.e. “open failed on a remote node”), I found a couple posts that suggest that this is an issue with your MPI_IO and MPI installation.  You are using a very recent version of mpich (3.0.4).  I suggest installing an older, but more stable version of mpich2 (e.g. mpich2-1.4.1).  I have had issues with newer version.  Other than that, I am not sure what to suggest.

 

From here on out, we need to move this conversation over to the pflotran-dev mailing list as there are PETSc developers on that list that can possibly help us out.  But please try mpich2-1.4.1.  If that works, we can revisit the issue with mpich-3.0.4 with scientists at Argonne.

 

Glenn

[igal.t...@gmail.com]

Hi

I've tried to use the older mpich2-1.4.1 by typing for the master machine:
/usr/local/bin/mpiexec -n 4 .... machines.txt .. ./pflotran ...
and got the same errors on HDF5 write.

Igal

[Charlotte Barbier]

Hi Igal,

I have the same problem. I installed with openmpi using:

./config/configure.py --with-cc=gcc --with-fc=gfortran --download-f-blas-lapack=1 --download-openmpi=1 --download-hdf5=1 --download-metis=1 --download-parmetis=1 --with-debugging=1 --with-c2html=0

and install pflotran with "make pflotran".

The pflotran runs are working (serial or parallel) as long as I don't use a h5 file for the mesh, otherwise I get the error below. I tested with the simple example given in pflotran-dev/example_problems/umesh/mixed/ : it works on the login node but not when I submit to the cluster. The cluster is a SGI Altix machine...  

 

HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:

  #000: H5F.c line 1582 in H5Fopen(): unable to open file

    major: File accessability

    minor: Unable to open file

  #001: H5F.c line 1271 in H5F_open(): unable to open file: time = Thu Nov 21 10:40:05 2013

, name = 'mixed.h5', tent_flags = 0

    major: File accessability

    minor: Unable to open file

  #002: H5FD.c line 987 in H5FD_open(): open failed

    major: Virtual File Layer

    minor: Unable to initialize object

  #003: H5FDmpio.c line 1052 in H5FD_mpio_open(): MPI_File_open failed

    major: Internal error (too specific to document in detail)

    minor: Some MPI function failed

  #004: H5FDmpio.c line 1052 in H5FD_mpio_open(): MPI_ERR_NO_SUCH_FILE: no such file or directory

    major: Internal error (too specific to document in detail)

    minor: MPI Error String

HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:

  #000: H5D.c line 316 in H5Dopen2(): not a location

    major: Invalid arguments to routine

    minor: Inappropriate type

  #001: H5Gloc.c line 253 in H5G_loc(): invalid object ID

    major: Invalid arguments to routine

    minor: Bad value

HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:

  #000: H5D.c line 437 in H5Dget_space(): not a dataset

    major: Invalid arguments to routine

    minor: Inappropriate type

HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:

  #000: H5S.c line 794 in H5Sget_simple_extent_ndims(): not a dataspace

    major: Invalid arguments to routine

    minor: Inappropriate type

[Igal Tsarfis]

Charlotte,

Thank you for this information, I've contacted the PETSc support group with this issue.

I'll post any insight, if reached.

Igal

[Greg Lackey]

Igal,

I'm writing to ask if you ever found a resolution to this problem?

I have recently been getting a similar HDF5-DIAG error on a remote machine that I have been using. It only occurs when I use more than one processor and try to write an h5 output file.

Thanks

Greg Lackey

[Greg Lackey]

Hello everyone,

I'm still experiencing issues with creating pflotran output files on a remote machine using more than one processor.

It appears to be the same issue that Igal had back in 2013. I get the following HDF5-DIAG error when I output an h5 file:

 --> appending to hdf5 output file: pflotran.h5

HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) MPI-process 0:

  #000: H5D.c line 141 in H5Dcreate2(): not a location ID

    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5Gloc.c line 253 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value

This conversation was moved to pflotran-dev and I'm just wondering if there was ever a resolution.

Thanks,

Greg Lackey

[Hammond, Glenn E]

To my knowledge this has not been addressed, and I doubt it ever will be addressed until we can replicate the issue on one of our machines (I actually cannot remember whether anyone attempted to run your input deck on a developer machine.  Can someone remind me?).

Glenn

[Greg Lackey]

I ended up reconfiguring petsc and specifying my mpi directory (vs. --download-mpich=yes). This cleared up the issue with outputting HDF5 files using multiple processors. Not sure if it was an issue specific to my remote machine.

Thanks

[miru...@colorado.edu]

Greg, how did you do this? something like:

./configure --download-hdf5=yes --with-mpi-dir=<dir>

[Greg Lackey]

Mickey,

Yes, I believe that is what I did. But that was awhile ago now so I'm not 100% sure.