PFLOTRAN MPI Initialization Error with HDF5

139 views
Skip to first unread message

Won Woo Yoon

unread,
Nov 25, 2024, 7:15:04 AM11/25/24
to pflotran-users

Hi all,

I'm trying to run PFLOTRAN on a cluster with 8 threads per node, utilizing SSH for communication between machines. I've verified that PFLOTRAN runs properly on all three machines independently and the connection between the machines via SSH without issues.

After then, I attempted the following command to utilize all nodes:

mpirun -n 24 -f /home/geofluids/hostfile $PFLOTRAN_DIR/src/pflotran/pflotran -input_prefix sample_197

The hostfile I used is

IP of machine1:8
IP of machine2:8
IP of machine3:8

 Unfortunately, this returned an initialization error while creating the HDF5 file, with the error message starting as follows:

HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 16: #000: H5G.c line 231 in H5Gcreate2(): unable to synchronously create group

Since the complete error message is quite long, I’ve attached the as a .txt file for reference.

I would greatly appreciate any guidance or suggestions on how to resolve this issue.

Thank you in advance for your help!

Best regards,

Won Woo Yoon

error.txt

Hammond, Glenn E

unread,
Nov 27, 2024, 5:58:57 PM11/27/24
to pflotra...@googlegroups.com

Won Woo,

 

I am unsure that to recommend. Can you send the file $PETSC_DIR/$PETSC_ARCH/lib/petsc/configure.log to pflotr...@googlegroups.com?

 

Glenn

 

From: pflotra...@googlegroups.com <pflotra...@googlegroups.com> on behalf of Won Woo Yoon <yys...@yonsei.ac.kr>
Date: Monday, November 25, 2024 at 4:15
AM
To: pflotran-users <pflotra...@googlegroups.com>
Subject: [pflotran-users: 8155] PFLOTRAN MPI Initialization Error with HDF5

Check twice before you click! This email originated from outside PNNL.

 

--
You received this message because you are subscribed to the Google Groups "pflotran-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pflotran-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pflotran-users/aca50fd0-1dce-44f8-949a-cd31ed4eb1den%40googlegroups.com.

Hammond, Glenn E

unread,
Dec 3, 2024, 11:37:01 AM12/3/24
to pflotra...@googlegroups.com

Won Woo,

 

You are running Ubuntu on a cluster (outside my expertise). I assume that the issue does not exist on a single node….  If you can replicate the issue on a PETSc test problem that employs HDF5, perhaps the PETSc developers can help. I assume that you have searched Google for answers…?

 

Glenn

 

From: pflotra...@googlegroups.com <pflotra...@googlegroups.com> on behalf of Won Woo Yoon <yys...@yonsei.ac.kr>
Date: Monday, November 25, 2024 at 4:15
AM
To: pflotran-users <pflotra...@googlegroups.com>
Subject: [pflotran-users: 8155] PFLOTRAN MPI Initialization Error with HDF5

Check twice before you click! This email originated from outside PNNL.

 

Hi all,

--

Barry Smith

unread,
Dec 3, 2024, 7:48:23 PM12/3/24
to pflotran-users
The configure.log has no issues. Can you try running a HDF5 test code (there must be some distributed in the $PETSC_DIR/$PETSC_ARCH/externalpages/hdf5 directory) on exactly the same  the same hosts with the same HDF5 libraries.

Won Woo Yoon

unread,
Dec 9, 2024, 9:02:41 PM12/9/24
to pflotra...@googlegroups.com

Thank you for your kind advice, and sorry for the late response.

I have identified some potential issues with the HDF5 installation version, but I have not yet had the chance to test a fix. 

I plan to test it as soon as possible, and I will share an update if I am successful in resolving the issue.

Thank you once again for your valuable suggestions.

Best regards,
Won Woo Yoon


2024년 12월 4일 (수) 오전 9:48, Barry Smith <pets...@gmail.com>님이 작성:

laurin76....@gmail.com

unread,
May 8, 2025, 8:00:04 AMMay 8
to pflotran-users
Dear all,

I have the same issue on our cluster using a Debian 12 virtual machine. The issue seems to be caused by running the current version of pflotran / petsc on an input file that is located on a NFS-mounted disc. The error occurs during writing of the grid/mesh to the h5 output file (potentially as a separate process). The same installation works when running the input file on /tmp. 
As an example, I use regression_tests/general/gas_injection.in with FORMAT HDF5 (removing FORMAT TECPLOT POINT and the skip/noskip keywords in OUTPUT).

Best regards,
Laurin

================================
  PFLOTRAN Development Version
================================

"grid_structured_type" set to default value.
pflotran card:: NUMERICAL_METHODS
pflotran card:: REGRESSION
pflotran card:: GRID
pflotran card:: FLUID_PROPERTY
"FLUID_PROPERTY,diffusion_coefficient units" set to default value.
pflotran card:: MATERIAL_PROPERTY
  Name :: soil
pflotran card:: MATERIAL_PROPERTY
  Name :: impermeable
"MATERIAL_PROPERTY,ROCK_DENSITY units" set to default value.
"MATERIAL_PROPERTY,THERMAL_CONDUCTIVITY_DRY units" set to default value.
"MATERIAL_PROPERTY,THERMAL_CONDUCTIVITY_WET units" set to default value.
"MATERIAL_PROPERTY,HEAT_CAPACITY units" set to default value.
pflotran card:: CHARACTERISTIC_CURVES
  Name :: cc1
 WARNING: Mualem-van Genuchten relative permeability function is being used without SMOOTH option.
pflotran card:: CHARACTERISTIC_CURVES
  Name :: cc2
 WARNING: Mualem-van Genuchten relative permeability function is being used without SMOOTH option.
pflotran card:: OUTPUT
"OUTPUT,FORMAT,HDF5,# FILES" set to default value.
pflotran card:: TIME
pflotran card:: REGION
all
pflotran card:: REGION
top
pflotran card:: REGION
bottom
pflotran card:: REGION
impermeable_zone
pflotran card:: REGION
top_pt
pflotran card:: REGION
bottom_pt
pflotran card:: REGION
well_2
pflotran card:: OBSERVATION
pflotran card:: OBSERVATION
pflotran card:: FLOW_CONDITION
bottom_pressure
pflotran card:: FLOW_CONDITION
atmosphere
pflotran card:: FLOW_CONDITION
injection
pflotran card:: INITIAL_CONDITION
"Initial Condition name" set to default value.
pflotran card:: BOUNDARY_CONDITION
pflotran card:: BOUNDARY_CONDITION
pflotran card:: SOURCE_SINK
pflotran card:: SOURCE_SINK
pflotran card:: STRATA
pflotran card:: STRATA
pflotran card:: END_SUBSURFACE

Extent of Gridded Domain
 X: 0.000000E+00 - 9.000000E+00
 Y: 0.000000E+00 - 1.000000E+00
 Z: 0.000000E+00 - 1.000000E+01

Size of Gridded Domain
 Number of grid cells: 90

Min/Max Cell Volumes [m^3]
 Maximum: 1.000000E+00
 Minimum: 1.000000E+00

Number of active grid cells: 90
Number of inactive grid cells: 0

Default HDF5 method is used in Initialization
  Beginning setup of FLOW SNES
 number of dofs =   3, number of phases =   2
 mode = General: p, sg/X, T
  Finished setting up FLOW SNES

 Requested processors and decomposition =     1, npx,y,z=   -1  -1  -1
 Actual decomposition: npx,y,z=    1   1   1

WARNING: Waypoint at time:  3.1536E+05 is beyond the end of simulation
WARNING: Waypoint at time:  1.5768E+07 is beyond the end of simulation

FLOW Time Stepper SNES
      maximum number of steps: 999999
constant time steps threshold: 5
       maximum number of cuts: 16
             reduction factor: 5.000000E-01
        maximum growth factor: 2.000000E+00
                 acceleration: on
       acceleration threshold: 5
       number of ramp entries: 5
                ramp entry #1: 2.0000
                ramp entry #2: 2.0000
                ramp entry #3: 2.0000
                ramp entry #4: 2.0000
                ramp entry #5: 2.0000
                ramp entry #6: 1.8000
                ramp entry #7: 1.6000
                ramp entry #8: 1.4000
                ramp entry #9: 1.2000
               ramp entry #10: 1.0000
               ramp entry #11: 1.0000
               ramp entry #12: 1.0000
               ramp entry #13: 1.0000

FLOW Newton Solver
                         atol: 1.000000E-50
                         rtol: 1.000000E-08
                         stol: 1.000000E-08
                         dtol: 1.000000E+09
                      maxnorm: 1.000000E+20
                    inftolres: -9.990000E+02
                    inftolupd: -9.990000E+02
                 inftolrelupd: -9.990000E+02
                 inftolsclres: -9.990000E+02
                     max iter: 15
                     min iter: 1
                         maxf: 10000
                  matrix type: baij
         precond. matrix type: baij
               inexact newton: off
            print convergence: on
   print detailed convergence: off
          check infinity norm: on

FLOW Linear Solver
                       solver: preonly
               preconditioner: lu
                         atol: 1.000000E-50
                         rtol: 1.000000E-05
                         dtol: 1.000000E+04
            maximum iteration: 10000

 --> creating hdf5 output file: gas_injection.h5
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO
      0.05 Seconds to write HDF5 file.

 Finished Initialization


== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 7.88E-02 2x: 0.00E+00 2u: 0.00E+00 ir: 2.32E-02 iu: 0.00E+00 rsn:   0
  1 2r: 5.56E-05 2x: 1.65E+06 2u: 3.62E+05 ir: 1.85E-05 iu: 8.34E+04 rsn:   0
  2 2r: 3.43E-09 2x: 1.65E+06 2u: 1.29E+01 ir: 1.90E-09 iu: 3.86E+00 rsn: 999

 Step      1 Time=  3.17098E-08 Dt=  3.17098E-08 [y] conv_reason: 999
  newton =   2 [       2] linear =     2 [         2] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.342845E-08 0.380939E-10 0.189745E-08
  --> max chng: dpl=   8.3360E+04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   1.5825E-09  dt=   5.9746E-04 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 4.79E-03 2x: 0.00E+00 2u: 0.00E+00 ir: 9.76E-04 iu: 0.00E+00 rsn:   0
  1 2r: 1.00E-06 2x: 1.53E+06 2u: 1.44E+05 ir: 3.63E-07 iu: 2.37E+04 rsn:   0
  2 2r: 1.61E-10 2x: 1.53E+06 2u: 2.28E+00 ir: 5.24E-11 iu: 1.12E+00 rsn: 999

 Step      2 Time=  9.51294E-08 Dt=  6.34196E-08 [y] conv_reason: 999
  newton =   2 [       4] linear =     2 [         4] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.160785E-09 0.178650E-11 0.523837E-10
  --> max chng: dpl=   2.3748E+04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   3.1649E-09  dt=   1.7027E-04 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 9.53E-04 2x: 0.00E+00 2u: 0.00E+00 ir: 1.39E-04 iu: 0.00E+00 rsn:   0
  1 2r: 1.83E-07 2x: 1.48E+06 2u: 5.67E+04 ir: 5.29E-08 iu: 1.25E+04 rsn:   0
  2 2r: 8.53E-11 2x: 1.48E+06 2u: 2.56E+00 ir: 2.22E-11 iu: 2.47E+00 rsn: 999

 Step      3 Time=  2.21969E-07 Dt=  1.26839E-07 [y] conv_reason: 999
  newton =   2 [       6] linear =     2 [         6] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.853368E-10 0.948187E-12 0.221942E-10
  --> max chng: dpl=   1.2535E+04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   6.3299E-09  dt=   5.9031E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 1.80E-04 2x: 0.00E+00 2u: 0.00E+00 ir: 2.41E-05 iu: 0.00E+00 rsn:   0
  1 2r: 3.51E-08 2x: 1.46E+06 2u: 2.72E+04 ir: 2.53E-08 iu: 1.43E+04 rsn:   0
  2 2r: 4.34E-11 2x: 1.46E+06 2u: 1.74E+00 ir: 1.14E-11 iu: 1.67E+00 rsn: 999

 Step      4 Time=  4.75647E-07 Dt=  2.53678E-07 [y] conv_reason: 999
  newton =   2 [       8] linear =     2 [         8] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.434191E-10 0.482434E-12 0.114132E-10
  --> max chng: dpl=   1.4296E+04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   1.2660E-08  dt=   6.1651E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 3.11E-05 2x: 0.00E+00 2u: 0.00E+00 ir: 1.26E-05 iu: 0.00E+00 rsn:   0
  1 2r: 5.89E-09 2x: 1.46E+06 2u: 2.03E+04 ir: 3.98E-09 iu: 1.28E+04 rsn:   0
  2 2r: 1.94E-11 2x: 1.46E+06 2u: 3.89E+00 ir: 5.55E-12 iu: 2.46E+00 rsn: 999

 Step      5 Time=  1.00000E-06 Dt=  5.24353E-07 [y] conv_reason: 999
  newton =   2 [      10] linear =     2 [        10] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.193517E-10 0.215019E-12 0.555270E-11
  --> max chng: dpl=   1.2824E+04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   2.6168E-08  dt=   5.5304E-05 dsg=   0.0000E+00

 --> appending to hdf5 output file: gas_injection.h5
      0.03 Seconds to write HDF5 file.

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 9.81E-06 2x: 0.00E+00 2u: 0.00E+00 ir: 5.46E-06 iu: 0.00E+00 rsn:   0
  1 2r: 1.26E-09 2x: 1.46E+06 2u: 7.93E+03 ir: 7.92E-10 iu: 5.41E+03 rsn:   0
  2 2r: 2.28E-11 2x: 1.46E+06 2u: 7.22E-01 ir: 6.81E-12 iu: 6.61E-01 rsn: 999

 Step      6 Time=  1.50736E-06 Dt=  5.07357E-07 [y] conv_reason: 999
  newton =   2 [      12] linear =     2 [        12] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.228124E-10 0.253471E-12 0.681326E-11
  --> max chng: dpl=   5.4123E+03 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   2.5320E-08  dt=   2.3342E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 3.96E-06 2x: 0.00E+00 2u: 0.00E+00 ir: 2.38E-06 iu: 0.00E+00 rsn:   0
  1 2r: 4.15E-10 2x: 1.46E+06 2u: 4.07E+03 ir: 2.89E-10 iu: 2.94E+03 rsn:   0
  2 2r: 9.58E-12 2x: 1.46E+06 2u: 2.56E-01 ir: 2.38E-12 iu: 2.48E-01 rsn: 999

 Step      7 Time=  2.52207E-06 Dt=  1.01471E-06 [y] conv_reason: 999
  newton =   2 [      14] linear =     2 [        14] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.957917E-11 0.106435E-12 0.237720E-11
  --> max chng: dpl=   2.9432E+03 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   5.0638E-08  dt=   1.2699E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 1.04E-06 2x: 0.00E+00 2u: 0.00E+00 ir: 6.48E-07 iu: 0.00E+00 rsn:   0
  1 2r: 9.77E-11 2x: 1.46E+06 2u: 1.20E+03 ir: 8.38E-11 iu: 9.08E+02 rsn:   0
  2 2r: 5.25E-12 2x: 1.46E+06 2u: 3.00E-02 ir: 1.47E-12 iu: 2.77E-02 rsn: 999

 Step      8 Time=  4.55150E-06 Dt=  2.02943E-06 [y] conv_reason: 999
  newton =   2 [      16] linear =     2 [        16] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.524907E-11 0.583230E-13 0.147259E-11
  --> max chng: dpl=   9.0790E+02 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   1.0127E-07  dt=   3.9293E-06 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.67E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.62E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.89E-11 2x: 1.46E+06 2u: 1.93E+02 ir: 1.17E-11 iu: 1.50E+02 rsn:   0
  2 2r: 2.58E-12 2x: 1.46E+06 2u: 7.61E-03 ir: 6.55E-13 iu: 7.21E-03 rsn: 999

 Step      9 Time=  8.61035E-06 Dt=  4.05885E-06 [y] conv_reason: 999
  newton =   2 [      18] linear =     2 [        18] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.258017E-11 0.286685E-13 0.654514E-12
  --> max chng: dpl=   1.4960E+02 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   2.0254E-07  dt=   7.7379E-06 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.21E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.62E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.25E-11 2x: 1.46E+06 2u: 1.34E+01 ir: 6.42E-12 iu: 1.06E+01 rsn:   0
  2 2r: 7.53E-12 2x: 1.46E+06 2u: 6.77E-04 ir: 2.74E-12 iu: 6.11E-04 rsn: 999

 Step     10 Time=  1.00000E-05 Dt=  1.38965E-06 [y] conv_reason: 999
  newton =   2 [      20] linear =     2 [        20] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.753376E-11 0.837085E-13 0.274244E-11
  --> max chng: dpl=   1.0567E+01 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   6.9343E-08  dt=   2.6490E-06 dsg=   0.0000E+00

 --> appending to hdf5 output file: gas_injection.h5
      0.02 Seconds to write HDF5 file.

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.20E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.62E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.27E-11 2x: 1.46E+06 2u: 3.28E+00 ir: 1.16E-11 iu: 2.61E+00 rsn:   0
  2 2r: 1.38E-12 2x: 1.46E+06 2u: 6.17E-05 ir: 4.10E-13 iu: 5.41E-05 rsn: 999

 Step     11 Time=  1.81177E-05 Dt=  8.11771E-06 [y] conv_reason: 999
  newton =   2 [      22] linear =     2 [        22] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.138120E-11 0.153466E-13 0.409511E-12
  --> max chng: dpl=   2.6133E+00 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   4.0503E-07  dt=   1.5461E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.20E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.62E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.01E-11 2x: 1.46E+06 2u: 1.40E-01 ir: 8.78E-12 iu: 1.13E-01 rsn:   0
  2 2r: 6.69E-13 2x: 1.46E+06 2u: 1.12E-06 ir: 2.45E-13 iu: 1.07E-06 rsn: 999

 Step     12 Time=  3.43531E-05 Dt=  1.62354E-05 [y] conv_reason: 999
  newton =   2 [      24] linear =     2 [        24] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.668964E-12 0.743293E-14 0.244502E-12
  --> max chng: dpl=   1.1253E-01 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   8.0990E-07  dt=   3.0870E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.20E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.62E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.73E-11 2x: 1.46E+06 2u: 2.96E-03 ir: 1.53E-11 iu: 2.39E-03 rsn:   0
  2 2r: 3.33E-13 2x: 1.46E+06 2u: 1.15E-07 ir: 1.12E-13 iu: 7.67E-08 rsn: 999

 Step     13 Time=  6.68239E-05 Dt=  3.24708E-05 [y] conv_reason: 999
  newton =   2 [      26] linear =     2 [        26] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.332582E-12 0.369536E-14 0.112347E-12
  --> max chng: dpl=   2.3933E-03 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   1.6192E-06  dt=   6.1531E-05 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.19E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.61E-07 iu: 0.00E+00 rsn:   0
  1 2r: 9.22E-12 2x: 1.46E+06 2u: 2.62E-04 ir: 5.95E-12 iu: 8.68E-05 rsn:   0
  2 2r: 3.43E-13 2x: 1.46E+06 2u: 1.14E-07 ir: 1.38E-13 iu: 8.55E-08 rsn: 999

 Step     14 Time=  1.00000E-04 Dt=  3.31761E-05 [y] conv_reason: 999
  newton =   2 [      28] linear =     2 [        28] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.343172E-12 0.381302E-14 0.138061E-12
  --> max chng: dpl=   8.6797E-05 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   1.6537E-06  dt=   6.2650E-05 dsg=   0.0000E+00

 --> appending to hdf5 output file: gas_injection.h5
      0.03 Seconds to write HDF5 file.

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.19E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.61E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.15E-11 2x: 1.46E+06 2u: 5.27E-04 ir: 7.54E-12 iu: 1.70E-04 rsn:   0
  2 2r: 1.80E-13 2x: 1.46E+06 2u: 2.82E-07 ir: 5.44E-14 iu: 9.13E-08 rsn: 999

 Step     15 Time=  1.64942E-04 Dt=  6.49417E-05 [y] conv_reason: 999
  newton =   2 [      30] linear =     2 [        30] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.180144E-12 0.200160E-14 0.543591E-13
  --> max chng: dpl=   1.6976E-04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   3.2345E-06  dt=   1.2181E-04 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.18E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.60E-07 iu: 0.00E+00 rsn:   0
  1 2r: 1.61E-11 2x: 1.46E+06 2u: 1.05E-03 ir: 1.45E-11 iu: 3.38E-04 rsn:   0
  2 2r: 9.13E-14 2x: 1.46E+06 2u: 3.13E-07 ir: 2.29E-14 iu: 1.08E-07 rsn: 999

 Step     16 Time=  2.94825E-04 Dt=  1.29883E-04 [y] conv_reason: 999
  newton =   2 [      32] linear =     2 [        32] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.912858E-13 0.101429E-14 0.229379E-13
  --> max chng: dpl=   3.3818E-04 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   6.4589E-06  dt=   2.4037E-04 dsg=   0.0000E+00

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.16E-07 2x: 0.00E+00 2u: 0.00E+00 ir: 1.57E-07 iu: 0.00E+00 rsn:   0
(0): State Transition: Liquid -> 2 Phase at Cell        5
  1 2r: 1.02E-05 2x: 1.46E+06 2u: 2.09E-03 ir: 8.53E-06 iu: 6.71E-04 rsn:   0
  2 2r: 9.09E-03 2x: 1.46E+06 2u: 1.49E+04 ir: 7.56E-03 iu: 1.49E+04 rsn:   0
  3 2r: 6.26E-05 2x: 1.46E+06 2u: 1.41E+04 ir: 5.22E-05 iu: 1.41E+04 rsn:   0
  4 2r: 6.58E-06 2x: 1.46E+06 2u: 2.30E+02 ir: 5.41E-06 iu: 2.30E+02 rsn:   0
  5 2r: 2.96E-07 2x: 1.46E+06 2u: 3.20E+01 ir: 2.49E-07 iu: 3.20E+01 rsn:   0
  6 2r: 1.12E-10 2x: 1.46E+06 2u: 9.03E-01 ir: 9.50E-11 iu: 9.02E-01 rsn:   0
  7 2r: 4.92E-14 2x: 1.46E+06 2u: 3.70E-04 ir: 1.75E-14 iu: 3.69E-04 rsn: 999

 Step     17 Time=  5.54592E-04 Dt=  2.59767E-04 [y] conv_reason: 999
  newton =   7 [      39] linear =     7 [        39] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 7 / 7
  --> SNES Residual: 0.491636E-13 0.546262E-15 0.175165E-13
  --> max chng: dpl=   1.0579E+01 dpg=   0.0000E+00 dpa=   0.0000E+00
                dxa=   9.1075E-06  dt=   3.4584E-04 dsg=   0.0000E+00
 Dt limited by Unknown: Val=-9.990E+02, Gov=-9.990E+02, Scale=0.33


== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 4.81E-06 2x: 0.00E+00 2u: 0.00E+00 ir: 4.21E-06 iu: 0.00E+00 rsn:   0
  1 2r: 2.45E-05 2x: 1.46E+06 2u: 2.56E+02 ir: 2.05E-05 iu: 2.56E+02 rsn:   0
  2 2r: 6.29E-07 2x: 1.46E+06 2u: 5.70E+01 ir: 5.56E-07 iu: 5.69E+01 rsn:   0
  3 2r: 2.39E-10 2x: 1.46E+06 2u: 1.42E+00 ir: 1.97E-10 iu: 4.88E-01 rsn: 999

 Step     18 Time=  6.40315E-04 Dt=  8.57230E-05 [y] conv_reason: 999
  newton =   3 [      42] linear =     3 [        42] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 3 / 3
  --> SNES Residual: 0.238825E-09 0.265361E-11 0.197250E-09
  --> max chng: dpl=   1.5220E+01 dpg=   1.9934E+02 dpa=   1.9934E+02
                dxa=   4.3290E-06  dt=   1.6452E-04 dsg=   1.1431E-03

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 1.24E-05 2x: 0.00E+00 2u: 0.00E+00 ir: 1.10E-05 iu: 0.00E+00 rsn:   0
(0): State Transition: Liquid -> 2 Phase at Cell       14
(0): State Transition: Liquid -> 2 Phase at Cell       23
(0): State Transition: Liquid -> 2 Phase at Cell       32
  1 2r: 1.22E-05 2x: 1.46E+06 2u: 8.30E+01 ir: 5.62E-06 iu: 8.04E+01 rsn:   0
  2 2r: 9.57E-03 2x: 1.46E+06 2u: 1.86E+04 ir: 7.32E-03 iu: 1.81E+04 rsn:   0
  3 2r: 8.02E-04 2x: 1.46E+06 2u: 1.76E+04 ir: 6.41E-04 iu: 1.71E+04 rsn:   0
  4 2r: 2.26E-05 2x: 1.46E+06 2u: 1.66E+03 ir: 1.51E-05 iu: 1.65E+03 rsn:   0
  5 2r: 4.69E-07 2x: 1.46E+06 2u: 5.98E+01 ir: 3.76E-07 iu: 5.40E+01 rsn:   0
  6 2r: 5.93E-11 2x: 1.46E+06 2u: 1.53E+00 ir: 4.68E-11 iu: 1.47E+00 rsn:   0
  7 2r: 6.11E-14 2x: 1.46E+06 2u: 3.85E-04 ir: 1.78E-14 iu: 3.67E-04 rsn: 999

 Step     19 Time=  8.11761E-04 Dt=  1.71446E-04 [y] conv_reason: 999
  newton =   7 [      49] linear =     7 [        49] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 7 / 7
  --> SNES Residual: 0.611178E-13 0.679087E-15 0.178148E-13
  --> max chng: dpl=   8.5318E+01 dpg=   1.0930E+02 dpa=   1.0930E+02
                dxa=   4.2997E-06  dt=   1.6478E-04 dsg=   7.8605E-04
 Dt limited by Unknown: Val=-9.990E+02, Gov=-9.990E+02, Scale=0.33


== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 2.06E-05 2x: 0.00E+00 2u: 0.00E+00 ir: 1.40E-05 iu: 0.00E+00 rsn:   0
  1 2r: 1.98E-05 2x: 1.46E+06 2u: 5.20E+02 ir: 1.30E-05 iu: 3.35E+02 rsn:   0
  2 2r: 5.59E-08 2x: 1.46E+06 2u: 7.83E+01 ir: 3.43E-08 iu: 5.54E+01 rsn:   0
  3 2r: 1.58E-12 2x: 1.46E+06 2u: 3.77E-01 ir: 1.23E-12 iu: 1.98E-01 rsn: 999

 Step     20 Time=  8.68338E-04 Dt=  5.65772E-05 [y] conv_reason: 999
  newton =   3 [      52] linear =     3 [        52] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 3 / 3
  --> SNES Residual: 0.157505E-11 0.175006E-13 0.123061E-11
  --> max chng: dpl=   8.5084E+01 dpg=   2.7950E+02 dpa=   2.7950E+02
                dxa=   1.3241E-06  dt=   5.3423E-05 dsg=   1.2703E-03

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 3.13E-05 2x: 0.00E+00 2u: 0.00E+00 ir: 1.85E-05 iu: 0.00E+00 rsn:   0
  1 2r: 1.36E-05 2x: 1.46E+06 2u: 2.76E+02 ir: 1.00E-05 iu: 1.63E+02 rsn:   0
  2 2r: 8.11E-09 2x: 1.46E+06 2u: 6.09E+01 ir: 6.52E-09 iu: 4.96E+01 rsn:   0
  3 2r: 7.90E-12 2x: 1.46E+06 2u: 3.52E-01 ir: 6.29E-12 iu: 2.77E-01 rsn: 999

 Step     21 Time=  9.81492E-04 Dt=  1.13154E-04 [y] conv_reason: 999
  newton =   3 [      55] linear =     3 [        55] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 3 / 3
  --> SNES Residual: 0.789906E-11 0.877673E-13 0.629101E-11
  --> max chng: dpl=   7.7114E+01 dpg=   1.1317E+02 dpa=   1.1318E+02
                dxa=   5.3300E-06  dt=   2.0918E-04 dsg=   1.5311E-03

== GENERAL MULTIPHASE FLOW =====================================================
  0 2r: 1.70E-05 2x: 0.00E+00 2u: 0.00E+00 ir: 1.11E-05 iu: 0.00E+00 rsn:   0
  1 2r: 1.58E-07 2x: 1.46E+06 2u: 3.88E+01 ir: 1.15E-07 iu: 1.11E+01 rsn:   0
  2 2r: 1.60E-12 2x: 1.46E+06 2u: 8.67E-01 ir: 1.13E-12 iu: 5.88E-01 rsn: 999

 Step     22 Time=  1.00000E-03 Dt=  1.85080E-05 [y] conv_reason: 999
  newton =   2 [      57] linear =     2 [        57] cuts =  0 [   0]
  --> SNES Linear/Non-Linear Iterations = 2 / 2
  --> SNES Residual: 0.159526E-11 0.177251E-13 0.113206E-11
  --> max chng: dpl=   1.3238E+01 dpg=   1.0497E+01 dpa=   1.0497E+01
                dxa=   9.4334E-07  dt=   3.6966E-05 dsg=   2.2015E-04

 --> appending to hdf5 output file: gas_injection.h5
      0.02 Seconds to write HDF5 file.

PMCSubsurfaceFlow
 Total Time: 5.5063E-02 seconds

 FLOW TS SNES steps = 22  newton = 57  linear = 57  cuts = 0
 FLOW TS SNES Wasted Linear Iterations = 0
 FLOW TS SNES Wasted Newton Iterations = 0
 FLOW TS SNES time = 0.1 seconds

 --> write regression output file: gas_injection.regression

 Wall Clock Time:  2.3398E-01 [sec]   3.8997E-03 [min]   6.4995E-05 [hr]

Hammond, Glenn E

unread,
May 13, 2025, 11:49:13 AMMay 13
to pflotra...@googlegroups.com

Laurin,

 

Sorry for the delayed response. Have you compared the permissions on both directories (and the hierarchy of directories above the local directory on the NFS-mounted disk? I suspect that this is a permissions issue.


Glenn

 

laurin76....@gmail.com

unread,
May 22, 2025, 6:48:01 AMMay 22
to pflotran-users
Dear Glen,

We compared and checked permissions, without success.
I reinstalled PFLOTRAN maint/v5.0 with PETSC 3.20.0 at the exact same location and the example "general/gas_injection.in with HDF5 OUTPUT" works fine.
The issue with the latest version (PFLOTRAN v6.0 with PETSC 3.21.5) seems to be the writing of the mesh/grid to hdf5 by the MPI process 0. The rest of the results is written ok.
Please note that the error message appears near the top of the screen output (not the output file), when the mesh is written to hdf5.
I am on Debian 12.11.

I have given the error message to ChatGPT (apologies for my ignorance) and the answer sounds quite plausible (as always):
- NFS lacks robust support for MPI_File_sync, which is required in some cases by the ROMIO MPI-IO implementation (used by MPICH/OpenMPI).
- Parallel HDF5 uses MPI-IO under the hood. If you’re doing independent I/O (not collective), and the driver decides that MPI_File_sync() is required, it will fail on NFS.
- This is an issue with HDF5 1.14.x and newer ROMIO implementations, which have become stricter.

Thanks a lot for your help!
Laurin

HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:
  #000: H5D.c line 1371 in H5Dwrite(): can't synchronously write data
    major: Dataset
    minor: Write failed
  #001: H5D.c line 1317 in H5D__write_api_common(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5VLcallback.c line 2282 in H5VL_dataset_write_direct(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #003: H5VLcallback.c line 2237 in H5VL__dataset_write(): dataset write failed
    major: Virtual Object Layer
    minor: Write failed
  #004: H5VLnative_dataset.c line 420 in H5VL__native_dataset_write(): can't write data
    major: Dataset
    minor: Write failed
  #005: H5Dio.c line 744 in H5D__write(): unable to adjust I/O info for parallel or selection I/O
    major: Dataset
    minor: Unable to initialize object
  #006: H5Dio.c line 1296 in H5D__ioinfo_adjust(): Can't perform independent write when MPI_File_sync is required by ROMIO driver.
    major: Dataset
    minor: Can't perform independent IO



gas_injection.in

Hammond, Glenn E

unread,
May 28, 2025, 8:09:59 PMMay 28
to pflotra...@googlegroups.com

Laurin,

 

Please send the configure.log for PETSc 3.20.0 to pflotr...@googlegroups.com. I want to see which HDF5 version PETSc installs.

 

The correct syntax is in place for independent I/O:

https://bitbucket.org/pflotran/pflotran/src/fa3ffa2533fb6a1fc50c992c34f4e1f0f2eae431/src/pflotran/output_hdf5.F90#lines-1512

which leads me to believe that the issue is HDF5 1.14.x with ROMIO, the last bullet that ChatGPT reports.

 

Thanks,

 

Glenn

 

laurin76....@gmail.com

unread,
Jun 6, 2025, 3:31:37 AMJun 6
to pflotran-users
Hi Glenn,

The hdf5 version of petsc 2.20.0 seems to be hdf5-1.12.2.
I just sent the configure.log.

Thanks,
Laurin
Reply all
Reply to author
Forward
0 new messages