Write results usign HDF5 and the eXtensible data model and format

[Andrés Castillo Castellanos]

Hello everyone,

I have been running some tests using the latest version of Basilisk for natural convection using the Boussinesq approximation.
I've been working on implementing a parallel I/O strategy similar to the one used in Alya (and other large computing codes) 

http://www.prace-ri.eu/IMG/pdf/Implementing_a_XDMF_HDF5_Parallel_File_System_in_Alya-2.pdf

The main idea consists in separating our results in two types: a light data, which includes information regarding our data,
and heavy data, which contains large data sets. The Light data is an XML file which can be edited using gedit, whereas
the heavy data is stored using the HDF5 file format (https://en.wikipedia.org/wiki/Hierarchical_Data_Format).
Instead of using MPI I/O calls, we use a parallel version of HDF5 libraries which are usually available in most computing centers.
An excellent description of how this works can be found here (https://hdfgroup.org/wp/2015/08/parallel-io-with-hdf5/)

I have included a model of such implementation working for the multigrid solver in MPI, output_hdf5.h and a minimal example
for the differentially heated cavity in 3D using 8 MPI tasks, which is an useful benchmark (see Tric & Labrosse, 2000.  http://www.sciencedirect.com/science/article/pii/S0017931000000375)

To compile we have to link to the hdf5 libraries, which can be done using the h5cc or ph5cc linking tool. In MPI we can do the following;

qcc -source -D_MPI=1 minimal_example.c
h5pcc -Wall -std=c99 -O2 -D_MPI=1 _minimal_example.c -o minimal_example.out -lm

then execute using mpirun -np 8 ./odb-mgrd3d.out

This example takes a little less than 2 minutes using 8 MPI Tasks.

The resulting files are output_basilisk.h5 (heavy), output_basilisk.xmdf (light), and output_basilisk.xdf (light). These files can be read
directly using Paraview, Visit or others.




I have tested this routines up to 512 MPI tasks using ADA @ IDRIS  without problems.
# Multigrid, 3136 steps, 15578.6 CPU, 1.565e+04 real, 2.69e+07 points.step/s, 25 var
# 512 procs, MPI: min 2.9e+03 (18%) avg 3.8e+03 (24%) max 4.2e+03 (27%)


List the contents of the output file:
>> h5ls -r output_basilisk.h5
/                        Group
/field3d                 Group
/field3d/steady          Group
/field3d/steady/T        Dataset {512, 512, 512}
/field3d/steady/u.x      Dataset {512, 512, 512}
/field3d/steady/u.y      Dataset {512, 512, 512}
/field3d/steady/u.z      Dataset {512, 512, 512}
/grid                    Group
/grid/center             Group
/grid/center/CoordinateX Dataset {512}
/grid/center/CoordinateY Dataset {512}
/grid/center/CoordinateZ Dataset {512}
/grid/vertex             Group
/grid/vertex/CoordinateX Dataset {513}
/grid/vertex/CoordinateY Dataset {513}
/grid/vertex/CoordinateZ Dataset {513}

One last observation, everything is stored in single precision: 512*512*512*4 = 536MB.

What do you think? We might consider a more general implementation for other types of grids.



Best regards,

[Antoon van Hooft]

Hallo Andrés,

This looks great! I would be very interested to load octree data into paraview. I will keep an eye on this thead!  

gr

Antoon

[felixh...@hispeed.ch]

hola Andrés

i tried to run your example, but ran into an error, the functoin MESG was not found (it seems to be a more or less normal write statement). i use openmpi. is it a question of which mpi and therefore hdf libs I use?


cheers felix

[Andrés Castillo Castellanos]

Hey Felix,

Apparently I made a mistake and omitted a piece of code when I made the previous post.
I made a simpler self-contained example, which should allow you to create and to read back the results in XDMF.

I'm also attaching the resulting files so you can compare if everything runs ok:
- output_basilisk.h5                                           : Heavy data
- output_basilisk.xdmf                                       : Light Data
- output_basilisk000000.xmf                              : Initial conditions
- output_basilisk000001.xmf                              : First snapshots

I've testes this locally on a fresh install. Only comment is that
certain versions of paraview (5.0.1) might have issues reading XDMF. 

Please let me know if everything works for you.

Best regards,

[felixh...@hispeed.ch]

Hola Andres

tnx a lot. I'll install the latest paraview and try as soon as possible, I really have 5.0.1 actually installed :-O

muchos saludos Felix