Problem with HDF5-mpi h5py on d3

207 views
Skip to first unread message

Jose Rafael

unread,
Oct 14, 2021, 6:22:23 PM10/14/21
to Dedalus Users
Hi, 

Recently, I have been running into a strange issue when using d3. I can run simulations without any issues, but when I plot some files 

(e.g., using the plot_sphere.py file on the d3 branch, shell example) I obtain this error

2021-10-13 18:33:07,447 post 0/4 INFO :: Visiting set slices/slices_s1.h5
(start: 0, end: 13)
Traceback (most recent call last):
  File "plot_sphere.py", line 88, in <module>
    post.visit_writes(args['<files>'], main, output=output_path)
  File
"/home/jorafb/dedalus_env/lib/python3.8/site-packages/dedalus-2.1905-py3.8-linux-x86_64.egg/dedalus/
tools/post.py",
line 44, in visit_writes
    function(set_path, start, count, **kw)
  File "plot_sphere.py", line 53, in main
    data = dset[data_slices]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File
hon3.8/site-packages/h5py-3.1.0-py3.8-linux-x86_64.egg/h5py/_hl/dataset.py",
line 787, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (src and dest dataspaces have different number of
elements selected)
Traceback (most recent call last):
  File "plot_sphere.py", line 88, in <module>
    post.visit_writes(args['<files>'], main, output=output_path)
  File
"/home/jorafb/dedalus_env/lib/python3.8/site-packages/dedalus-2.1905-py3.8-linux-x86_64.egg/dedalus/
tools/post.py",
line 44, in visit_writes
    function(set_path, start, count, **kw)
  File "plot_sphere.py", line 53, in main
    data = dset[data_slices]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File
hon3.8/site-packages/h5py-3.1.0-py3.8-linux-x86_64.egg/h5py/_hl/dataset.py",
line 787, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (src and dest dataspaces have different number of
elements selected)
Traceback (most recent call last):
  File "plot_sphere.py", line 88, in <module>
    post.visit_writes(args['<files>'], main, output=output_path)
  File
"/home/jorafb/dedalus_env/lib/python3.8/site-packages/dedalus-2.1905-py3.8-linux-x86_64.egg/dedalus/
tools/post.py",
line 44, in visit_writes
    function(set_path, start, count, **kw)
  File "plot_sphere.py", line 53, in main
    data = dset[data_slices]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File
hon3.8/site-packages/h5py-3.1.0-py3.8-linux-x86_64.egg/h5py/_hl/dataset.py",
line 787, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (src and dest dataspaces have different number of
elements selected)
Traceback (most recent call last):
  File "plot_sphere.py", line 88, in <module>
    post.visit_writes(args['<files>'], main, output=output_path)
  File
"/home/jorafb/dedalus_env/lib/python3.8/site-packages/dedalus-2.1905-py3.8-linux-x86_64.egg/dedalus/
tools/post.py",
line 44, in visit_writes
    function(set_path, start, count, **kw)
  File "plot_sphere.py", line 53, in main
    data = dset[data_slices]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File
hon3.8/site-packages/h5py-3.1.0-py3.8-linux-x86_64.egg/h5py/_hl/dataset.py",
line 787, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (src and dest dataspaces have different number of
elements selected)
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus
causing
the job to be terminated. The first process to do so was:

  Process name: [[23543,1],3]
  Exit code:    1

However, if I download the files to my local machine, it works perfectly on my conda installation, so the files are not corrupt. 

has anyone run into the same problem?

Keaton Burns

unread,
Oct 18, 2021, 12:27:34 PM10/18/21
to dedalu...@googlegroups.com
Hi Jose,

Until the beta version of the spherical code is released, we'd prefer to keep discussion of it off the main mailing list. This is because the API is still undergoing rapid changes, so it's very likely that any code (and solutions to problems like this) will not be valid for very long, and will be confusing to people who come across it later.  If you email the developers off-list, though, we'd be happy to help!

Best,
-Keaton
--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/02077ef0-d5de-4e19-aec6-88ddf31f7d78n%40googlegroups.com.

Ben Brown

unread,
Oct 26, 2021, 3:09:08 PM10/26/21
to dedalus-users
Rafa,
       This particular problem is likely related to your stack, rather than to d3 itself.

When we've seen this issue before, it's been related to HDF5 library versions.  In particular, for at least some of the operations I've done, slicing through virtual files works on HDF5 1.12, but fails on HDF5 1.10.

So first thing to do is to install a clean stack that's fully up to date.  We strongly recommend the conda-based installs, based on our experience with NASA Pleiades.
--Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CADZXxBg35pk6w_js2F_uv30zsHOc53B9W4YKKZSQ5bF85ZGHYg%40mail.gmail.com.

Jose Rafael

unread,
Oct 26, 2021, 8:07:59 PM10/26/21
to Dedalus Users

Thanks for your reply Ben! I cannot use conda on Compute Canada machines, but I will ask the support team to update hdf5 to see what happens. 

Best

Jose Rafael

unread,
Oct 30, 2021, 12:08:11 PM10/30/21
to Dedalus Users

thanks Ben!!    For those who use computecanada, use hdf5-mpi/1.12.1 (for d3). It solves the problem!
Reply all
Reply to author
Forward
0 new messages