Neuroscience alert: Implementing parallel hdf5 for h5py on Windows

14 views
Skip to first unread message

Michael Hadler

unread,
Apr 15, 2020, 3:44:23 AM4/15/20
to h5py
Dear all,

I am a researcher working with Multi-Electrode Array recordings stored in hdf5-files and analyzed with a Python-based software SpyKING CIRCUS (Read the Docs - SpyKING CIRCUS). In principal, everything is functional and running, yet the recording files are rather large (>10 GB) and require alot of memory / run time. So I wanted to implement parallel HDF5 as suggested by the documentation from both SpyKING CIRCUS (here) and h5py (here). However, I have only found instructions for UNIX-based systems (as far as I can tell) and was wondering whether it is at all possible to implement on Windows?

Windows 10
MS MPI installed and added to system path
Python 3.6.10 in Anaconda 1.9.12 [MSC v. 1916 64 bit]
-mpi4py 2.0.0 installed
-h5py 2.10.0 installed
Parallel HDF5 build with CMake 3.17.1 [MSC v. 1916]

I am happy for any pointers in the right direction :)

Best,

Michael

Thomas Kluyver

unread,
Apr 15, 2020, 4:20:56 AM4/15/20
to h5...@googlegroups.com
Hi Michael,

If your analysis only needs to read the recordings in parallel, it should be possible without parallel HDF5 - several workers can open the same file and read data from it simultaneously, with no special precautions. As I understand it, parallel HDF5 is only needed if several processes need to write to the same file. I've tried to make this clearer in the latest version of the h5py docs (https://docs.h5py.org/en/latest/mpi.html ).

I've no useful information if you do need parallel HDF5 on Windows.

Best wishes,
Thomas

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/ff78ed34-f63a-4ed5-9d1e-30253168774f%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages