Neuroscience alert: Implementing parallel hdf5 for h5py on Windows

[Michael Hadler]

Dear all,

I am a researcher working with Multi-Electrode Array recordings stored in hdf5-files and analyzed with a Python-based software SpyKING CIRCUS (Read the Docs - SpyKING CIRCUS). In principal, everything is functional and running, yet the recording files are rather large (>10 GB) and require alot of memory / run time. So I wanted to implement parallel HDF5 as suggested by the documentation from both SpyKING CIRCUS (here) and h5py (here). However, I have only found instructions for UNIX-based systems (as far as I can tell) and was wondering whether it is at all possible to implement on Windows?

Windows 10

MS MPI installed and added to system path

Python 3.6.10 in Anaconda 1.9.12 [MSC v. 1916 64 bit]

-mpi4py 2.0.0 installed

-h5py 2.10.0 installed

Parallel HDF5 build with CMake 3.17.1 [MSC v. 1916]

I am happy for any pointers in the right direction :)

Best,

Michael

[Thomas Kluyver]

Hi Michael,

If your analysis only needs to read the recordings in parallel, it should be possible without parallel HDF5 - several workers can open the same file and read data from it simultaneously, with no special precautions. As I understand it, parallel HDF5 is only needed if several processes need to write to the same file. I've tried to make this clearer in the latest version of the h5py docs (https://docs.h5py.org/en/latest/mpi.html ).

I've no useful information if you do need parallel HDF5 on Windows.

Best wishes,

Thomas

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/h5py/ff78ed34-f63a-4ed5-9d1e-30253168774f%40googlegroups.com.