HDF5 swmr feature

[Vina C.]

Hello, everyone:
I was trying to use go HDF5 package and noticed that it only support version 1.8 for HDF5. Since SWMR feature (Single Write multiple read) only implemented in 1.10, I wonder if there is anyway to do the same task with current go HDF5 package? Any example would be greatly appreciated. In addition, anyone knows that if there is any intention to upgrade go HDF5 to match the latest HDF5 C version? Thanks

[Sebastien Binet]

hi,

the issue of the (minimal) HDF5 version to support has been briefly
touched upon there:
- https://github.com/gonum/hdf5/pull/78#discussion_r604723126

AFAIK, there isn't a consensus yet as to what's the best course of
action.

vendoring a given HDF5 gives more control to gonum/hdf5, but it's also
more work (to build and to maintain).

but it may well be the best option.
(that's what e.g. mattn/go-sqlite3 does)

thoughts?

-s

[Vina C.]

My personal thoughts would be follow the latest 1.X releases of C-HDF5. Thanks.

[Sebastien Binet]

I've dedicated a few cycles to see what piggy-backing on what
mattn/go-sqlite3 would mean applied to gonum/hdf5.

well, it doesn't look like a super easy job.
compared to sqlite3, hdf5 relies on libz, libsz and libaec (in addition
to libm, libc and libdl - sqlite3 also links against those), so the
configuration step is more involved than what go-sqlite can go away with.

I am not saying it isn't doable, but it's definitely more work than I
can pour into at the moment.

perhaps a more adiabatic-friendly way would be to add new gonum/hdf5 API
relying on these new features protected by Go build tags (by HDF5
version and/or by feature name).

//go:build hdf5_1_12 && gonum_hdf5_enable_swmr

-s

[Vina C.]

then just a further question. Is there any example that can do the similar task like SWMR with current go HDF5 package? Thanks

[Sebastien Binet]

hi,

On Mon Jun 21, 2021 at 15:47 CET, 'Vina C.' via gonum-dev wrote:
> then just a further question. Is there any example that can do the
> similar
> task like SWMR with current go HDF5 package? Thanks

I believe trying to do it "in Go" (so, from outside the HDF5 library, as
a simple user of that library), might get tricky, as HDF5 has its own
ref-counting/lock-ing mechanism for various HDF5 objects.
although, if you have control on the program that writes and reads data
off HDF5 storage, keeping everything managed by channels+goroutines
might cut it.
"it depends"

another avenue would be to create a PR against gonum/hdf5 where one
would wrap the swmr API in a new file, say h5_swmr.go, with an ad hoc
build guard stanza.
e.g.

//+build hdf5_1_10

package hdf5

// ...

so one could use that new API provided one builds gonum/hdf5 with:

$> cd gonum.org/v1/hdf5
$> go build -tags=hdf5_1_10 .

hth,
-s

> --
> You received this message because you are subscribed to the Google
> Groups "gonum-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gonum-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gonum-dev/443b8551-73df-446a-9877-78fb0c51e3f5n%40googlegroups.com.

[Andrey Kolkov]

Hi,

Quick update on the HDF5 ecosystem in Go:

While gonum/hdf5 is still limited to HDF5 1.8.x due to CGo wrapper constraints, there's now an alternative that might interest you: a pure Go HDF5 implementation that doesn't depend on the C library version at all.


CURRENT STATUS (2025):

Repository: https://github.com/scigolib/hdf5

Read support: Feature-complete (all HDF5 formats)
Write support: Beta (v0.11.1-beta)
SWMR: Planned for v0.12.0-RC (Q1 2026)


WHY THIS MATTERS FOR YOUR USE CASE:

1. No CGo = no C library version constraints
2. Pure Go implementation = format support independent of libhdf5
3. SWMR is a format-level feature, not a library dependency


CURRENT LIMITATIONS:

SWMR is not yet implemented in the pure-Go library. However, since it's implemented at the format level (file locking + metadata versioning), it's feasible to add without C library dependencies.

The roadmap shows SWMR planned for v0.12.0-RC, which should be Q1 2026.


WORKAROUND FOR NOW:

If you need SWMR today and can't wait:

Option 1: Use gonum/hdf5 with HDF5 1.10 C library
- Requires building HDF5 1.10 from source
- May need to modify gonum/hdf5 build flags

Option 2: Wait for pure-Go SWMR support (~3-4 months)
- No CGo complications
- Format-level implementation


REGARDING GONUM/HDF5 UPDATE TO 1.10:

That's a question for the gonum maintainers, but CGo wrappers typically lag behind C library releases. The pure-Go approach bypasses this issue entirely since format support isn't tied to library versions.


Installation (if you want to try the pure-Go approach):

go get github.com/scigolib/hd...@v0.11.1-beta


Hope this helps. If SWMR is critical for your workflow and you'd like to contribute or test early SWMR support in the pure-Go library, the project is actively developed and welcoming contributors.

Best,
Andrey Kolkov


Links:
- Pure Go HDF5: https://github.com/scigolib/hdf5
- Roadmap (SWMR timeline): https://github.com/scigolib/hdf5/blob/main/ROADMAP.md
- HDF Group recognition: https://forum.hdfgroup.org/t/loking-for-an-hdf5-version-compatible-with-go1-9-2/10021/7