HDF5-DIAG message in windows threadsafe build

[Joel B. Mohler]

Hi,

This is perhaps related to my other current mail thread (oops, pun) about windows thread issues.  I guess this should be something that will need to be checked into in the process of using --enable-threadsafe on the official h5py build of hdf5 (presuming that happens).

I'm getting an HDF5-DIAG stderr hunk from the following script.  If I replace the 'if True' with 'if False' running the "'xx' in f" conditional in the same thread, I don't get this HDF5-DIAG output.  It seems to me that the HDF5-DIAG is being overzealous here because I'm not actually trying to access the non-existent item.  However, the fact that the appearance of this output depends on the use of threading is quite unnerving to me -- this is built with --enable-threadsafe and there is no real thread concurrency in this code anyhow due to the 'join'.

***
import threading
import h5py

f = h5py.File("mythreadtest.hdf5", "w")

def test(f):
    print 'xx' in f

if True:
    thread = threading.Thread(target=test, args=(f,))
    thread.start()
    thread.join()
else:
    test(f)

f.close()
del f
****

The HDF5-DIAG hunk is:

****
HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 5548:
  #000: ..\..\src\H5Gdeprec.c line 875 in H5Gget_objinfo(): cannot stat object
    major: Invalid arguments to routine
    minor: Unable to initialize object
  #001: ..\..\src\H5Gdeprec.c line 1002 in H5G_get_objinfo(): name doesn't exist
    major: Symbol table
    minor: Object already exists
  #002: ..\..\src\H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #003: ..\..\src\H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #004: ..\..\src\H5Gdeprec.c line 906 in H5G_get_objinfo_cb(): 'xx' doesn't exist
    major: Symbol table
    minor: Object not found
****

Any input on whether this is legitimate or not would be appreciated.  It seems spurious to me and I'd like to silence it.  Searching for this type of DIAG output yields several posts in the HDF5 mailing list, but I'm not sure if they are relevant here.

Joel

[Andrew Collette]

Hi Joel,

> HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 5548:
> #000: ..\..\src\H5Gdeprec.c line 875 in H5Gget_objinfo(): cannot stat
> object
> major: Invalid arguments to routine
> minor: Unable to initialize object
> #001: ..\..\src\H5Gdeprec.c line 1002 in H5G_get_objinfo(): name doesn't
> exist

This is not an error to be concerned about; it's related to how
__contains__ works in the Cython layer. It's considered a bug and a
fix is in the works. There's (still!) no function in the HDF5 API to
simply check for existence of a particular group or dataset, given a
full path. So we run H5Gget_objinfo() and read the error code. H5py
automatically traps the error output for the main thread but there's
no way in HDF5 to silence errors in more than one thread at the same
time.

The correct way to fix this (which I haven't implemented yet) is to
write a custom function to carefully break a path into its component
parts and check them one at a time without triggering errors. It's a
little tricky because of things like external links, the fact that
".." is treated differently in HDF5 and the Python POSIX path tools,
etc.

Andrew

[stuarteberg]

Hi Andrew,

As far as you know, are these (harmless) error messages still shown when using h5py from a (non-main) thread?  Did you ever get the chance to implement the fix you mentioned above?  My users see these error messages, but I'm wondering if upgrading hdf5 or h5py will fix the issue.

Thanks,
Stuart

[Andrew Collette]

Hi Stuart,

Yes, this was fixed in December. Using h5py 2.1.1 or newer should
solve the problem.

Andrew

[stuarteberg]

Hi,

(I'm reviving this old thread about console error output when writing HDF5 from a non-main thread.)

After recently upgrading both hdf5 and h5py, I'm seeing "harmless" error messages popping up in my application, similar to the messages shown earlier in this thread.  (I haven't yet tried a binary search to find out which version of hdf5 and/or h5py introduce the new error messages.)

Is this a new issue, or is this a regression of this previously fixed issue?

See below for an example of the sort of errors I'm seeing.  I'm using h5py 2.5.0 and hdf5 1.8.14 (both installed via conda, if that matters).

Thanks,
Stuart

HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 139784760563456:
  #000: H5Gdeprec.c line 622 in H5Gunlink(): couldn't delete link
    major: Links
    minor: Can't delete message
  #001: H5L.c line 2344 in H5L_delete(): can't unlink object
    major: Symbol table
    minor: Unable to remove object
  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table
    minor: Object not found

  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table
    minor: Callback failed

  #004: H5L.c line 2297 in H5L_delete_cb(): can't delete self
    major: Symbol table
    minor: Can't delete message
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 139784760563456:
  #000: H5Gdeprec.c line 622 in H5Gunlink(): couldn't delete link
    major: Links
    minor: Can't delete message
  #001: H5L.c line 2344 in H5L_delete(): can't unlink object
    major: Symbol table
    minor: Unable to remove object
  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table
    minor: Object not found

  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table
    minor: Callback failed

  #004: H5L.c line 2297 in H5L_delete_cb(): can't delete self
    major: Symbol table
    minor: Can't delete message
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 139784760563456:
  #000: H5Gdeprec.c line 622 in H5Gunlink(): couldn't delete link
    major: Links
    minor: Can't delete message
  #001: H5L.c line 2344 in H5L_delete(): can't unlink object
    major: Symbol table
    minor: Unable to remove object
  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table
    minor: Object not found

  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table
    minor: Callback failed

  #004: H5L.c line 2297 in H5L_delete_cb(): can't delete self
    major: Symbol table
    minor: Can't delete message
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 139784760563456:
  #000: H5Gdeprec.c line 622 in H5Gunlink(): couldn't delete link
    major: Links
    minor: Can't delete message
  #001: H5L.c line 2344 in H5L_delete(): can't unlink object
    major: Symbol table
    minor: Unable to remove object
  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table
    minor: Object not found

  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table
    minor: Callback failed

  #004: H5L.c line 2297 in H5L_delete_cb(): can't delete self
    major: Symbol table
    minor: Can't delete message

[Andrew Collette]

Hi,

> After recently upgrading both hdf5 and h5py, I'm seeing "harmless" error
> messages popping up in my application, similar to the messages shown earlier
> in this thread. (I haven't yet tried a binary search to find out which
> version of hdf5 and/or h5py introduce the new error messages.)
>
> Is this a new issue, or is this a regression of this previously fixed issue?

I'm not aware of regressions. Is your code multithreaded or do you
see this in the main thread?

Andrew

[Stuart Berg]

> Is your code multithreaded or do you see this in the main thread?

My code is multithreaded.  I'll try to come up with a reduced test case and get back to you in a day or two.

-Stuart

[stuarteberg]

Just a quick follow-up to my earlier message: I was wrong when I said my new issue is caused by a recent upgrade of h5py.  It was a change in my own code which introduced these errors.

FWIW, The errors were caused by the following simple (ok, unnecessary) function from my code:

def deleteIfPresent(group, name):
    try:
        del group[name]
    except KeyError:
        pass

Apparently, if 'name' doesn't exist in 'group', hdf5 can sometimes print harmless error messages as shown previously.  Unfortunately, I don't know exactly which circumstances trigger the errors messages.  They aren't easy to reproduce.

For me, the obvious workaround works just fine:

def deleteIfPresent(group, name):
    if name in group:
        del group[name]

Best,
Stuart

[Andrew Collette]

Hi Stuart,

> Apparently, if 'name' doesn't exist in 'group', hdf5 can sometimes print
> harmless error messages as shown previously. Unfortunately, I don't know
> exactly which circumstances trigger the errors messages. They aren't easy
> to reproduce.

Thanks for the update. These messages certainly shouldn't appear, but
it's benign... we no longer use the error-handler machinery in HDF5
for Python exceptions. So it will have no effect on the logic of your
program.

Andrew

[Stuart Berg]

FWIW, the test case shown below produces error output on the console, on Linux at least.  As you can see, I'm using h5py version 2.5 (installed via the conda package manager).

Let me know if you'd like me to open an issue about this in the github issue tracker.

Best,

Stuart

import h5py
import threading

def test():
    with h5py.File('/tmp/newfile.h5', 'w') as newfile:
        try:
            doesnt_exist = newfile['doesnt_exist'].value
        except KeyError:
            pass
    print "test complete."

th = threading.Thread(target=test)
th.start()
th.join()

print "hdf5:", h5py.version.hdf5_version
print "h5py:", h5py.__version__
print "DONE."

CONSOLE OUTPUT:

HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 140017089001216:
  #000: H5O.c line 246 in H5Oopen(): unable to open object
    major: Symbol table
    minor: Can't open object
  #001: H5O.c line 1357 in H5O_open_name(): object not found

    major: Symbol table
    minor: Object not found

  #002: H5Gloc.c line 430 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table
    minor: Object not found

  #004: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table
    minor: Callback failed

  #005: H5Gloc.c line 385 in H5G_loc_find_cb(): object 'doesnt_exist' doesn't exist

    major: Symbol table
    minor: Object not found

test complete.
hdf5: 1.8.14
h5py: 2.5.0
DONE.

--
You received this message because you are subscribed to the Google Groups "h5py" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h5py+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Collette]

> FWIW, the test case shown below produces error output on the console, on
> Linux at least. As you can see, I'm using h5py version 2.5 (installed via
> the conda package manager).
>
> Let me know if you'd like me to open an issue about this in the github issue
> tracker.

Thanks Stuart! Please open an issue, so we don't forget about this.

Andrew

[Stuart Berg]

Done: https://github.com/h5py/h5py/issues/580

Andrew