name in h5 is True, but h5[name] is KeyError

[Brian Jackel]

This idiom

  group = h5[name] if name in h5 else h5.create_group(name)

is giving me an unexpected result (see below).  As far as I can tell

  name in h5 == True

but h5[name] is an error.

What am I doing wrong?

ps, simply using 

 h5.require_group(name) 

also fails (see below).

================================================================

#open example file

h5 = h5py.File(r'c:\data\tmp\h+storm\level1\2015\07\28\20150728__atha_feso-01_level1.hd5')

#look at top branch, wander down a bit

h5['/level1']

Out[6]: <HDF5 group "/level1" (7 members)>

h5['/level1'].keys()

Out[7]: [u'channel', u'dark', u'data', u'ephemeris', u'file', u'step', u'time']

h5['/level1/step'].keys()

Out[8]: [u'aim', u'map', u'table']

h5['/level1/time'].keys()

Out[10]: [u'scan', u'step']

'/level1/time/step' in h5

Out[13]: True

# this is expected

h5['/level1/time/step']

Out[15]: <HDF5 dataset "step": shape (498, 5440), type "|V8">

'/level1/time/scan' in h5

Out[14]: True

# this is unexpected

h5['/level1/time/scan']

---------------------------------------------------------------------------

KeyError Traceback (most recent call last)

<ipython-input-16-a0bb9280b612> in <module>()

----> 1 h5['/level1/time/scan']

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2463)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2420)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_hl\group.pyc in __getitem__(self, name)

162 raise ValueError("Invalid HDF5 object reference")

163 else:

--> 164 oid = h5o.open(self.id, self._e(name), lapl=self._lapl)

165

166 otype = h5i.get_type(oid)

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2463)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2420)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\h5o.pyd in h5py.h5o.open (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\h5o.c:3373)()

KeyError: 'Unable to open object (Component not found)'

==========================================

In [20]: h5.require_group('/level1/time/scan')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-20-61df89f144c1> in <module>()
----> 1 h5.require_group('/level1/time/scan')

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_hl\group.pyc in require_group(self, name)
    148             if not name in self:
    149                 return self.create_group(name)
--> 150             grp = self[name]
    151             if not isinstance(grp, Group):
    152                 raise TypeError("Incompatible object (%s) already exists" % grp.__class__.__name__)

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2463)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2420)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_hl\group.pyc in __getitem__(self, name)
    162                 raise ValueError("Invalid HDF5 object reference")
    163         else:
--> 164             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    165
    166         otype = h5i.get_type(oid)

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2463)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\_objects.pyd in h5py._objects.with_phil.wrapper (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\_objects.c:2420)()

C:\Users\bjackel\AppData\Local\Enthought\Canopy\User\lib\site-packages\h5py\h5o.pyd in h5py.h5o.open (C:\pisi\tmp\h5py-2.5.0-3\work\h5py-2.5.0\h5py\h5o.c:3373)()

KeyError: 'Unable to open object (Component not found)'

[Andrew Collette]

Hi Brian,

> # this is unexpected
>
> h5['/level1/time/scan']

> KeyError: 'Unable to open object (Component not found)'

It looks like there is an object at "/level1/time/scan", but HDF5
can't open it for some reason. Maybe a broken external link?

You could try using h5ls:

$ h5ls myfile.hdf5 /path/to/object

or h5dump to see what's in there.

Andrew

[Brian Jackel]

Hi Andrew

Yes, that was it: a broken external link.

Thanks very much for pointing me in the right direction.

And more general thanks to everyone for making HDF5 such a useful tool for data analysis in Python.

Brian