How to save data of 'list' type on HDF5 files?

[Jinfang Wang]

Hi,

I would like to write data of 'list' type on HDF5 files, but an error appears as follows. From the documents I found, they says that only writing 'numpy.ndarray' data is feasible. Could you tell me how to save data of 'list' type on HDF5 files?

Thank you.

Jinfang

--------------------------------------------------------------------------------------------------

bob.bio.base@2017-08-17 15:58:23,162 -- ERROR: During the execution, an exception was raised: error setting object `array' of HDF5 file `/media/sf_winshare/features1.hdf5': no support for storing objects of type `list' on HDF5 files

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 902, in debugfile

    debugger.run("runfile(%r, args=%r, wdir=%r)" % (filename, args, wdir))

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/bdb.py", line 400, in run

    exec cmd in globals, locals

  File "<string>", line 1, in <module>

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 880, in runfile

    execfile(filename, namespace)

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile

    builtins.execfile(filename, *where)

  File "/home/jfw/anaconda2/envs/bob_py27/bin/verify.py", line 6, in <module>

    sys.exit(bob.bio.base.script.verify.main())

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/script/verify.py", line 454, in main

    verify(cml_args, command_line_parameters)

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/script/verify.py", line 434, in verify

    if not execute(cml_args):

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/script/verify.py", line 290, in execute

    force = cml_args.force)

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/tools/algorithm.py", line 53, in train_projector

    extractor.write_feature(train_features, outfeature_file)    

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/extractor/Extractor.py", line 111, in write_feature

    utils.save(feature, feature_file)

  File "/home/jfw/anaconda2/envs/bob_py27/lib/python2.7/site-packages/bob/bio/base/utils/io.py", line 97, in save

    f.set("array", data, compression=compression)

TypeError: error setting object `array' of HDF5 file `/media/sf_winshare/features1.hdf5': no support for storing objects of type `list' on HDF5 files

[André Anjos]

Hello,

A list is a (potentially) heterogeneous type in Python. IOW, a list can be composed of objects of different types. That does not directly match HDF5's intent, which is to store arrays composed of data of the same type. In order to save an homogeneous list (say composed only of floats) to an HDF5 file, first convert it to a numpy array. The standard way to do it is to use the function "numpy.array", which takes whatever and returns an homogeneous array. Example:

>>> l = [1.5, 2.5]

>>> a = numpy.array(l) #now a is an homogeneous array of numpy.float64 objects

>>> # call bob's function using a instead of l

FYI, we don't have an implicit conversion implemented on purpose. In this case, it could destructively change the types of input objects and lead to unexpected effects. So, we prefer explicit instead of implicit.

Best, Andre

--
-- You received this message because you are subscribed to the Google Groups bob-devel group. To post to this group, send email to bob-...@googlegroups.com. To unsubscribe from this group, send email to bob-devel+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/bob-devel or directly the project website at http://idiap.github.com/bob/
---
You received this message because you are subscribed to the Google Groups "bob-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bob-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Dr. André Anjos
Idiap Research Institute
Centre du Parc - rue Marconi 19
CH-1920 Martigny, Suisse
Phone: +41 27 721 7763
Fax: +41 27 721 7712
http://andreanjos.org

[Manuel Günther]

Dear Jinfang, Andre,

first of all, Andre, I am sorry to contradict you, but automatic conversion to a numpy.ndarray works just fine (at least with Python 2.7, but I have no reason to believe that Python 3 would make a difference):

>>> import bob.io.base

>>> h = bob.io.base.HDF5File("test.hdf5", "w")

>>> h.set("x", [0., 1., 1.5])

>>> h.get("x")

array([ 0. ,  1. ,  1.5])

>>> h.get("x").dtype

dtype('float64')

as long as the list can be converted to a numpy.ndarray. What I believe happens to Jinfang is that his list cannot automatically be converted into a numpy.ndarray. 

As Andre pointed out, HDF5 by default does not support storing lists of various types. However, it is possible to store the contents of the list -- given that each element of it can be written to HDF5:

data = ["x", 1., numpy.ones((2,4))]

for index, element in enumerate(data):

  h.set("Element_%d"%index, element)

Reading back the list from HDF5 is as simple:

data = []
index = 0
while h.has_key("Element_%d"%index):
  data.append(h.get("Element_%d"%index))
  index += 1

There is one limitation: HDF5 will not store None values.

I hope this helps

Manuel