Reading HDF5 input

[lavaniac]

Hi,

I am having some trouble loading HDF5 data into my caffe model. I have a text file which has the full path for the data file (hdf5) and label file (hdf5) in seperate lines. When I try to train I get the following errror :

I1128 16:06:00.405066  4322 net.cpp:67] Creating Layer data

I1128 16:06:00.405083  4322 net.cpp:356] data -> data

I1128 16:06:00.405114  4322 net.cpp:356] data -> label

I1128 16:06:00.405131  4322 net.cpp:96] Setting up data

I1128 16:06:00.405143  4322 hdf5_data_layer.cpp:57] Loading filename from /s0/train.txt

I1128 16:06:00.405292  4322 hdf5_data_layer.cpp:69] Number of files: 2

I1128 16:06:00.405308  4322 hdf5_data_layer.cpp:29] Loading HDF5 file/s0/train_data.hdf5

HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:

  #000: H5D.c line 358 in H5Dopen2(): not found

    major: Dataset

    minor: Object not found

  #001: H5Gloc.c line 430 in H5G_loc_find(): can't find object

    major: Symbol table

    minor: Object not found

  #002: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

    major: Symbol table

    minor: Object not found

  #003: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

    major: Symbol table

    minor: Callback failed

  #004: H5Gloc.c line 385 in H5G_loc_find_cb(): object 'data' doesn't exist

    major: Symbol table

    minor: Object not found

F1128 16:06:00.417641  4322 io.cpp:131] Check failed: status >= 0 (-1 vs. 0) Failed to get dataset ndims for data

*** Check failure stack trace: ***

    @     0x7f8f5c20f694  google::LogMessage::Fail()

    @     0x7f8f5c20f5f4  google::LogMessage::SendToLog()

    @     0x7f8f5c20eff4  google::LogMessage::Flush()

    @     0x7f8f5c211d79  google::LogMessageFatal::~LogMessageFatal()

    @           0x4a8b5b  caffe::hdf5_load_nd_dataset_helper<>()

    @           0x4a8270  caffe::hdf5_load_nd_dataset<>()

    @           0x54ed12  caffe::HDF5DataLayer<>::LoadHDF5FileData()

    @           0x54f973  caffe::HDF5DataLayer<>::LayerSetUp()

    @           0x47f493  caffe::Net<>::Init()

    @           0x4818ed  caffe::Net<>::Net()

    @           0x4621b0  caffe::Solver<>::InitTrainNet()

    @           0x467ea1  caffe::Solver<>::Init()

    @           0x4682e5  caffe::Solver<>::Solver()

    @           0x42a110  caffe::GetSolver<>()

    @           0x425c46  train()

    @           0x429b1b  main

    @       0x3d3961ed5d  (unknown)

    @           0x423119  (unknown)

Now I was able to read the /s0/train_data.hdf5 file using hp5y python package. Does caffe need the names of the objects as 'data' and 'label' ?

[Leon]

What did you change in the python script ? you shouldn't need to change anything other than the SGD part of the python script, note that all you need to do is comment out the SGD part (assuming you don't need it), and dump your own data in numpy.array format (mind the dimension). the script should handle the rest. 

I am working on HDF5 as well, we should be BFF. 

[Armin Kappeler]

Maybe you found out by yourself by now, but if anyone has the same thing (like me), this is what I found;

-> your HDF5 file is probably too large. make sure it does not go over 2GB. If it does, split it into several files, each <=2GB.

See this one here, same thing

 https://groups.google.com/forum/#!searchin/caffe-users/hdf5/caffe-users/Jm4hIFoA-9A/jcVs2khqKlUJ