Loading HDF5 data into caffe.io.array_to_datum

[AlexK]

Hi I finally have figured out how to train-test models in Caffe using my own non-image data.

Now I am trying to use pretrained models to make predictions for new unlabelled data.

I have followed the available examples and I am stuck at loading hdf5 file into caffe.io.array_to_datum:

Could show me any example of loading vectors into caffe.io.array_to_datums because I getting the error:

Incorrect array shape.

My data have the input_dim: [1,1024,1,1]

Any suggestion or hint would greatly appreciated.

 

[Ashwani Rai]

def array_to_datum(arr, label=0):
	"""Converts a 3-dimensional array to datum. If the array has dtype uint8,
	the output data will be encoded as a string. Otherwise, the output data
	will be stored in float format.

[Ashwani Rai]

I think the problem is that you are passing in 4 dimensional array when it is expecting 3 dimensional 

[Raffael]

As Ashwani already suggested - you have to unwrap the first dimension. If arr is the original array then arr[0] should work.

[AlexK]

Thanks for your replies.

I have changed my code now:

Which looks like:

def runModelPython(ifile):
    MODEL_FILE = '/projects/huanlab/AlexKoutsoukas_Folder/Caffe_DeepLearning/Exp/TwoClassNet_A.prototxt'
    PRETRAINED = '/projects/huanlab/AlexKoutsoukas_Folder/Caffe_DeepLearning/Exp/test__iter_10000'
    h5Data,Lbls = LoadFromHDF5(ifile)
    max_value = 644
    caffe_data = h5Data.reshape(max_value,1024,1,1)
    data4D = np.zeros([max_value,1024,1,1])
    data4DL = np.zeros([max_value,1,1,1])
   
   
    net = caffe.Net (MODEL_FILE,PRETRAINED)
    net.set_phase_test()
    net.set_mode_gpu()
    net.set_input_arrays(caffe_data.astype(np.float32),data4DL.astype(np.float32))
    prediction = net.forward()

But I am getting this segmentation fault when trying to net.forward().

Any idea why is this happening?

I0510 19:39:24.472733 14154 net.cpp:66] Creating Layer data
I0510 19:39:24.472743 14154 net.cpp:290] data -> data
I0510 19:39:24.472756 14154 net.cpp:290] data -> label
I0510 19:39:24.472772 14154 net.cpp:83] Top shape: 644 1024 1 1 (659456)
I0510 19:39:24.472780 14154 net.cpp:83] Top shape: 644 1 1 1 (644)
I0510 19:39:24.472785 14154 net.cpp:130] data does not need backward computation.
I0510 19:39:24.472792 14154 net.cpp:66] Creating Layer ip1
I0510 19:39:24.472797 14154 net.cpp:329] ip1 <- data
I0510 19:39:24.472805 14154 net.cpp:290] ip1 -> ip1
I0510 19:39:24.483449 14154 net.cpp:83] Top shape: 644 1024 1 1 (659456)
I0510 19:39:24.483474 14154 net.cpp:125] ip1 needs backward computation.
I0510 19:39:24.483484 14154 net.cpp:66] Creating Layer relu1
I0510 19:39:24.483490 14154 net.cpp:329] relu1 <- ip1
I0510 19:39:24.483497 14154 net.cpp:280] relu1 -> ip1 (in-place)
I0510 19:39:24.483510 14154 net.cpp:83] Top shape: 644 1024 1 1 (659456)
I0510 19:39:24.483515 14154 net.cpp:125] relu1 needs backward computation.
I0510 19:39:24.483522 14154 net.cpp:66] Creating Layer ip2
I0510 19:39:24.483526 14154 net.cpp:329] ip2 <- ip1
I0510 19:39:24.483533 14154 net.cpp:290] ip2 -> ip2
I0510 19:39:24.485532 14154 net.cpp:83] Top shape: 644 200 1 1 (128800)
I0510 19:39:24.485548 14154 net.cpp:125] ip2 needs backward computation.
I0510 19:39:24.485554 14154 net.cpp:66] Creating Layer relu2
I0510 19:39:24.485559 14154 net.cpp:329] relu2 <- ip2
I0510 19:39:24.485566 14154 net.cpp:280] relu2 -> ip2 (in-place)
I0510 19:39:24.485574 14154 net.cpp:83] Top shape: 644 200 1 1 (128800)
I0510 19:39:24.485579 14154 net.cpp:125] relu2 needs backward computation.
I0510 19:39:24.485585 14154 net.cpp:66] Creating Layer ip3
I0510 19:39:24.485589 14154 net.cpp:329] ip3 <- ip2
I0510 19:39:24.485596 14154 net.cpp:290] ip3 -> ip3
I0510 19:39:24.485995 14154 net.cpp:83] Top shape: 644 200 1 1 (128800)
I0510 19:39:24.486007 14154 net.cpp:125] ip3 needs backward computation.
I0510 19:39:24.486014 14154 net.cpp:66] Creating Layer relu3
I0510 19:39:24.486018 14154 net.cpp:329] relu3 <- ip3
I0510 19:39:24.486026 14154 net.cpp:280] relu3 -> ip3 (in-place)
I0510 19:39:24.486032 14154 net.cpp:83] Top shape: 644 200 1 1 (128800)
I0510 19:39:24.486037 14154 net.cpp:125] relu3 needs backward computation.
I0510 19:39:24.486043 14154 net.cpp:66] Creating Layer ip4
I0510 19:39:24.486047 14154 net.cpp:329] ip4 <- ip3
I0510 19:39:24.486054 14154 net.cpp:290] ip4 -> ip4
I0510 19:39:24.486070 14154 net.cpp:83] Top shape: 644 2 1 1 (1288)
I0510 19:39:24.486078 14154 net.cpp:125] ip4 needs backward computation.
I0510 19:39:24.486085 14154 net.cpp:66] Creating Layer prob
I0510 19:39:24.486090 14154 net.cpp:329] prob <- ip4
I0510 19:39:24.486111 14154 net.cpp:290] prob -> prob
I0510 19:39:24.486129 14154 net.cpp:83] Top shape: 1 1 1 1 (1)
I0510 19:39:24.486135 14154 net.cpp:125] prob needs backward computation.
I0510 19:39:24.486140 14154 net.cpp:156] This network produces output label
I0510 19:39:24.486145 14154 net.cpp:156] This network produces output prob
I0510 19:39:24.486156 14154 net.cpp:402] Collecting Learning Rate and Weight Decay.
I0510 19:39:24.486163 14154 net.cpp:167] Network initialization done.
I0510 19:39:24.486168 14154 net.cpp:168] Memory required for data: 0
Segmentation fault

[Ashwani Rai]

Sorry I am not sure of the new error. However, perhaps you could help me.

I finished training a model in Caffe using my own non-image data as well.

However I am not sure how to go about using the model to make predictions for new unlabeled data.

Could you share the file TwoClassNet_A.prototxt file with me please?

Thanks

[Ashwani Rai]

    net = caffe.Net (MODEL_FILE,PRETRAINED)
    net.set_phase_test()
    net.set_mode_gpu()
    net.set_input_arrays(caffe_data.astype(np.float32),data4DL.astype(np.float32))
    prediction = net.forward()

May I also check with you how you how you called

set_mood_gpu

and

set_phase_test

from caffe.Net?

>>> import caffe
>>> dir(caffe)
['Classifier', 'Detector', 'Net', 'SGDSolver', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '_caffe', 'classifier', 'detector', 'io', 'proto', 'pycaffe', 'set_device', 'set_mode_cpu', 'set_mode_gpu', 'set_phase_test', 'set_phase_train']
>>> dir(caffe.Net)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__instance_size__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_backward', '_batch', '_blob_names', '_blobs', '_forward', '_layer_names', '_set_input_arrays', 'backward', 'blobs', 'channel_swap', 'deprocess', 'forward', 'forward_all', 'forward_backward_all', 'input_scale', 'inputs', 'layers', 'mean', 'outputs', 'params', 'preprocess', 'raw_scale', 'reshape', 'save', 'set_channel_swap', 'set_input_arrays', 'set_input_scale', 'set_mean', 'set_raw_scale']

...

[Ashwani Rai]

It appears you are passing in an empty array. 

    caffe_data = h5Data.reshape(max_value,1024,1,1)
    data4D = np.zeros([max_value,1024,1,1])
    data4DL = np.zeros([max_value,1,1,1])
   
   
    net = caffe.Net (MODEL_FILE,PRETRAINED)
    net.set_phase_test()
    net.set_mode_gpu()
    net.set_input_arrays(caffe_data.astype(np.float32),data4DL.astype(np.float32))
    prediction = net.forward()

Do you mean to use

caffe_data

instead?

On Monday, 11 May 2015 08:42:37 UTC+8, AlexK wrote: