How to actually use trained model to predict new unlabeled data(non picture) for regression purpose?

[Michael Liu]

I have a normal regression problem which I'm exploring NN method to improve the result. The target is to minimize MSE.

I have split labeled training data for training purpose and select EUCLIDEAN_LOSS.

The basic network is

name: "MSE regression"
layers {
  name: "data"
  type: HDF5_DATA
  top: "data"
  top: "label"
  hdf5_data_param {
    source: "cv_train.txt"
    batch_size: 10
  }
  include: { phase: TRAIN }
}

layers {
  name: "data"
  type: HDF5_DATA
  top: "data"
  top: "label"
  hdf5_data_param {
    source: "cv_test.txt"
    batch_size: 10
  }
  include: { phase: TEST }
}
layers {
  name: "fc1"
  type: INNER_PRODUCT
  bottom: "data"
  top: "fc1"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 40
    weight_filler {
      type: "gaussian"
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "fc1"
  top: "fc1"
}
layers {
  name: "fc2"
  type: INNER_PRODUCT
  bottom: "fc1"
  top: "fc2"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 1
    weight_filler {
      type: "gaussian"
      std: 1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
layers {
  name: "loss"
  type: EUCLIDEAN_LOSS
  bottom: "fc2"
  bottom: "label"
  top: "loss"
}

the training process finishes all right. However, I have no clue how to  apply the trained model to unlabeled data for actual prediction.
Will someone be kindly enough to shed some light on this?

[Arthur B.]

I'm having the exact same problem. Did you figure out a way to do it?

[Bartosz Ludwiczuk]

I encountered same problem. And here is how I resolve it:

1. Change structure of net in .prototxt. 

      - remove last later (EUCLIDEAN_LOSS) 

      - change first layer from HF5 (I used this format of data) to MEMORY_DATA, where I have to define input size

2. My code:

###########################

# Predict using Caffe model

###########################

Make sure that caffe is on the python path:

caffe_root = '/home/user/CODE/caffe/'  # this file is expected to be in {caffe_root}/examples

import sys

sys.path.insert(0, caffe_root + 'python')

import caffe

# Set the right path to your model definition file, pretrained model weights,

# and the image you would like to classify.

MODEL_FILE = '/home/melgor/CODE/caffe/examples/Try/train_val.prototxt'

PRETRAINED = '/home/melgor/CODE/caffe/examples/Try/_iter_45000.caffemodel'

net = caffe.Net (MODEL_FILE,PRETRAINED)

net.set_phase_test()

net.set_mode_cpu()

data4D = np.zeros([100,1,1,3593]) #create 4D array, first value is batch_size, last number of input

data4DL = np.zeros([100,1,1,1])  # need to create 4D array as output, first value is batch_size, last number of output

data4D[0:100,0,0,:] = xtest[0:100,:] # fill value of input

net.set_input_arrays(data4D.astype(np.float32),data4DL.astype(np.float32))

pred = net.forward()

Then variable pred has all prediction score. 

This code predict value, but I am not sure about quality. I did not check it with simple case like there. I will check then, if LinearRegression do same stuff like this code.

Maybe any of the developers could look at this methodology and say if it is correct. 

[Arthur B.]

Do you have a "label" output in the memory data layer?

I get the error
F1007 19:58:37.770376  1271 blob.cpp:14] Check failed: width >= 0 (-7 vs. 0)

And I'm not qure sure what it means

[Arthur B.]

Ah, could it be that the label is expected to be one dimensional? I'm working with an 8 dimensional output.

[Bartosz Ludwiczuk]

My MEMORY_LAYER does not have 'labels', because I do not know it.
Labels was used in loss function which I delete.

About error: code which I deliver is for one output. If you want more, you must chane the dimmension of numpy array, find this in comment.

And remember to set promerties of memory later like batch size and width(width is a number of input in this case)

[Arthur B.]

Thanks, I got it to work.

So.... did you use a split layer to run a conv layer on the spectrum only or did you build a fully connected network :) ?

[Bartosz Ludwiczuk]

Here is my .prototxt file:

 name: "LogisticRegressionNet"
layers {
  name: "data"
  type: MEMORY_DATA
  top: "data"
  top: "label"
  memory_data_param {
    batch_size: 1100 #batch size, so how many prediction youu want to do at once. Best is "1", but higher number get better performance
    channels: 1
    height: 1
    width: 3593 #number of input

  }
}
layers {
  name: "fc1"
  type: INNER_PRODUCT
  bottom: "data"
  top: "fc1"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {

    num_output: 50 #number of input of first layer
    weight_filler {
      type: "gaussian"
      std: 0.01

    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {

  name: "encode1neuron"
  bottom: "fc1"
  top: "encode1neuron"
  type: SIGMOID
}
layers {
  name: "fc3"
  type: INNER_PRODUCT
  bottom: "encode1neuron"
  top: "fc3"

  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {

    num_output: 1 #number of output from regression
    weight_filler {
      type: "gaussian"
      std: 0.01

    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

And here is my training .prototxt

name: "LogisticRegressionNet"

layers {
  name: "data"
  type: HDF5_DATA
  top: "data"
  top: "label"
  hdf5_data_param {

    source: "/home/melgor/CODE/caffe/examples/Ind/soc/hdf5_classification/data/soc_train.txt" #input data in HDF5
    batch_size: 800

  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: HDF5_DATA
  top: "data"
  top: "label"
  hdf5_data_param {

    source: "/home/melgor/CODE/caffe/examples/Ind/soc/hdf5_classification/data/soc_test.txt"
    batch_size: 400

  }
  include: { phase: TEST }
}
layers {
  name: "fc1"
  type: INNER_PRODUCT
  bottom: "data"
  top: "fc1"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {

    num_output: 50 # number of output in first layer
    weight_filler {
      type: "gaussian"
      std: 0.01

    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {

  name: "encode1neuron"
  bottom: "fc1"
  top: "encode1neuron"
  type: SIGMOID
}
layers {
  name: "fc3"
  type: INNER_PRODUCT
  bottom: "encode1neuron"
  top: "fc3"

  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {

    num_output: 1 # number of output from regression task
    weight_filler {
      type: "gaussian"
      std: 0.01

    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}


layers {
  name: "loss"

  type: EUCLIDEAN_LOSS #loos function
  bottom: "fc3"
  bottom: "label"
  top: "loss"
}

So, as I said, only thing which I done is:

- remove layer with HDF5 data (train and test phase)

- add memort_data_layer, where set batch size and number of input 

- delete loss function (I do not have label to compute it)

Doing it and running code:

##########################
# Predict using Caffe model
###########################

# Make sure that caffe is on the python path:
caffe_root = '/home/melgor/CODE/caffe/'  # this file is expected to be in {caffe_root}/examples

import sys
sys.path.insert(0, caffe_root + 'python')


import caffe
# Set the right path to your model definition file, pretrained model weights,
# and the image you would like to classify.

MODEL_FILE = 'p/train_val.prototxt'
PRETRAINED = 'p/_iter_10000.caffemodel'
# from sklearn.feature_selection import RFE

max_value = batch_size from .prototxt
net = caffe.Net (MODEL_FILE,PRETRAINED)
net.set_phase_test()
net.set_mode_cpu()
data4D = np.zeros([max_value,1,1,3593]) #create 4D array, first value is batch_size, last number of inputs
data4DL = np.zeros([max_value,1,1,1])  # need to create 4D array as output, first value is batch_size, last number of outputs
data4D[0:max_value,0,0,:] = xtrain[0:max_value,:] # fill value of input xtrain is your value which you would like to predict
print [(k, v[0].data.shape) for k, v in net.params.items()]
net.set_input_arrays(data4D.astype(np.float32),data4DL.astype(np.float32))
pred = net.forward()
pred_normal = np.zeros([max_value,5])
for i in range(0,max_value):
 pred_normal[i,0] = pred['fc3'][i][0]


#plot result
import matplotlib.pyplot as plt
time = [ i for i in range(0,max_value)]
plt.plot(time, ytrain, 'b',label='Real')
plt.plot(time, pred_normal[:,0], 'r',label='Predict')
plt.grid(True)
plt.legend()
plt.show()

This is everything what I done.

So, I do not add any of Spit layer and Fully-connected.

Only staff which I add in this answer.

Hopes that help:)

[Michael Liu]

Much much thanks, Bartosz， for sharing such detailed experience!!

[Mender]

Hi, Bartosz! 

I am a bit confused that the MEMORY_DATA Layer only specifies the size of test data, but how can caffe know where to load these data?

Another question： MODEL_FILE = 'p/train_val.prototxt' and training.prototxt is the same one?

[Soda Heng]

Has anyone come up with a more official way of predicting HDF5 inputs?

I used the method mentioned here and it works as long as all the data can fit into one batch. If i break it up into different batches, the prediction becomes completely random.

Would be great if anyone can explain why it would do this. My script below:

with h5py.File(DATA_NAME, 'r') as f:
    h5Data = f['data'][()]
    h5Img = f['img'][()]

data4D = numpy.zeros([BATCH_SIZE,1,40,60])
resultsArray = []

for i in xrange(0, DATA_SIZE, BATCH_SIZE):
    stop = i + BATCH_SIZE
    #print('Loading data from {0} to {1}'.format(i, stop))
    data4D = h5Data[i:stop]
    if (data4D.shape[0] < BATCH_SIZE):
        rows = data4D.shape[0]
        extraRows = numpy.zeros([BATCH_SIZE, 1, 40, 60])
        extraRows[:rows] = data4D
        data4D = extraRows

    data4DLabels = numpy.zeros([BATCH_SIZE,1,1,1])
    net.set_input_arrays(data4D.astype(numpy.float32), data4DLabels.astype(numpy.float32))
    prediction = net.forward()
    if (i == 0):
        resultsArray = prediction['ip2']
    else:
        resultsArray = numpy.concatenate((resultsArray, prediction['ip2']), axis=0)

# Get the results
predictionArray = numpy.zeros(DATA_SIZE)
for i in xrange(0, DATA_SIZE):
    predictionArray[i] = resultsArray[i].argmax()

itemIndex = numpy.where(predictionArray == 1)
itemIndex = numpy.asarray(itemIndex)
print("Indexes: {0} - {1}".format(itemIndex.shape, itemIndex))

[Tao Liu]

Hi Mender,
    I got the same confusion. Did you get a solution?

[Sathisha Basavaraju]

Hi Bartosz Ludwiczuk,

I have one small doubt, In the lines :

data4D = np.zeros([max_value,1,1,3593]) #create 4D array, first value is batch_size, last number of inputs
data4DL = np.zeros([max_value,1,1,1])  # need to create 4D array as output, first value is batch_size, last number of outputs




we have created 4 dimension arrays, first parameter understood and what about meaning of each of the other three parameters?



[biped...@gmail.com]

It is strange that MemoryData layer must have a top named 'label' though it seems meaningless, isn't it?

在 2014年10月9日星期四 UTC+8上午4:23:54，Bartosz Ludwiczuk写道：