Multi-task learning Input

[James Guo]

Dear All,

I'm currently trying to implement a Multi-task Learning network with Caffe and encountered a problem recently. Say I have two datasets and each data set has the same image data but different 5-element-vectors as label. First I try HDF5 data layer since it could use vector as labels and define data layers as follows:

layers {

  name: "data1"

  type: HDF5_DATA

  top: "data1"

  top: "label1"

  hdf5_data_param {

    source: "/Users/JamesGuo/Documents/MasterThesis/Multi_Task_db/mtl_data1_list.txt"

    batch_size: 200

  }

  transform_param {

    scale: 0.00390625

  }

  include: { phase: TRAIN }

}

layers {

  name: "data2"

  type: HDF5_DATA

  top: "data2"

  top: "label2"

  hdf5_data_param {

    source: "/Users/JamesGuo/Documents/MasterThesis/Multi_Task_db/mtl_data2_list.txt"

    batch_size: 200

  }

  transform_param {

    scale: 0.00390625

  }

  include: { phase: TRAIN }

}

Then I only process "data1" and compute loss with "label1" and "label2", but the problem is that Caffe will automatically identify "data2" as output and report a very long results, which is not what I expect. However, I checked the source code and found that HDF5 data layer must give two top blobs otherwise the model check will fail. Therefore I would like to use lmdb data since the base data layer could have only one top blob. In this way, I will have three data layers and one layer for image data and two layers for label1 and label2. My question is how can I convert a vector into lmdb? Could you provide any examples or link to the tutorial?  Thank you very much!

James

[James Guo]

I found that there is a slice layer in Caffe maybe helpful. I could concatenate all labels together as a long vector and pass it by HDF5 layer, then inside the network, I use slice layer to parse this vector as different labels. Hope it will work and I will post my test results. 

[James Guo]

The slice layer works, now I could train my multi-task regression.

On Tuesday, 4 November 2014 16:55:55 UTC+1, James Guo wrote:

[ayst]

Hi, Can I ask the question how to obtain the source file for HDF5 layer from image ?    I don't know how to store image and label data in txt file. 

Thanks.

[James Guo]

Check here /caffe-dev/src/caffe/test/test_data generate_sample_data.py

[Soda Heng]

Hey James,

Could you share how you were able to slice up the label vectors?

I'm trying to do something similar where my labels are a 3 dimensional vector with numbers [0-9] each. Do you know if I can slice up the labels and feed them individually into 3 separate accuracy and loss layers?

Thanks!

[James Guo]

Yes, you could. In your case, you just need two slice layer to separate 3 labels. You could check example in http://caffe.berkeleyvision.org/gathered/examples/siamese.html

Good luck!

[echoaimaomao]

Hi, James Guo . I have a question about using hdf5 data . My hdf5 data was generated by matlab ,this is the data information.


h5disp('test.h5')
HDF5 test.h5
Group '/'
    Dataset 'data'
        Size:  20x20x1x500
        MaxSize:  20x20x1x500
        Datatype:   H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  0.000000
    Dataset 'label'
        Size:  1x500
        MaxSize:  1x500
        Datatype:   H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  0.000000
>> h5disp('train.h5')
HDF5 train.h5
Group '/'
    Dataset 'data'
        Size:  20x20x1x2000
        MaxSize:  20x20x1x2000
        Datatype:   H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  0.000000
    Dataset 'label'
        Size:  1x2000
        MaxSize:  1x2000
        Datatype:   H5T_IEEE_F64LE (double)
        ChunkSize:  []
        Filters:  none
        FillValue:  0.000000
>> Is there any problem about it ? The  leveldb  format of my data which can reach  satiesfying results whereas the hdf5 format could learn nothing.
Could you do me a favor to show your hdf5 data in matlab? I am not sure about the data type of label and not familiar with Python language.Thank you very much.

[James Guo]

Sorry echoaimaomao, I used Python to generate HDf5 data, so I'm not sure how things went wrong with your data. For Python script, it's quite similar to Matlab and this example might be helpful https://github.com/BVLC/caffe/blob/master/src/caffe/test/test_data/generate_sample_data.py

Cheers

[Soda Heng]

Thanks for the reply James.

I was able to use the slice layer to successfully get the training started by splitting the labels vector and feeding them into different accuracy and softmax without using siamese. My only problem is now the accuracy layer calculation is not independent of eachother. I am testing on labels with vector of 2 dimensions with digits 0-9. The total of both accuracy layers add up to 1 but never more than 1. Somethnig like accuracy1: 0.68 and accuracy2: 0.32 . Would this be a problem with the accuracy layer or the back propagation you think?

My model is like this:

[data]
[slice -> labels1 and labels2]
[convolution] x 5
[pool] x 5
....
[inner product]
[accuracy layer1 feeding in innner product and labels1]
[softmax layer1 feeding in inner product and labels1]
[accuracy layer2 feeding in inner product and label2]
[softmax layers2 feeding in inner product and labels2]

[Dinesh]

For future reference, I have confirmed that it is possible to use export HDF5 files from Matlab (that look identical to echo's file above with h5disp), and it works. In fact, if you use Matlab's "low-level" HDF interface functions, you can get identical HDF5 files from Matlab/Python.

The one thing I had to be careful about when generating my HDF5 file was that HDF5_DATA layers do not seem to apply the transform_param to the data (even though they do not throw an error if passed transform_param in the definition). So you need to rescale data beforehand, and this makes a huge difference. I went from near-chance performance on MNIST to the usual >99% with this one change. 

[echoaimaomao]

Thanks Dinesh. I have already known the transform_param  did not work in HDF5_DATA layer,what actually made me confused was that the network worked  very well with no data re-process in LMDB data type, but it did not work in HDF5_DATA layer. That is to say , I did not define transform_param neither in the HDF5_layer nor the LMDB_layer, the only difference was the data type which resulted in quite different performance. Have the data already been changed different by different data layer? Or does diffeent data type suit some specific classification task?


在 2015年1月13日星期二 UTC+8上午10:50:09，Dinesh写道：

[Dinesh]

In my experiment with mnist data, I found no such effect. My LMDB data failed just as badly when I disabled transform_param in DATA_LAYER.

You could check what your data looks like, by using extract_feat to dump the data right after it is read by the HDF5_DATA layer, before any further processing. 

[James Guo]

Hi Soda,

For accuracy layer here, it should output some sort of probability, right? Then is it possible that your two labels are complementary to each other?

[James Guo]

Maybe the problem is that you only have one inner product layer to output :)

On Monday, 12 January 2015 03:19:30 UTC+1, Soda Heng wrote:

[Wei Guo]

How to convert a multi task dataset and labels to caffe data format?

For example, if my input data is a image from a file, and the labels are classification_id, regression_vector_with_float value. How to convert the dataset including these labels (classification and regression) to caffe data format such as LMDB?

Thanks,  

[James Guo]

Please kindly find my new post here 

https://groups.google.com/forum/#!topic/caffe-users/_NdX1tpHt5Y

[marina]

hi,Wei Guo. Did your problem has been solved? I also had this problem.

在 2015年2月6日星期五 UTC+8下午2:48:54，Wei Guo写道：

[Andrea Isoni]

Hi James Guo,

can I kindly ask you if you can send me the prototxt files to train the multitask network with caffe (1 image multiple labels)? My mail is ciccib...@tiscali.it

Many Thanks,