Since this question was asked a lot in this group and I think it's worthwhile to provide my personal experience to solve it. I'm currently using up to 20 different labels(including vectors and scalars) for the same data. The dataset format I used is HDF5. The advantage is in HDF5, you could use a vector as label, and the length of vector could be as long as you want. The idea here is you put all labels you want into this long vector, and slice it in .prototxt file. For example, you have an image data, and this data is corresponding to 3 different labels, say coordinates of left eye (x, y), age of people in the image (a) and gender of people in the image (g). The label in your HDF5 is simply (x, y, a, g)
Then in your .prototxt file, define slice layers as follows:
layers {
name: "image"
type: HDF5_DATA
top: "data"
top: "label"
hdf5_data_param {
source: path to your files
batch_size: whatever you want
}
....
....
}
layers {
name: "slice0"
type: SLICE
bottom: "label"
top: "left_eye_position"
top: "label1"
slice_param {
slice_dim: 1
slice_point: 2
}
}
layers {
name: "slice1"
type: SLICE
bottom: "label1"
top: "age"
top: "gender"
slice_param {
slice_dim: 1
slice_point: 1
}
}
Now you could do whatever you want in Caffe with these three labels
For detailed examples for HDF5 dataset and slice layer in Caffe, you could find them here
Good luck!