Urgent Help needed with Multiple input - Single label Network

[Sreejith Balakrishnan]

Hi Everyone, 

This is my first post in this usergroup. So please go easy on me if its a stupid question.

I am trying to create a network to recreate the work mentioned in this paper: GazeFollow. I have attached the prototxt for this network. The network has three input branches. 

1) One takes in raw image which I have used the ImageData layer to provide. This layer provides two tops - "data" and "label". 

2) The second layer takes in cropped head image which also is provided using ImageData layer. Here, the tops are "face" and "label1". 

3) Third layer takes in the position of the eye in the original image when it is split into a 13 x 13 grid. So this value is between 0 and 168. This is provided using the Data layer using lmdb. The tops are "eyes_grid" and "label2"

Since the network is trained end to end (no separate training for each branch), all the labels are the same. This means label, label1 and label2 are identical.

For my final Softmax layer, I have only connected it to label. label1 and label2 are left unconnected (connected to a silence layer). 

I realized that after training, the accuracy is really bad. I checked the weights of some of the CNN layers and I believe some branches are not trained well. So I want to make sure it is not because of the way I have connected my labels.

Is it ok to leave label1 and label2 unconnected? If not, what should I do (ImageData expects two tops). 

Also, I am curious to make sure that the data between the three inputs is always in sync since I am not explicitly using any shuffling options.

Thank you. I am using this for a project with a tight deadline. So any reply would be appreciated. 

Best regards,

Sreejith

[Patrick McNeil]

I have done exactly what you are talking about, but with different data.  I have multiple inputs (up to 10 right now) and use separate data layers (DATA1 - DATA10) and only have the DATA1 send the label to the end (the rest I don't have the label connected).

In my data prep process, I had to make sure the data are input in the same order.  I didn't shuffle the data, but if you wanted to, you can shuffle, but it has to be on a per row basis.

If you are getting really bad performance, it sounds like there is an issue with the data (some of the rows contain data from different labels for example).  I ran into that issue initially (a couple of files in my dataset were missing for some of the inputs and from that point forward all of the rows were off).  It took a couple of weeks to figure what was going on.  I ended up writing an app to read my source text files and compare line by line to ensure the label matched on all inputs.  I removed the rows that had a missing line from the cleaned text file.  I used the cleaned text files to create the LMDB files. 

Hopefully this helps out.

Patrick

[Sreejith Balakrishnan]

Thank you Patrick. I think there was a mistake in my architecture. I am working with the author to see if I can get a better result. But like you said, having labels not connected is OK. I followed your advice and made sure they are all consistent. 

Thank you again.

[Medhani Menikdiwela]

Hi,
I'm trying to train a network  with two lmdb data inputs. One for data layer and one for lables. Is it possible to train in that way?


Medhani