Dataset and Caffe are in two different hard drives. Does that affect the training time?

[Jumabek Alikhanov]

Hi,

as title says,

I installed caffe on my first HDD which has 350GB space,

I have my lmdb dataset of imagenet in another HDD which has 1Terabyte space.

The reason :

I just wanted to save the caffe models (experimental results) in my first HDD (350GB) 

and save dataset in second HDD(1T).

Do you think it is smart?

or should I install caffe on second HDD?

I have two GPUs:

Titan X 

Tesla K40

Even though they have 12 GB ram each

they don't seem to use it fully

Also during training there is a lot of blocking queue like below screen-shot:

And lastly even if I have 24 GB RAM, only 45% is working during training.

Is that normal?

Thanks for the help!

[xv1nx last]

I think caffe allocate GPU Ram for the whole Net, train and test. Maybe it only loads 1 or 2 batches in the cache, but not more. I observes the same behavior. You could reduce the blocking message if you reduce your batch size. Caffe will need more GPU Ram if you net or your batch is larger.
I think there would no or just negligible speed up if you install caffe in the same disk like your data base. But if you use an SSD to store your LMDB, that would help a lots.