Training with Python Layer is slow and depends on how much data i load on system ram ?

[Sagar Kale]

Hi,

I am trying to train a object detection model using caffe. I have defined a python layer for classification and regression loss. Training process is taking too much time.

I can understand python layer is slow compared to other layers. But, I am observing strange behavior.

System: Linux Ubutu 16.04

System RAM: 16GB

GPU: NVIDIA GTX 1080 8GB

Input Image Size: 3x48x48

Batch Size: 256

Training Data Size: ~12 lacs

Observations:

If i load 5000 images and start training, I can see 0.05 seconds per iteration - This is ok

If i load 100000 images and start training, I can see 4 seconds per iteration - This is slow

If i load 12lacs images and start training, I can see 12 seconds per iteration - This is very slow

I am not understanding, Why training time per iteration is changing ?

Regards,

Sagar