Performance using reshapes and forwards

[Deon Joubert]

I am using a fully convolutional VGG network to classify very large images. As each image, together with the network's memory allocations, is too large to fit into our GPU's memory, I divided the image into a number of smaller blocks. Each each block is then processed (with forward) and the outputs are concatenated together. The total processing time for each image is about 60 seconds, which is very long. 

What I have found is that, as not all the image blocks are the same size, the reshaping of the network for each different sized block causes the processing to take very long. For a series of blocks, I get the following measurements:

Time to process the first block of size A: 5 seconds

Time to process the second block of size A: 7 seconds

Time to process the rest of the blocks of size A: 0.2 seconds

Time to process the first block of size B: 5 seconds

Time to process the second block of size B: 7 seconds

Time to process the rest of the blocks of size B: 0.2 seconds

etc...

So, it seems to me that reshaping the network takes some time, but that during the processing of the second block there is some sort of pre-configuration that occurs, which takes quite long but that greatly speeds up any subsequent same block size processing. 

What exactly is causing this behaviour and is there any way for me to control this behaviour through pycaffe? Ideally what I would like is to have the pre-configuration take place when processing the first block and not on the second one.

Any help would be appreciated