Low GPU utilization when loading data

[Rasool Fakoor]

Hi,

I have a function which returns the next mini-batch during training/evaluation. This function reads an HDF5 file which is ~50GB. This hdf5 contains preprocessed data (i.e. a tensor with the N*F*D1*D2 dimension). Every time, it returns a tensor of size B*F*D1*D2 where B is the batch size.

Anyway, the code reads data without no problem from disk and returns the requested minibatch. However, when this function executes, the GPU utilization almost becomes zero, i.e. not using gpu of course, for this period of time which is predictable given I/O operations. 

I found that these ' self.minibatch:resize(batch_size, F, d1, d2):zero()' and 'h5_file:read('mykey'):partial()' causes the gpu to not be efficiently utilized. I tried a couple of different tricks and various implementations but no concrete solutions except using threads. 

My question is that is there any better way to deal with this problem in Torch and utilize GPU when loading 4D-tensors and scenario like me? Or using thread is the only solution?

Any help would be highly appreciated.

Thanks.

PS: a simplified version of code is pasted here 

function DataLoader:get_next_batch(batch_size)

    local N  = 10000 --total number of example  

    local F  =  40  

    local d1 = 512

    local d2 = 100

    self.minibatch:resize(batch_size, F, d1, d2):zero()

     for b = 1, batch_size do

           local idx = get_next_id()

            self.minibatch[{{b}, {}, {}, {}}] = self.h5_file:read('mykey'):partial({idx, idx},{1,F},{1,d1},{1,d2})

    end

     local data = {}

    data.mb = self.minibatch

    return data

end

 

[Kiran Vaidhya]

I don't see why you would need GPU to do the said operations. Even if you wanted to do some tensor operations on GPU, you would ideally transfer the said tensor to the GPU from ram using :cuda()

For example, if you wanted to do the resizing in GPU, you could do self.minibatch = self.minibatch:cuda():resize(batch_size,F,d1,d2):zero()