Memory problem with torch.cat/ big tensor

[Tuan Tran Nguyen]

Hi,

I tried to load about 100000 images which I stored as *.jpg.
Using img = image.load I can load each single image, but when I go through all pics and concat them to one tensor, and somehow my OS got frozen after about 600th image already (Ubuntu via Virtual Box)


for name in lfs.dir(dot) do
   ..
   ..
   local img = image.load(filename)
   if images == nil then
      images = img
       -- resolution 120x160 px
   else
      images =   torch.cat(img, images, 1)
     
  end

end

and it gets frozen even when I tried this

x= torch.Tensor(100000, 120,160):fill(1)

Does Torch keep the whole tensor in RAM memory? or this happened to me cause I use Virtual Box?

Thanks

Tuan

[Francisco Vitor Suzano Massa]

Regarding your last question, yes, when you create a Tensor in Torch it is stored in RAM memory. The tensor you are creating (x= torch.Tensor(100000, 120,160):fill(1)) allocates around 15GB of memory, so maybe you don't have enough RAM to do it.

If you do have enough RAM to load the whole Tensor, this problem you are experiencing happens because lua hasn't collected the references to unused tensors.
First, put a collectgarbage() call inside the for loop every ~50 iterations.
Second, torch.cat allocates new memory each time you call it. To make things faster, you could preallocate the big tensor in memory, and then copy the loaded image directly to it.
For example:

images = torch.Tensor(100000,120,160)
i = 1

for name in lfs.dir(dot) do

  local img = image.load(filename)
  images[i] = img
  if i%50 == 0 then
    collectgarbage()
  end
  i = i + 1
end

[Tuan]

Thank you for your answer

But unfortunately I dont have so much RAM memory, so how can I load all the images and feed them to nn?

Or I can imagine one way to get around of it -->  that I have to reload the every X images (e.g. X=20000) during training, feed them to the NN and reload the next X images. Is there any better solution?

[Francisco Vitor Suzano Massa]

That's the way to go when you have huge datasets, like Imagenet. You can also use threads to load the images in parallel, like in soumith ImageNet training code. https://github.com/soumith/imagenet-multiGPU.torch