How much space does it take to convert lsvrc12 to lmdb?

[Hossein Hasanpour]

Hello everyone, I'am currently in the middle of conversion, so far its taken 128 Gigabytes of space! and only 451,000 files are processed!! 

the raw dataset takes up 144 Gigabytes, and since it contains images with differing resolutions (most of them seem to be beyond 400x600 as far as I noticed),

and in the lmdb file, I guess we have a compression mechanism, + resizing the images to 256x256, lmdb should take less than 144! 

Am I right? I only have 27 Gigabytes left and If you have done this before, I'd be grateful to give me an estimation or exact size after conversion. 

by the way does someone, has the mean/std values for lsvrc12 as well? 

Thanks in advance  

[Hossein Hasanpour]

OK. here is the result for someone in the future that might need to know this!

With an Intel Haswell Pentium 4 3.0 Ghz CPU, it took 10 hours to process 684,000 files(out of 1,281,167),

and then crashed due to insufficient space, It had taken 128 Gigabyte by then and only 27.5 Gigabyte was remaining at the time of crash!,

So I guess at least it needed another 100 or 120 Gigabytes which I clearly didn't have!.

I tried resizing to 54x54 instead of 256x256, it took 11 hours to complete the conversion. and it only took 16.7 Gigabytes of space. which is a huge difference (17 vs ~250 Gigabyte!)

[nila...@gmail.com]

For me, the training partition of lsvrc12 takes 239GB of disk space in lmdb format when i'm using the examples/imagenet/create_imagenet.sh script with resizing (256x256). Keep in mind that when creating the lmdb files you need to store the original images too, that gives you a total of 380GB disk space that is required, with 140GB free after deleting the original images.