Storage requirement for LMDB from ilsvrc12

[Yan Ma]

I'm trying to execute create_imagenet.sh script on ilsvrc12 dataset. The ilsvrc12 training set is 139 Gb, but when I create the train_lmdb, its size blew up to more than 550Gb and I was out of storage.

Just wondering where they is a 4x+ storage requirement, and if I'm not doing anything wrong, how big of an hard drive do I need?

Thanks!

Yan

[Christos Nikolaou]

In my system the ilsvrc12_train_lmdb is 240 Gb and the ilsvrc12_val_lmdb is 9.4 Gb (after running du -h), whereas the space required for the JPEG images is equal to what you mentioned (139 Gb). I do not know why your train_lmdb is so much larger and I do not thinl that it makes sense at all to be that large.

[Christos Nikolaou]

Also, take a look at that which will answer your question better: https://groups.google.com/forum/#!searchin/caffe-users/storage/caffe-users/tBMCC1N3Ts8/cn54tQdnA9oJ

[Yan Ma]

Thanks Christos, did you use the encoding option for generating your 240Gb lmdb? What command are you using to generate it, presuming that you're using $CAFFE_HOME/example/imagenet/create_imagenet.sh?

Cheers,

Yan

[Yan Ma]

Also note that I'm using the NVIDIA fork of Caffe, will try the BVLC repo and get back to you.

Cheers,

Yan

[Yan Ma]

Found the answer. Turns out the create_imagenet.sh script does not enforce resize on default. After I set the resize parameter to true, the result corresponds to what Christos described. Thanks!

Yan

[Christos Nikolaou]

I have made a new PR on the Github to the project to clarify those things for future users. Hope it helps when it will be merged.