out of memory during lmdb creation

[nila...@gmail.com]

Hi everyone,

I'm trying to convert a rather big dataset into lmdb format. On my 8GB RAM machine I'm getting an out of memory error after ~16000 images, I did not experience similiar issues before when using a smaller dataset. As I'm trying to do segmentation, I based my code for lmdb creation on the snippet from https://github.com/BVLC/caffe/issues/1698 . This is happening when I'm trying to save the input images into an lmdb, so no ground truth images have been touched so far. I was under the impression, that there is no need to worry about lmdb memory consumption as the lmdb library shouldn't allocate more memory than there is available. I also observed no increase in the amount of swap memory used. I'm getting the following error message:

Unexpected error: <type 'exceptions.MemoryError'>
in_idx=16362
Traceback (most recent call last):
  File "convert_lmdb_2x2.py", line 50, in <module>
    im = np.array(im)
MemoryError

Did anyone experience something similiar before? Also, is there a way to make the code snippet from Shelhamer I linked above more memory efficient or limit lmdb memory consumption? I don't have much understanding of the underlying mechanisms of lmdb unfortunately.


Best regards
Nilas

[Ahmed Ibrahim]

one thing i would do is to use the provided binary tool from a shell window , i guess that will be better optimized than the python script

[Jonas]

I have the same problem with python. i just start my script to push 10 000 images, than the next and so on :) works well

[nila...@gmail.com]

I've been working around this issue for some time now, but ultimately I found the fix at [1] in adding writemap=True to the lmdb.open call. I just wanted to leave this here for future reference.

[1] https://github.com/dw/py-lmdb/issues/113

Am Mittwoch, 16. März 2016 17:56:07 UTC+1 schrieb nila...@gmail.com: