How to assign proper mapsize for LMDB?

[vision fans]

Caffe hard coded 1099511627776(1TB) as the mapsize for LMDB. See the screenshot below:

That would create a 1TB memory mapped file in the hard disk. However, it's too large for my disk to hold. 

My OS is Win7(64bit). The partition I used to store data is 100GB.

So ,what is the proper way to use LMDB? Just modify `1099511627776` to a smaller size, e.g. 1GB, then recompile Caffe? 

If I met a larger dataset thatn 1GB afterwards, re-modify the size and re-compile?

Would the memmaped file used by LMDB resize automaticly when needed?

Best,

Sun

[Jonathan L Long]

The 1TB value is taken directly from the lmdb docs (see http://lmdb.readthedocs.org/en/release/). This value is not a disk space requirement, and should not cause any problem related to available disk space (or memory), as long as there is sufficient space to actually hold the data.

JLL

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/7a89b308-9edc-4a53-88a4-014b72baf75d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Howard Chu]

On Windows the OS sets the filesize equal to the mapsize. (MacOS requires that too, and allocates all of the physical space up front, it doesn't support sparse files.) The mapsize should not be hardcoded into software, it needs to be reconfigurable. On Windows and MacOS you really shouldn't set it larger than the amount of free space on the filesystem.

Best,

Sun

[Jonathan L Long]

Ah, okay, I was not aware of such issues, thanks for the correction.

Perhaps we should break the map size out into Makefile.config or a gflags option...

You're welcome to create an issue or PR on the github to address this.

(Actually I did not find the docs so clear; from the Python docs, we have

"On 64-bit there is no penalty for making [the map size] huge (say 1TB). "

and from the doxygen

"The value should be chosen as large as possible, to accommodate future growth of the database."

while elsewhere it's noted that using writemap=True "may cause filesystems that don’t support sparse files, such as OSX, to immediately preallocate map_size= bytes of underlying storage." However, afaict we don't use the MDB_WRITEMAP option. Have I missed something?)

JLL

--

You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/d8b0811c-177e-4eb9-91b5-6ac4ce8ea35f%40googlegroups.com.

[Howard Chu]

On Wednesday, October 8, 2014 8:29:26 PM UTC+1, Jonathan L Long wrote:

Ah, okay, I was not aware of such issues, thanks for the correction.

Perhaps we should break the map size out into Makefile.config or a gflags option...

You're welcome to create an issue or PR on the github to address this.

(Actually I did not find the docs so clear; from the Python docs, we have

"On 64-bit there is no penalty for making [the map size] huge (say 1TB). "

There is no penalty in terms of address-space consumption.

and from the doxygen

"The value should be chosen as large as possible, to accommodate future growth of the database."

"as large as possible" is, by definition, less-than-or-equal-to the amount of free disk space.

while elsewhere it's noted that using writemap=True "may cause filesystems that don’t support sparse files, such as OSX, to immediately preallocate map_size= bytes of underlying storage." However, afaict we don't use the MDB_WRITEMAP option. Have I missed something?)

Windows sets the filesize to the mapsize, regardless of WRITEMAP. I don't recall whether OSX does or not, we don't test on OSX very often.

[Luke Yeager]

FYI, the LMDB map size issue should be fixed with this PR that was recently merged:
https://github.com/BVLC/caffe/pull/3731

You may still run into issues on other platforms when dealing with LMDBs that are truly large, but the code no longer defaults to using 1TB.