LMDB vs File System as Key Value Store

[Alex Rothberg]

I have seen some pieces that compare LMDB vs either LevelDB or HDF5 for use as a key value store when running caffe:

http://deepdish.io/2015/04/28/creating-lmdb-in-python/

http://caffe.berkeleyvision.org/tutorial/data.html

I am curious as to what are the advantages of using LMDB vs just storing the binary data as individual files and then using the file system as the key value store?

[Howard Chu]

Using a filesystem in the way you suggest would be extremely low performance, requiring several system calls per data access. LMDB operates with zero system calls per data access. There is nothing else anywhere near as fast and efficient as LMDB.

[Alex Rothberg]

Are there any benchmarks that you know of comparing the performance of the two?

Further are you able to hide the data access penalties through pipelining the loads?

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/WqBfwUJ98c0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/52df0ff1-78e6-47b9-b386-6a32aff6eae7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Howard Chu]

On Saturday, June 27, 2015 at 5:23:07 PM UTC+1, Alex Rothberg wrote:

Are there any benchmarks that you know of comparing the performance of the two?

No, but you can write one yourself easily enough. You could start by simply writing a program to create a file, write a few hundred bytes of data to it, and close it again. Run this in a loop for a miillion times (or some suitably large number) and time how long it takes. That would simulate using a file per record of a "key value store". Note that this only shows an unattainable best case, since in a real database environment you also need to implement locking to make sure the data you're accessing doesn't change out from under you.

Having measured how long it takes to add the records, you could run a similar program that opens a file, reads the contents, and closes it again. That would represent your data fetch operation. Again, run the sequence for an appropriately large number of iterations and time it.

I personally have no need to write or run such a benchmark because I already know that crossing from userland thru the syscall barrier involves a large number of instructions, and then threading thru the syscall dispatcher to the filesystem handler to perform open/read/write/close/etc. with all the ensuing filesystem cache interactions is many thousands of lines of code, many of which take locks, block, or sleep while executing. Whereas LMDB itself has a hot code path of only a few hundred lines of code, none of which lock or block while executing. It would be a total waste of time for me, I've been there and done that ages ago. But if you really don't know well enough already and need to convince yourself, it's a simple exercise. Go for it, and publish your results.

[Achal Dave]

Sorry to bring up an old thread, but one thing to note is that the file system may be faster if you need random key access, as far as I can tell. I've seen very low performance with searching for arbitrary keys in LMDBs.

[Howard Chu]

Nonsense. Yes, random key lookups are more expensive than sequential. That's true for both LMDB *and* filesystems. There is no scenario where opening a file to retrieve a single record will be within even 2 orders of magnitude of LMDB and it's utterly irresponsible of you to even suggest such a thing.

Show code and test results. Otherwise you have no business speaking on the topic.

[Achal Dave]

Hm, thanks for the heads up. I'll double check what I have and see what's wrong. I'm using LMDB for the first time, and while playing with it yesterday in Caffe through a Python data layer, it was taking me around ~1 second to search for an arbitrary key and load an image (256x256x3, about 1.5MBs).

Perhaps there was high load on the machine at the time, as I'm seeing much better performance today (~50 images in ~4s including post processing). I'll try and run some explicit benchmarks to see if checking if a directory exists is faster than checking if a key exists in LMDB, and hopefully I'll see what you suggest.

[Achal Dave]

(Let me preface again by saying I'm posting to ask a question rather claiming the file system is faster than LMDB.)

I've written a python script that

1. takes an LMDB as input
2. writes all key-values to disk
3. shuffles the keys
4. attempts to read the corresponding key values in shuffled order from LMDB
5. attempts to read the corresponding key values in shuffled order from the file system

I'm sure something must be wrong, but using https://github.com/rkern/line_profiler/, it seems that while the LMDB version is faster, it's not an order of magnitude faster (~9.5ms for LMDB vs ~11ms for FS to search for a key and load it). Any chance you have a pointer that might help me debug this?

The source and the timings are here: https://gist.github.com/achalddave/8e9490efd348cdcabf21232358ec55eb