lmdb usage notes

[dv]

Hi! Just got going with lmdb and so far so good.  Now, looking for some examples on how to setup an Env (for multiple databases) and using Cursors.  Thx!

[Nan Jiang]

Hi dv,

According to liblmdb's terminology, an Env stands for a single mmap file, which in turn contains a number of DBs.

Here is some basic usage,

>> env, txn, db = mdb.mdb_write_handle('/tmp/mdb', 10*mdb.MB, 'sub-db1')

>> db.put(txn, 'key0', 'value0')

>> db.put(txn,  'key1', 'value1')

>> txn.commit()

Cursor is no longer frequently used since DB wraps most of the cursor manipulations.

>> env, txn, db = mdb.mdb_read_handle('/tmp/mdb', 'sub-db1')

>> db.get(txn, 'key')          # get value for the given key

>> db.get_dup(txn, 'key')  # get all duplicate values for the given key

>> db.get_lt(txn, 'key')      # get all key/values whose key is less than the given key

>> db.get_le(txn, 'key')     # get all key/values whose key is less or equal than the given key

>> db.mget(txn, ['key', 'key1'])    # get values for a list of keys

>> db.items(txn)               # traverse all key value pairs

>> txn.commit()  

More examples are available here,

https://github.com/chango/hustle/tree/develop/deps/liblmdb

Hope this is helpful.

cheers,

Nan

[dinesh vadhia]

That is helpful and so are the examples on the github page.  A few more basic questions:

How do you create/open/close/drop an Env to support one or more databases?

Once an Env is created, how do you create/open/close/drop a specific database?

What is the parameter and value for no duplicates allowed?

I'm using, db.put('key0', 'value0').  In the example, db.put(txn, 'key0', 'value0'), what role does txn (transaction) play?  

[Tim Spurway]

You need to create an Env to do anything.  Think of the Env and the actual memory mapped disk file as synonymous.  You can only create one Env per file.

1 File -->  1 Env

Create an Env with the Env constructor method.

You need a Txn to do anything.  A Txn is a transaction, which is either read or write and can be committed or aborted - it's a normal database transaction.

Env.begin_txn() method creates a Txn.

Each Env can contain many DBs.  A DB is a key value store and has many other options (types, read/write, duplicates, etc). 

Env.open_db() method creates a DB of specific type.

A DB can be read and written, and is a key/value store.  

DB.get() and DB.mget() get values given a key.

DB.put() sets a given key/value pair

DB.delete() does what you think

There are also a number of 'range' methods for iterating over ranges of keys/values:

DB.get_lt(), DB.get_gt(), DB.get_eq(), DB.get_ne(), DB.get_ge(), DB.get_le(), DB.get_range() and DB.items() are for this.  

The native LMDB library supports Cursor objects, but we have abstracted them behind this higher level interface.

Hope that gets you going

--
You received this message because you are subscribed to the Google Groups "hustle-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hustle-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dinesh vadhia]

This is embarrasing but stumbled at the first hurdle creating an Env with:

>>> import mdb

>>> fn = "mdbtest"

>>> maxsize = 100 * 1024 * 1024

>>> env = mdb.Env(fn, mapsize=maxsize, max_readers=1024)

Traceback (most recent call last):

  File "<pyshell#9>", line 1, in <module>

    env = mdb.Env(fn, mapsize=maxsize, max_readers=1024)

  File "db.pyx", line 250, in mdb.Env.__init__ (db.c:5050)

Exception: Error opening environment: No such process

[Tim Spurway]

We haven't really created any additional documentation for our LMDB library.  Our lib is really just a wrapper around the original C library.  I don't know that you will be able to understand all of the subtleties of this library by looking at our code.  Your best bet is to fully understand the C lib first.  Check out http://symas.com/mdb/doc/ 

[Tim Spurway]

I would also like to stress to anyone else reading that it is completely unnecessary to know the details of the low-level LMDB library to use Hustle.

[dinesh vadhia]

The path statement was incorrect, thinking it was both path+db_name but just path is needed ie.

DB_PATH = "/path/to/mdb/db/directory"

env = mdb.Env(DB_PATH, mapsize=, etc.etc.)

Instead, can use a high-level abstraction to create both the Env and a db eg.

env, txn, db = mdb.mdb_write_handle(DB_PATH, 10*mdb.MB, 'db1')

Question: If multiple db's are created and open eg.

env, txn, db1 = mdb.mdb_write_handle(DB_PATH, 10*mdb.MB, 'db1')

env, txn, db2 = mdb.mdb_write_handle(DB_PATH, 10*mdb.MB, 'db2')

then, are different transaction handles needed if operating in the same namespace ie txn1, txn2 or is txn a generic handler?