LMDB Usage

[Riddhiman Dasgupta]

Hi,

I'm facing some difficulties in trying to use the lmdb wrapper from https://github.com/eladhoffer/lmdb.torch

This is the code snippet I am using to create a database, fill it with some tensors, and then trying to read all the stored tensors again.

require 'lmdb'

require 'xlua'

cmd = torch.CmdLine()

cmd:option('-rows',3333,'Number of samples')

cmd:option('-cols',2,'Number of dimensions')

opt = cmd:parse(arg or {})

db= lmdb.env{ Path = './testDB', Name = 'testDB' }

db:open()

txn = db:txn() --Write transaction

x=torch.Tensor(opt.rows,opt.cols):byte():fill(10)

-------Write-------

for i=1,opt.rows,1 do

    xlua.progress(i,opt.rows)

    txn:put(i,x[i])

end

txn:commit()

print("Writing done")

print('db : '..db:stat().entries)

print("Reading now")

reader = db:txn(true) --Read-only transaction

y = {}

-------Read-------

for i=1,opt.rows,1 do

    xlua.progress(i,opt.rows)

    value = reader:get(i)

    table.insert(y,value)

end

print('x : '..x:size(1))

print('y : '..#y)

reader:abort()

db:close()

Almost every time I use a tensor with more than 2000 rows or 2 columns, it turns out that all the keys are not stored to the database, and even if they are, all keys cannot be retrieved from the database. More specifically, this is the error message

Error in LMDB function mdb_get : .......MDB_NOTFOUND: No matching key/data pair found

What seems to be the problem, and how can this be solved?

[soumith]

Hey Riddhiman,

I dont get the errrors that you see. This is my output when I run the script:

th lmt.lua

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

Writing done

db : 3333

Reading now

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

x : 3333

y : 3333

I am using lmdb-0.9.14 and latest lmdb.torch

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at http://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Riddhiman Dasgupta]

Huh. Could you try running it multiple times? Or maybe change the dimension to something larger? I keep getting the error almost everytime I run this script with different sized matrices. I keep deleting the database between runs.

On Sunday, October 18, 2015 at 8:05:44 PM UTC+5:30, smth chntla wrote:

Hey Riddhiman,

I dont get the errrors that you see. This is my output when I run the script:

th lmt.lua

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

Writing done

db : 3333

Reading now

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

x : 3333

y : 3333

I am using lmdb-0.9.14 and latest lmdb.torch

[Riddhiman Dasgupta]

OK, so this is what I'm getting now when I run 

$ rm -rf testDB/; th lmdb.lua -rows 4000 -cols 5

Error in LMDB function mdb_cursor_get : MDB_NOTFOUND: No matching key/data pair found

 [============================================================== 4000/4000 =========================================================>] ETA: 0ms | Step: 0ms          

Writing done

db : 3998

Reading now

Error in LMDB function mdb_cursor_get : ........MDB_NOTFOUND: No matching key/data pair found........................................] ETA: 692ms | Step: 0ms        

Error in LMDB function mdb_get : MDB_NOTFOUND: No matching key/data pair found

/torch/install/bin/luajit: inconsistent tensor size at /torch/pkg/torch/lib/TH/generic/THTensorCopy.c:16

stack traceback:

[C]: at 0x7f2b3384e3c0

[C]: in function '__newindex'

lmdb.lua:48: in main chunk

[C]: in function 'dofile'

...iman/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:133: in main chunk

[C]: at 0x00406670

Segmentation fault (core dumped)

On Sunday, October 18, 2015 at 8:05:44 PM UTC+5:30, smth chntla wrote:

Hey Riddhiman,

I dont get the errrors that you see. This is my output when I run the script:

th lmt.lua

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

Writing done

db : 3333

Reading now

 [================================================================================================== 3333/3333 ==============================================================================================>] ETA: 0ms | Step: 0ms

x : 3333

y : 3333

I am using lmdb-0.9.14 and latest lmdb.torch

[soumith]

ok I reproduced it over multiple runs, after making sure I deleted the database.

Another easy way to reproduce was to make the rows to be 10x more. (33333)

It seems that lmdb doesn't like the keys to be doubles. I converted the keys to be strings and it was happy:

require 'lmdb'

require 'xlua'

cmd = torch.CmdLine()

cmd:option('-rows',33333,'Number of samples')

cmd:option('-cols',2,'Number of dimensions')

opt = cmd:parse(arg or {})

os.execute('rm -rf testDB')

db= lmdb.env{ Path = './testDB', Name = 'testDB' }

db:open()

txn = db:txn() --Write transaction

x=torch.Tensor(opt.rows,opt.cols):byte():fill(10)

-------Write-------

for i=1,opt.rows,1 do

   xlua.progress(i,opt.rows)

   txn:put(tostring(i),x[i])

end

txn:commit()

print("Writing done")

print('db : '..db:stat().entries)

print("Reading now")

reader = db:txn(true) --Read-only transaction

y = {}

-------Read-------

for i=1,opt.rows,1 do

   xlua.progress(i,opt.rows)

   value = reader:get(tostring(i))

   table.insert(y,value)

end

print('x : '..x:size(1))

print('y : '..#y)

reader:abort()

db:close()

[Riddhiman Dasgupta]

This works. I was wondering whether keys have to be strings, but I assumed the wrapper would be handling it.

Thanks a ton Soumith! :)

[Cédric Deltheil]

On Monday, October 19, 2015 at 10:29:54 AM UTC+5:30, smth chntla wrote:

It seems that lmdb doesn't like the keys to be doubles. I converted the keys to be strings and it was happy:

It's indeed pretty strange since a tostring convert is automatically done (via the is_key param: https://github.com/eladhoffer/lmdb.torch/blob/8a385db/init.lua#L42-L48) at put (https://github.com/eladhoffer/lmdb.torch/blob/8a385db/DB.lua#L151) and get (https://github.com/eladhoffer/lmdb.torch/blob/8a385db/DB.lua#L161) time. There might be something wrong somewhere... (possibly a garbage collection problem since after the write loop the nb. of elements varies along runs).

[Riddhiman Dasgupta]

I tried committing and collecting garbage in between the put calls, but the end results remains the same. With a large number of tensors, the number of elements that are actually put into the database varies along runs. And the gets fail as a result.

[Junonia]

I am getting similar problems.  I think the lmdb.MDB_val function in init.lua is problematic. At line 51 in init.lua, ffi.cast require value be still available when the cast pointer is used. Since "value" is local and it may be garbage collected before its usage. One solution is to return the value and keep it until transaction is complete. Another one would be explicitly allocate the memory and copy the content of the value to the allocated memory. Something like the code below to replace the ffi.cast line.

local p = ffi.new('char['.. value:len()..']')

ffi.copy(p, value)

mdb_val[0].mv_data = p

[Mansi Rankawat]

Hi,

I am trying to use this package (https://github.com/eladhoffer/lmdb.torch) to read an already existing LMDB database. However when I do so I get the following error:

Error in LMDB function mdb_get : MDB_NOTFOUND: No matching key/data pair found

Here is my code for reading the database:

local db=lmdb.env{

Path = './data/CamVid-train-lmdb',

Name = 'CamVid-train-lmdb'

}

rows=367

db:open()

local reader = db:txn(true)

inputs=torch.Tensor(rows,1,240,320)

for i=1,rows do

        inputs[i]=reader:get(tostring(i))

end

reader:abort()

db:close()

Thanks

Mansi

[Iaman Swtrse]

I haven't used torch7 or even plan to.
I found this post while searching for that Error Message "MDB_NOTFOUND: No matching key/data pair found".

I run into this error when access my openladap mdb backent with LemonLDAP that is based on PHP.
The Error statet in the ldap-Log is "<= mdb_dn2id: get failed: MDB_NOTFOUND: No matching key/data pair found (-30798)"
The strange thing is ~1 out of 10 times it is working the other times this error occurs.

Based on that and your post I came to the conclusion that there should be nothing wrong with torch7 but with the openLDAP settings and the the solution should lie inside the mdb configuration.
Maybe an compare of the mdb backend settings bings some light to this case.

I would post mine but I am currently at work and have no access to my server right know.

Regards