convert leveldb to lmdb cifar10

[Z.L]

Hi

since leveldb only allows one process to read, I would like to convert my cifar10 leveldb data to lmdb format! Then I wrote the following python code

from __future__ import print_function

import os

import caffe

import leveldb

import lmdb

import numpy as np

from caffe.proto import caffe_pb2

def pause():

    programPause = raw_input("Press the <ENTER> key to continue ...")

data_path='cifar10-gcn-leveldb-splits-copy5/cifar10_test_leveldb'

db = leveldb.LevelDB(data_path)

datum = caffe_pb2.Datum()

count = 0

datum_list =[]

for key, value in db.RangeIter():

    # read from leveldb database

    datum.ParseFromString(value)

    datum_list.append(datum)

    count = count + 1

    if count%1000 == 0:

        print(count)

# create lmdb database

env=lmdb.open('cifar10_test_lmdb_v1', map_size=int(1e12))

# create lmbd database

with env.begin(write=True) as txn:

    # txn is a Transaction object

    for i in range(count):

        str_id = '{:0>8d}'.format(i)

        # The encode is only essential in Python 3

        #txn.put(str_id.encode('ascii'), datum.SerializeToString())

        txn.put(str_id, datum_list[i].SerializeToString())

it is executable, but it seems something was wrong! when I try on original leveldb data, it have better performance, but when I use converted lmdb data, it is so bad! The results from both leveldb and lmdb dataset should be same or at least similar.

If anyone could help, that would be really appreciate!

zhe