How can I read from the pickle files and convert it to lmdb/leveldb format?

[Hossein Hasanpour]

Hello everyone.

I'm trying to convert a numpy array that was created using pylearn2 into leveldb/lmdb so that I can use in Caffe.

This is the script that I used to created the dataset: 

https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/scripts/datasets/make_cifar10_gcn_whitened.py

After running this script, couple of files are generated, among which there are test.pkl, test.npy, train.pkl, train.npy

I dont know if there is a direct way for converting to leveldb/lmdb, so assuming there is no way, I need to be able to read each image and its corresponding label, so that I can then save it into a leveldb/lmdb database. I was told I need to use the pickle file for reading since it provides a dictionary like structure. However, trying to do

import cPickle as pickle
pickle.load( open( "N:/pylearn2-master/datasets/cifar10/pylearn2_gcn_whitened/test.pkl", "rb" ) )

outputs

<pylearn2.datasets.cifar10.CIFAR10 at 0xde605f8>

and I dont know what the correct way of accessing the items in a pickle file is and or whether I need to read from the numpy array directly.

Any help in this regard is extremely appreciated