How to prepare CIFAR100 for Caffe?

[Hossein Hasanpour]

Hello All, I'm trying to run some test on CIFAR100 as well.
I started by changing the CIFAR10 scripts and tried to download the CIFAR100 dataset. This is the scripts content :

#!/usr/bin/env sh
# This scripts downloads the CIFAR100 (binary version) data and unzips it.

DIR="$( cd "$(dirname "$0")" ; pwd -P )"
cd $DIR

echo "Downloading..."

wget --no-check-certificate https://www.cs.toronto.edu/~kriz/cifar-100-binary.tar.gz

echo "Unzipping..."

tar -xf cifar-100-binary.tar.gz && rm -f cifar-100-binary.tar.gz
mv cifar-100-batches-bin/* . && rm -rf cifar-100-batches-bin

# Creation is split out because leveldb sometimes causes segfault
# and needs to be re-created.

echo "Done."

After executing it, this is what I get :

hossein@hossein:/media/hossein/tmpstore/caffe_new$ ./data/cifar100/get_cifar100.sh
Downloading...
wget: /home/hossein/anaconda2/lib/libcrypto.so.1.0.0: no version information available (required by wget)
wget: /home/hossein/anaconda2/lib/libssl.so.1.0.0: no version information available (required by wget)
--2016-03-22 09:46:45--  https://www.cs.toronto.edu/~kriz/cifar-100-binary.tar.gz
Resolving www.cs.toronto.edu (www.cs.toronto.edu)... 128.100.3.30
Connecting to www.cs.toronto.edu (www.cs.toronto.edu)|128.100.3.30|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 168513733 (161M) [application/x-gzip]
Saving to: ‘cifar-100-binary.tar.gz’

100%[=======================================================>] 168,513,733 79.0KB/s   in 32m 12s

2016-03-22 10:19:00 (85.2 KB/s) - ‘cifar-100-binary.tar.gz’ saved [168513733/168513733]

Unzipping...
mv: cannot stat ‘cifar-100-batches-bin/*’: No such file or directory
Done.

As you see , the download is completed successfully, the files are extracted successfully as well,
but there is an error stating the move command failed! which I have no idea what its talking about!
mv: cannot stat ‘cifar-100-batches-bin/*’: No such file or directory
Here this is the contents of the cifar-100 contents after extraction

https://lh3.googleusercontent.com/-ISYLXrFdMo8/VvDoH1-7S0I/AAAAAAAAAbs/Am0Sq_W0NcEUM4d9hPNG2DBP4k4-ts7mg/s1600/Cifar100_Extracted_Contents.png

After that, I tried to create the cifar lmdb like this :

#!/usr/bin/env sh
# This script converts the cifar data into leveldb format.

EXAMPLE=examples/cifar100
DATA=data/cifar100
DBTYPE=lmdb

echo "Creating $DBTYPE..."

rm -rf $EXAMPLE/cifar100_train_$DBTYPE $EXAMPLE/cifar100_test_$DBTYPE

./build/examples/cifar100/convert_cifar_data.bin $DATA $EXAMPLE $DBTYPE

echo "Computing image mean..."

./build/tools/compute_image_mean -backend=$DBTYPE \
  $EXAMPLE/cifar100_train_$DBTYPE $EXAMPLE/mean.binaryproto

echo "Done."

Which gives me this :

hossein@hossein:/media/hossein/tmpstore/caffe_new$ ./examples/cifar100/create_cifar100.sh
Creating lmdb...
./examples/cifar100/create_cifar100.sh: 12: ./examples/cifar100/create_cifar100.sh: ./build/examples/cifar100/convert_cifar_data.bin: not found
Computing image mean...
./build/tools/compute_image_mean: /home/hossein/anaconda2/lib/liblzma.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libunwind.so.8)
F0322 10:57:26.664922  4866 db_lmdb.hpp:14] Check failed: mdb_status == 0 (2 vs. 0) No such file or directory
*** Check failure stack trace: ***
    @     0x7f13eb383daa  (unknown)
    @     0x7f13eb383ce4  (unknown)
    @     0x7f13eb3836e6  (unknown)
    @     0x7f13eb386687  (unknown)
    @     0x7f13eb86423e  caffe::db::LMDB::Open()
    @           0x402074  main
    @     0x7f13ea38fec5  (unknown)
    @           0x402aba  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)
Done.

Whats wrong here? How should I go about this?
Id be grateful to know how to get this done.
thank in advance

[Jan]

The root cause seems to be that the convert_cifar_data executable is missing. It is usually built along with all the other caffe binaries. So you just need to build caffe successfully and the problem should vanish. The other errors are just follow-ups caused by this error.

Jan

[Hossein Hasanpour]

oh, yes, I just found out thanks to you.
so before that I need to change the convert_cifar_data source code right and then compile it? what exactly should I change there?
Can I simply use the cifar10s executable ?

[Jan]

I am not so familiar with the CIFARs, so I don't know if there is any difference in the formatting of the data. Just try it as-is; if it doesn't work, look into it and maybe adjust the code of the convert_cifar_data.

Jan

[Hossein Hasanpour]

Thanks again for your time, I really appreciate it :)