How to create_imagenet with GPU?

[Marcella Astrid]

I am new to caffe and now I am trying to use caffe for scene recognition on Places2 dataset.

I am using Amazon Web Service GPU instance g2.2xlarge.

I use Ubuntu Server 14.04 LTS (HVM) - CUDA 6.5 CAFFE Python  AMI that I found in community AMI. 

I installed cuDNN using tutorial in this website https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)

This is the output of

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Thu_Jul_17_21:41:27_CDT_2014
Cuda compilation tools, release 6.5, V6.5.12

I modify create_imagenet.sh become create_places2.sh for making lmdb for Places2 dataset. I only change the directory and seems to have no problem with this.

Just in case, this is the part of create_places2.sh which I change

EXAMPLE=~/places2_lmdb
DATA=data/Places2_devkit/data
TOOLS=build/tools

TRAIN_DATA_ROOT=~/places2/Places2/train/train/
VAL_DATA_ROOT=~/places2/Places2/val/

I put the places2 images on other magnetic volume and I made another magnetic volume for saving the lmdb files.

When I run, the processing takes around 4 minutes 33 seconds for each 1000 image data. When I check the GPU utilization with

nvidia-smi -i 0 -l -q -d UTILIZATION

the output  is

==============NVSMI LOG==============

Timestamp                           : Sun Dec  6 07:57:44 2015
Driver Version                      : 340.46


Attached GPUs                       : 1
GPU 0000:00:03.0
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    GPU Utilization Samples
        Duration                    : 16.37 sec
        Number of Samples           : 99
        Max                         : 0 %
        Min                         : 0 %
        Avg                         : 0 %
    Memory Utilization Samples
        Duration                    : 16.37 sec
        Number of Samples           : 99
        Max                         : 0 %
        Min                         : 0 %
        Avg                         : 0 %
    ENC Utilization Samples
        Duration                    : 16.37 sec
        Number of Samples           : 99
        Max                         : 0 %
        Min                         : 0 %
        Avg                         : 0 %
    DEC Utilization Samples
        Duration                    : 16.37 sec
        Number of Samples           : 99
        Max                         : 0 %
        Min                         : 0 %
        Avg                         : 0 %

This is the output of command

nvidia-smi

+------------------------------------------------------+
| NVIDIA-SMI 340.46     Driver Version: 340.46         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P0    41W / 125W |     66MiB /  4095MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+


+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0      1798  build/tools/convert_imageset                          53MiB |
+-----------------------------------------------------------------------------+

I wonder if there is way to utilize GPU for create_imagenet.sh (in my case create_places2.sh) because it takes too long, I think. There are 8,097,967 images. If each 1000 images takes 4 min 33 sec, then it needs 25.5 days??

But this is my first time, is it really this long? And then if there is information I need to give, please don't bother to ask me.

Thank you~

[Marcella Astrid]

Update from myself~

changing the storage to provisioned IOPS help... but, still take long time... and after few iteration, it gets faster until around 1 min 20 second after 130000 data. Early iterations are around 3 half minutes. But seems not to get faster after that. Does it really need this long time?