Skip to first unread message
Alexey Abramov
Apr 19, 2016, 10:59:46 AM4/19/16
to Caffe Users
Hello everyone,
I'm running the Caffe library on GeForce GTX Titan X with CUDA 7.5 (Ubuntu 14). I'm not sure whether Caffe is properly configured for my setup. My dataset consists of images with 256 x 256 pixels (3 channels), 100000 training / 10000 test samples. For the very first test I'm using AlexNet with new_height=256, new_width=256, crop_size=227. Running 1000 training iterations on one Titan X with batch_size=256 takes about 17 minutes... Is it not too slow for this hardware?
Any help and advices are kindly appreciated!
Many thanks in advance!
Cheers,
Alexey
Andriy Lysak
Apr 19, 2016, 12:14:12 PM4/19/16
to Caffe Users
I run the same type of network on a GTX 680 and it takes ~3 min for a 100 training iterations. So seems like your card is a lot faster. Another trick you can try is using OpenBLAS instead of Atlas. If you are using Atlas that might bottle neck your card.
Regards,
Andriy
Andriy
Jan
Apr 21, 2016, 3:28:03 AM4/21/16
to Caffe Users
Atlas, MKL and OpenBLAS are CPU-level BLAS libraries, i.e. if you use GPU mode you do not use any of these libraries. The GPU-level BLAS package that is used by default is cublas and (if installed and compiled with) cuDNN.
I don't have the hardware to compare that timing to, but if you look at the reported reference timings (http://caffe.berkeleyvision.org/performance_hardware.html), your performance does not look so bad. I'd say there is nothing wrong or to worry about with your setup.
Jan
I don't have the hardware to compare that timing to, but if you look at the reported reference timings (http://caffe.berkeleyvision.org/performance_hardware.html), your performance does not look so bad. I'd say there is nothing wrong or to worry about with your setup.
Jan
Alexey Abramov
Apr 22, 2016, 8:07:16 AM4/22/16
to Caffe Users
Hi Jan, thanks for your reply!
Daniela G
Apr 22, 2016, 8:19:11 AM4/22/16
to Caffe Users
Hi!
The Makefile.config file has Atlas as default. So if we want to use GPU, do we have to define BLAS, BLAS_INCLUDE and BLAS_LIB fields with cublas? Or when we choose GPU mode, the cublas will be selected automatically?
The Makefile.config file has Atlas as default. So if we want to use GPU, do we have to define BLAS, BLAS_INCLUDE and BLAS_LIB fields with cublas? Or when we choose GPU mode, the cublas will be selected automatically?
Jan
Apr 25, 2016, 9:27:45 AM4/25/16
to Caffe Users
No no, the BLAS, BLAS_INCLUDE and BLAS_LIB switches are only for the CPU BLAS, i.e. the BLAS to be used when caffe is run in CPU mode. In GPU mode caffe uses cublas by default. If cuDNN is available (and the USE_CUDNN switch was set at compilation time), then caffe uses cuDNN in GPU mode (at least for all layers that have a cudnn implementation, but most of them are implemented in cuDNN afaik).
Jan
Jan
Daniela G
Apr 29, 2016, 4:56:38 AM4/29/16
to Caffe Users
Ok! Thank you.
Reply all
Reply to author
Forward
0 new messages