caffe.set_mode_cpu() still use gpu?

437 views
Skip to first unread message

Yucheng Zhu

unread,
Apr 12, 2017, 5:43:25 PM4/12/17
to Caffe Users
Hi Guys,

I've been struggling with this weird caffe behaviour for days:
I have a Ubuntu 16.04 machine with a Nvidia GTX 1080( driver version 375.39, cuda tool kit 8.0). The problem is, when I open the python and do "caffe.set_mode_cpu()"
I found caffe is using GPU. Why? 

Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> caffe.set_mode_cpu()
>>> 


+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39                 Driver Version: 375.39                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:02:00.0     Off |                  N/A |
| 33%   28C    P8    12W / 190W |    152MiB /  8114MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  NVS 315             Off  | 0000:03:00.0     N/A |                  N/A |
| 30%   38C    P8    N/A /  N/A |     44MiB /   956MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     12821    C   python                                         149MiB |
|    1                  Not Supported                                         |
+-----------------------------------------------------------------------------+

My collegue first found this when he tried to run multiple tasks on CPU. The above simple python example is just how I recreate the issue.

Here is my Makefile.config that I used to compile the caffe( newest version pulled from git )

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#       You should not set this flag if you will be reading LMDBs with any
#       possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda-8.0-no-cuDNN
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
                -gencode arch=compute_20,code=sm_21 \
                -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_52,code=sm_52 \
                -gencode arch=compute_60,code=sm_60 \
                -gencode arch=compute_61,code=sm_61 \
                -gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
 MATLAB_DIR := /usr/local/MATLAB/R2016b
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
                /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
                # $(ANACONDA_HOME)/include/python2.7 \
                # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @
                                                                                                                                                                                          113,1         Bot
Can anyone help us?

Yucheng

Hieu Do Trung

unread,
Apr 13, 2017, 2:39:27 AM4/13/17
to Caffe Users
# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

-> This might be the problem?

Yucheng Zhu

unread,
Apr 13, 2017, 11:51:43 AM4/13/17
to Caffe Users
Hi, How it can be the problem? it is commented out so we are not using CPU_ONLY mode. We want to freely switch from cpu and gpu mode.

Evan Shelhamer

unread,
Apr 14, 2017, 1:37:02 AM4/14/17
to Yucheng Zhu, Caffe Users
For legacy reasons when Caffe is compiled with GPU capability it loads certain libraries like cuBLAS and cuRAND even when in CPU mode. I admit it is unexpected and annoying, but when in CPU mode there won't be any further usage than a little bit of occupied memory for these libraries.

Evan Shelhamer





--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/2477d95a-0f38-490a-9cf6-8ce5636bacd1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Yucheng Zhu

unread,
Apr 14, 2017, 11:06:57 AM4/14/17
to Caffe Users
Hi, Evan,

Thanks for replying. So in my case it is using about 100 my gpu memory. If I want to run 10 parallel caffe CPU jobs( I have a powerful CPU), then it is better to compile a CPU only caffe? Otherwise I would lose 1000mb memory on GPU?

Yucheng

Volodymyr Turchenko

unread,
Apr 17, 2017, 1:28:07 PM4/17/17
to Caffe Users
Hi Evan,

thanks for replying.
it is something wrong with some libraries. I experienced this issue ONLY now in these conditions described by Yucheng above.
I had installed Caffe version on April 2016 on Ubuntu 16.04 machine and I had on that machine Titan X and everything worked perfectly, I was able to run multiple CPU cores and GPU simultaneously without any issues. Probably we should use previous cuda toolkit 7.5 version?

How it is working now, when we run on let say 8 cpus - it is not just using memory inside GPU, I think it actually runs it inside GPU 8 times, so all 8 cpus are running + GPU (probably 8 times inside). It

tremendously slows down the whole machine, a computational time is too long for a really small model in CPU mode, so it does not make sense to run tasks in CPU mode, but I want to use all CPUs I have + GPU, because I am running many experiments. So, any idea (except to install two separate versions of Caffe - for CPU and GPU modes independently) how to help us? Thanks!


Cheers,
Vlad

пʼятниця, 14 квітня 2017 р. 11:06:57 UTC-4 користувач Yucheng Zhu написав:
Reply all
Reply to author
Forward
0 new messages