Unsupported gpu architecture 'compute_20'

[Elavarasan Rajathurai]

Hello,

Good day.

I have an issue with installing Caffe into Ubuntu.

I summarized the issues below.

Background:

I tried to install Caffe into Ubuntu 16.04 by following below guidelines:

https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-Installation-Guide

I have made minor changes as below:

1. 

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

 

Instead, I use the below command:

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/aarch64-linux-gnu /usr/lib/aarch64-linux-gnu/hdf5/serial

2. 

 CUDA_DIR := /usr/local/cuda-8.0

Instead, I use the below command:

CUDA_DIR := /usr/local/cuda-9.0

Issue:

Everything goes fine until I encountered some errors when I execute 'make all' command.

Next, build the Caffe with the following command.

make all
make test
make runtest
make pycaffe      -should be finished already, so you can omit this one
make distribute

Error Message:

NVCC src/caffe/layers/base_data_layer.cu

nvcc fatal   : Unsupported gpu architecture 'compute_20'

Makefile:594: recipe for target '.build_release/cuda/src/caffe/layers/base_data_layer.o' failed

make: *** [.build_release/cuda/src/caffe/layers/base_data_layer.o] Error 1

Question:

What could cause this issue? Anyone could advise me?

I have stated my Hardware/Software as below for your reference.

SYSTEM Version Info

 NVIDIA Jetson TX2

 L4T 28.2.1 [ JetPack 3.3 or 3.2.1 ]

 Board: t186ref

 Ubuntu 16.04.6 LTS

 Kernel Version: 4.4.38-tegra

 CUDA 9.0.252

 Python 3.5.2

 TensorFlow 1.14.0

 OpenCV 4.1.0

 Keras 2.3.1

 pyyaml 5.1.2

 pydot 1.4.1

 pyparsing 2.4.2

numpy 1.16.1

scipy 0.17.0

cython 0.29.14

scikit learn 0.21.3

[E Merth]

In Makefile.config find this code:

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
-gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_61,code=compute_61

Change it to this:

CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_52,code=sm_52 \
                -gencode arch=compute_60,code=sm_60 \
                -gencode arch=compute_61,code=sm_61 \
                -gencode arch=compute_61,code=compute_61

Cuda is complaining that it does not know about a rather old GPU architecture (Fermi), that's all.

If you are running a Telsa or Volta based card then you probably want to add new lines so it looks like this:

CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \

                -gencode arch=compute_35,code=sm_35 \

                -gencode arch=compute_50,code=sm_50 \

                -gencode arch=compute_52,code=sm_52 \

                -gencode arch=compute_60,code=sm_60 \

                -gencode arch=compute_61,code=sm_61 \

                -gencode arch=compute_70,code=compute_70 \

                -gencode arch=compute_75,code=compute_75

Please understand that in the 70 & 75 lines it may be "sm_70" and "sm_75" for the code values. I've not built in a while and I might have remembered this detail incorrectly.

[E Merth]

I see now you are using a Jetson TX2. 

Change your CUDA_ARCH to this:

CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \

                -gencode arch=compute_35,code=sm_35 \

                -gencode arch=compute_50,code=sm_50 \

                -gencode arch=compute_52,code=sm_52 \

                -gencode arch=compute_60,code=sm_60 \

                -gencode arch=compute_61,code=sm_61 \

                -gencode arch=compute_62,code=sm_62 \

                -gencode arch=compute_70,code=compute_70 \

                -gencode arch=compute_75,code=compute_75

Because compute_62, sm_62 are the indicators for the TX2.

On Tuesday, 25 February 2020 12:28:51 UTC-7, Elavarasan Rajathurai wrote:

[Elavarasan Rajathurai]

Hello E Merth,

Thank you for your strong support.

However, when I change CUDA_ARCH as below, I receive another error message.

CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \

                -gencode arch=compute_35,code=sm_35 \

                -gencode arch=compute_50,code=sm_50 \

                -gencode arch=compute_52,code=sm_52 \

                -gencode arch=compute_60,code=sm_60 \

                -gencode arch=compute_61,code=sm_61 \

                -gencode arch=compute_62,code=sm_62 \

                -gencode arch=compute_70,code=compute_70 \

                -gencode arch=compute_75,code=compute_75

The error message I received is as below:

NVCC src/caffe/layers/base_data_layer.cu

nvcc fatal   : Unsupported gpu architecture 'compute_75'

Makefile:594: recipe for target '.build_release/cuda/src/caffe/layers/base_data_layer.o' failed

make: *** [.build_release/cuda/src/caffe/layers/base_data_layer.o] Error 1

In this case, do I need to change compute_75 to sm_75?

Please advise.

Thank you.

Regards,

Ela

[E Merth]

Hmmm. You are using CUDA 9. So I suggest you remove the 75 line and also the 70 line. CUDA 9 does not know about Volta or Tesla.

My apologies I thought you are using CUDA 10, not CUDA 9.

The basic problem you have is that your CUDA version does not know about all the architectures listed in CUDA_ARCH variable. Caffe is old enough that it has entries for deprecated Nvidia architectures (20, 21), and your CUDA version is old enough that it does not know about new  Nvidia architectures (70, 75).

The variable CUDA_ARCH in Makefile.config lists the GPU architectures you want Caffe to be able to run on. Each architecture has two values (gencode, code). You need the entry for TX2. You may have to experiment with the other values.

[Elavarasan Rajathurai]

Hello Merth, Thank you so much for your explanation and support.

I removed 75 line and 70 line and it could run now.

However, I have another error message as below:

NVCC src/caffe/layers/base_data_layer.cu

./include/caffe/util/cudnn.hpp(112): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000400d_00000000-4_base_data_layer.cpp4.ii".

Makefile:594: recipe for target '.build_release/cuda/src/caffe/layers/base_data_layer.o' failed

make: *** [.build_release/cuda/src/caffe/layers/base_data_layer.o] Error 1

Sorry for keep on asking you questions, but could you help?

Thank you.

[E Merth]

I'm not sure about that specific error, Elavarasan, but general knowledge tells me that there is a problem with the version or installation of Nvidia CUDNN toolkit on your computer.

You could try changing Caffe's Makefile.config and setting USE_CUDNN := 0

Do you have CUDNN installed? On x86 Linux installing CUDNN is done by downloading a .deb or .rpm file and installing it. I have never used a TX2, but googling I found these instructions:

>>   https://devtalk.nvidia.com/default/topic/1044642/jetson-tx2/how-to-install-cudnn-on-nvidia-jetson-tx2/

Good luck!

[E Merth]

A bit more comment:

Setting USE_CUDNN := 0 tells Caffe to not use CUDNN at all. Not using CUDNN will make Caffe a bit slower, but it will still be GPU accelerated.

If Caffe compiles after setting USE_CUDNN := 0 then you know you just have to either install CUDNN or figure out why your nvcc doesn't like your copy of CUDNN.

If nvcc does not like your copy there are many possible reasons and I am not the guy to help you there because I don't know a lot about CUDNN. But asking in the nVidia forum https://devtalk.nvidia.com/default/board/188/ would be a good idea as they are knowledgeable about the TX2  :-)

On Wednesday, 26 February 2020 16:19:31 UTC-7, Elavarasan Rajathurai wrote:

[Elavarasan Rajathurai]

Hello E Merth,

Finally I flash my TX2 again and install Caffe through below procedure:

https://devtalk.nvidia.com/default/topic/1072504/caffe-for-jetson-tx2-with-jetpack-4-3/?offset=2#5433342

Thank you so much for your support.