syncedmem.cpp:57] Check failed: error == cudaSuccess (3 vs. 0) initialization error

[Anirban Laha]

Hi All,

This is the caffe code(I am using bvlc caffenet pretrained version) i have in python for GPU initialization:

        self.algorithm_name = self.__class__.__name__

        self.url = 'http://caffe.berkeleyvision.org/model_zoo.html'

        self.net = caffe.Classifier(model_definition_file, model_weights_file,

            mean=np.load(image_mean_file).mean(1).mean(1),

            channel_swap=(2, 1, 0),

            raw_scale=255,

            image_dims=(256, 256))

        if mode == 'gpu':

            caffe.set_mode_gpu()

        else:

            caffe.set_mode_cpu()

        self.layer_names = self.net.blobs.keys()

This works fine when I set the mode to 'cpu', but when i set mode as 'gpu', i get the following error:

E0212 04:12:06.207414 23443 upgrade_proto.cpp:618] Attempting to upgrade input file specified using deprecated V1LayerParameter: /dccstor/anirlaha1/caffemodels/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel

I0212 04:12:06.480501 23443 upgrade_proto.cpp:626] Successfully upgraded file specified using deprecated V1LayerParameter

F0212 04:26:27.164417 24870 syncedmem.cpp:57] Check failed: error == cudaSuccess (3 vs. 0)  initialization error

*** Check failure stack trace: ***

F0212 04:26:27.199898 24871 syncedmem.cpp:57] Check failed: error == cudaSuccess (3 vs. 0)  initialization error

*** Check failure stack trace: ***

Would be great if somebody can provide some pointers to resolve this issue.

I am using cuda-7.5 (latest) version.

Thanks,

Anirban.

[Jan C Peters]

As the message says, there is some error in initializing CUDA. To make sure of that try "caffe device_query -gpu 0". It should fail with a similar error.

To solve the problem, (re)install the (most recent) proprietary NVidia driver. I assume that you do have a CUDA-capable NVidia card. Caffe does not work with other GPUs. Are you on Linux?

Jan

[Anirban Laha]

Yes I am on Linux. Can you please point to the installation steps on Linux?

[Jan C Peters]

Well, the most convenient and well-configured installation method highly depends on the specific distro you are using.

If you are using a Debian-derivative (like Ubuntu) probably the best way is to install the nvidia repo through the deb-file provided on https://developer.nvidia.com/cuda-downloads, and then install the complete cuda package (as described there) or just the nvidia driver (the package name is nvidia-352 or nvidia-current).

If you are on a RPM-based distro like RHEL, Fedora, CentOS, ..., a nice way to install is using the akmods architecture: install the akmod-nvidia package from the RPMFusion nonfree repositories. This will automatically rebuild the driver every time you install a new kernel.

If you are on a laptop with Optimus technology then the situation is more complicated. So, what is your system?

Jan

[Jan C Peters]

I should add, that on every Linux distro you can also just download the run-file from https://developer.nvidia.com/cuda-downloads, which is distro-agnostic (although the download website suggests it to be distro-specific). This is just a large binary executable, install from the commandline and be done with it. But usually the other installation methods I pointed out are better suited to the platform in question, in particular as far as updates of either the driver itself or the kernel are concerned.

Jan

[Olivia W]

I am running into a similar issue on prediction. However, when trying "caffe device_query -gpu 0" that works fine.

Also training/testing when building the net works fine on the gpu. It's only when I try to predict from a python script using set_mode_gpu() that it fails.

[Jan C Peters]

With a CUDA-specific error? I cannot think of a cause for that to fail as long as caffe device_query -gpu 0 works.

Jan

[xie...@gmail.com]

Anybody has solution for this wired problem?

在 2016年2月12日星期五 UTC+8下午5:52:41，Anirban Laha写道：

[Siyang Qin]

Run the script with sudo. Hope it helps.