I have found a maybe connecting issue, i was updating just normal on some nodes from 4.3 to 4.4 meanwhile reading the patch notes i found out about the cuda thing. So i removed, because i was curious, the variable in the config.sh before updating and it worked without problem but i was confused because it said cudnn dependencies did not change so out of more curiosity i tried on another node to update with --override and then i get an error. So together:
maybe your multi-gpu setup gets cycles confused somehow. maybe its just using the RT cores of just one of your three GPUs for optix but all the cuda cores of your three GPUs for cuda. then your result might make sense.
Notes: Yes, there is the possibility to install it via apt-get install cuda. I strongly suggest not to use it, as it changes the paths and makes the installation of other tools more difficult.
I got it to work after reading several posts:I had an ATI card in the computer already which turned out to be very useful. I installed GTX 1070 along side of the ATI and started installing Kubuntu 16.04. Only the display connected to the ATI card had image initially, which allowed me to install the driver NVIDIA-Linux-x86_64-367.27.run downloaded from the vendor's website. To install CUDA, I downloaded the cuda_7.5.18_linux.run file. I installed the cuda toolkit by using two switches:
When I was trying to install "cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb" on Ubuntu 16.04, I simply followed the instructions here -installation-guide-linux/index.html#post-installation-actions. However, I was not able to compile cuda-install-samples-8.0.61.sh \homeor nvcc -V
Having done this multiple times, successfully/unsuccessfully loosing my display, coming here - gaining insights - some cuda libs not in path, missing , not installed - the sane way is to just install the linux drivers for your nvidia-card -the-right-nvidia-driver-for-cuda-in-ubuntu-2d9ade437decand work on nvidia-cuda docker images - base or devel.
Its helpful to run a LTS version of Ubuntu which happens every even numbered year as in 16.04, 18.04, 20.04, 22.0 etc ... because its less of a moving target for driver owners AND the cuda download site specifically only lists LTS versions
You also have to change your malloc/new and free/delete calls to cudaMallocManaged and cudaFree so that you are allocating space on the GPU. Finally, you need to wait for a GPU calculation to complete before using the results on the CPU, which you can accomplish with cudaDeviceSynchronize.
DPCT only needs CUDA headers from the supported versions. So, if you could manage to get headers from these supported versions (given above) and provide its path to the --cuda-include-path flag, there are pretty high chances that it could work, without having to downgrade your current CUDA version.
Whether to use strict mode in SkipLayerNormalization cuda implementation. The default and recommanded setting is false. If enabled, accuracy improvement and performance drop can be expected. This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)
df19127ead