NVRTC runtime include error with afcuda backend

[Mike Mullen]

Hello,

I am trying to run the FFT benchmark example on Ubuntu, CUDA Runtime 12.2 with the v3.8 release and am shown no complaints during compilation or linking. When linking against the afcuda backend (again, links just fine, running the fft_cuda executable in the build directory generated by cmake), I get this error when running the executable:

ArrayFire v3.8.3 (CUDA, 64-bit Linux, build 987d5675a)
Platform: CUDA Runtime 12.2, Driver: N/A
[0] NVIDIA GeForce RTX 3050 Laptop GPU, 4096 MB, CUDA Compute 8.6
terminate called after throwing an instance of 'af::exception'
  what():  ArrayFire Exception (Internal error:998):
In function compileModule
In file src/backend/cuda/compile_module.cpp:289
NVRTC Error(6): NVRTC_ERROR_COMPILATION
Log:
cuda_fp16.h(4): catastrophic error: cannot open source file "vector_functions.h"
  #include "vector_functions.h"
                               ^

1 catastrophic error detected in the compilation of "4164363248796891264".
Compilation terminated.

Not sure why it says Drive: N/A, but the output of nvidia-smi is  

NVIDIA-SMI 535.54.06              Driver Version: 536.40       CUDA Version: 12.2

In /usr/local/cuda/include, both cuda_fp16.h and vector_functions.h exist. 

Has anyone seen something similar? Any thoughts on how to resolve this? I haven't changed any of the CMakeLists.txt files in the ArrayFire source. 

Thanks for any suggestions!

Mike

[Umar Arshad]

looks like an additional header was added to cuda_fp16.h file. This file was not added in the array here. This needs to be modified so to take this into account.

If you want to take a stab at it you can make the necessary changes to that file and create a PR. Or you can wait for a couple of days for me to make these changes. You will need to use the master branch if you want to take advantage of these changes. Its probably easier to downgrade to the previous CUDA toolkit to fix this issue.

Umar

[Mike Mullen]

I'll see if I can get master to run with your suggested change, thanks!

[Mike Mullen]

Strangely, after merging a solution for the original problem, updating my local branch, and rebuilding, the original problem is resolved, but resurfaces with the following simple program (where I am just experimenting with the API, directly setting an element, I recognize this is recommended against and I don't expect to do it often if at all. That being said, I don't think it should cause an error in this program assuming I am not doing something obviously incorrectly to a regular ArrayFire user):

#include <arrayfire.h>

using namespace af;

int main(int argc, char** argv) {

    int device = argc > 1 ? atoi(argv[1]) : 0;

    setDevice(device);

    info();

    std::vector<float> v;

    v.resize(4, 1.5);

    af::array A(2, 2, v.data());

    A(0,0) = 10.0f;

    af_print(A);

    return 0;

}

Commenting out the A(0,0) = 10.0f; line does not yield the error regarding cuda_fp16 and vector_functions.h. Is this a simple usage issue on my end? 

Here is my CMakeLists.txt:

cmake_minimum_required(VERSION 3.0)

project(GitHubIssues

  VERSION 0.0.1

  LANGUAGES CXX)

find_package(ArrayFire REQUIRED)

add_executable(testIssue afAPItest.cpp)

target_compile_features(testIssue PRIVATE cxx_std_11)

target_link_libraries(testIssue ArrayFire::afcuda)

and the output of info(): 

ArrayFire v3.9.0 (CUDA, 64-bit Linux, build 1eb6bcaef)

Platform: CUDA Runtime 12.2, Driver: N/A
[0] NVIDIA GeForce RTX 3050 Laptop GPU, 4096 MB, CUDA Compute 8.6

[Mike Mullen]

Looking at compile_modules.cpp a bit more, is this program being compiled with sourceIsJit=true maybe? I tried adding vector_functions.h to the header list in the linked block with no luck though (and rebuilding/linking arrayfire and the test program).

[Umar Arshad]

Can you try deleting everything in the ~/.arrayfire folder. It may be holding some cached kernels that are not properly compiled.

Umar

[Mike Mullen]

Was out of town, my apologies for the delay. Tried this, and searched everywhere on my path for anything related to arrayfire. No luck yet, will keep debugging locally and update in the next week or two if I make any progress. Thanks!

[Mike Mullen]

Ah, I read the new error too fast. I read "vector_functions.h" but it is now not finding "vector_types.h"

ArrayFire v3.9.0 (CUDA, 64-bit Linux, build d74656f7a)

Platform: CUDA Runtime 12.2, Driver: N/A
[0] NVIDIA GeForce RTX 3050 Laptop GPU, 4096 MB, CUDA Compute 8.6

ArrayFire Exception (Internal error:998):
In function compileModule

In file src/backend/cuda/compile_module.cpp:299
NVRTC Error(6): NVRTC_ERROR_COMPILATION
Log:
cuda_fp16.h(3): catastrophic error: cannot open source file "vector_types.h"
  #include "vector_types.h"
                           ^

1 catastrophic error detected in the compilation of "380948595291791473".
Compilation terminated.

I made similar changes as my earlier PR (accounting for the fact that vector_types.h is already in includeNames and was a dummy entry in sourceStrings) and cleared my ~/.arrayfire/ directory. Still get this error when running examples/getting_started/vectorize_cuda. Any guesses on things to try? 

[Mike Mullen]

Posted too soon, these are taking the sourceIsJit=true branch I mentioned earlier. Adding vector_types and vector_functions to those headers fixes it. Will open a PR today or tomorrow. Thank you!

[John Melonakos]

Great news, Mike!  Thanks for the update and the upcoming PR fix.

-John