nvrtcCompileProgram: NVRTC_ERROR_COMPILATION

[aaronbgr...@gmail.com]

Hello,

I wrote some GPU Ops for Theano a few months ago. In particular, "git show" on the version of Theano that I checked out returns:

commit d49b53682449a39175daf72724860024fd6b7b1f
Merge: 98c3625 bcf999f
Author: Frédéric Bastien <frederic...@gmail.com>
Date:   Fri Apr 7 09:52:30 2017 -0400

    Merge pull request #5820 from mike1808/patch-1
   
    fix theano.sparse.mul docs

At the time, I was using libgpuarray 0.6.3.

When I update to Theano 1.0.0rc1 and libgpuarray 0.7.4 and then go and try to compile my GPU Ops (i.e. as part of a Theano graph, by calling theano.function), I get the Nvidia compiler error, "nvrtcCompileProgram: NVRTC_ERROR_COMPILATION, " and no further information.

Now, I realize that libgpuarray 0.7.x is API and ABI incompatible with prior versions, so I expect that I will need to make some changes to my GPU Op code. However, typically this manifests itself as a compile error, where Theano points me to a C file that it's generated and is trying to compile. The file also usually includes some helpful compiler error message.

Has anyone seen this NVRTC compilation error before and / or have any advice for troubleshooting? I will continue looking through Theano documentation / code to see if I can find a solution. So far, Google searches have not been very helpful. There is one thread on this at https://github.com/Theano/Theano/issues/5943, which was eventually caused by old GPU hardware using an unsupported version of cuDNN.

For further background: I'm running this on Ubuntu 16.04, Python 2.7, Numpy 1.13, CUDA 8, and cuDNN 5.1. Updating to CUDA 9 / CUDNN 7 produces the same error. Reverting back to the Theano checkout cited above allows me to compile and run the GPU Ops again, though I have only tried this on CUDA 8 and not with the newer release. The GPU in question is a GTX 1060, though I can reproduce this problem on another machine that has a Titan X Pascal.

Any thoughts on this would be greatly appreciated. Thanks for your help.

Sincerely,
Aaron

[Pascal Lamblin]

If you have the option to check out libgpuarray from github and
recompile it with -DCMAKE_BUILD_TYPE=Debug, then you will get the code
for the cuda kernel as well as the error message from NVRTC.

Hth,

> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "theano-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to theano-dev+...@googlegroups.com
> <mailto:theano-dev+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Pascal Lamblin

[aaronbgr...@gmail.com]

Hi,

I figured it out: I had nested if statements in my GPU kernel, which for some reason caused the error. The first if statement is there so I can have a total execution grid size that's (slightly) larger than the number of elements I want to compute, and the nested if statement was a "mode" switch that makes the function do different things depending on the mode. (It's a pooling function - think max pooling vs sum / mean pooling)

I now have a GPU kernel for each mode, and it runs fine.

Best,
Aaron

> <mailto:theano-dev+unsub...@googlegroups.com>.