[llvm-dev] Debug info for CUDA code

[treinz via llvm-dev]

Hi all,

I found this http://lists.llvm.org/pipermail/llvm-dev/2017-November/118871.html when googling about compiling CUDA code using llvm. Is it still the case that one can't step into CUDA kernel code compiled by llvm in cuda-gdb? I'm using clang 7.0.

Thanks,

Char

 

[Valentin Churavy via llvm-dev]

AFAIK there is work under way from IBM (couple of PRs open), but yes Clang 7.0 doesn't produce the necessary information.

-V

_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Eric Christopher via llvm-dev]

Adding Alexey here who has been driving this effort in llvm.

There are about 5 patches waiting on my review:

   -: https://reviews.llvm.org/D54320

   -: https://reviews.llvm.org/D46189

   -: https://reviews.llvm.org/D51554

   -: https://reviews.llvm.org/D46061

   -: https://reviews.llvm.org/D45784

After which I think we're good.

-eric

[Alexey Bataev via llvm-dev]

Hi, yes, these patches are required to have tne basic debug info support. At least 2 more patches are required to be able to get values of the variables without additional manual data type casting in cuda-gdb. 

Best regards,

Alexey Bataev

4 дек. 2018 г., в 18:06, Eric Christopher <echr...@gmail.com> написал(а):

[Alexey Bataev via llvm-dev]

Hi Char, sorry, you need to wait a little bit. It is almost ready, but not yet. For now, use nvcc if you need debug info for Cuda.

-------------
Best regards,
Alexey Bataev

14.12.2018 14:33, treinz пишет:

Hi Alex, Eric and Valentin,

Thanks for the information. I don't mean to push this but I'm in desperate need of debugging some cuda code. I'm not familiar with the llvm internal but it sounds like there's at least line info now, right? If so, can you point me to a branch of llvm that can help tracing the bug down to certain line of code. I believe my bug is simply a write/read out of array boundary.

Thanks,

Char

[Alexey Bataev via llvm-dev]

Actually, we need to commit just 2 more patches to enable basic debug info support for NVPTX. So, I hope, it is going to be the part of the next 8.0 release. It won't be backported to 7.0 definitely.

Sure, I will let you know when the patches are committed to the trunk, no problems.

Best regards,

Alexey Bataev

14 дек. 2018 г., в 16:55, treinz <tre...@163.com> написал(а):

Are you planning to release this as soon as it's ready or you want to make it into a major release? Is it possible to let me know (maybe by replying to this thread) once the code is ready? I know sometimes it takes a while to get things in the major release. I greatly appreciate your work on this!

Thanks,

Char

[Valentin Churavy via llvm-dev]

If you only need line-information that information is already present and should be useful. As long as you set `-g` you should get some and there is a flag to turn ptxas optimisation off to preserve as much debuginformation as possible.

You then can also run your code under cuda-memcheck

On Fri, Dec 14, 2018, 16:55 treinz <tre...@163.com wrote:

Are you planning to release this as soon as it's ready or you want to make it into a major release? Is it possible to let me know (maybe by replying to this thread) once the code is ready? I know sometimes it takes a while to get things in the major release. I greatly appreciate your work on this!

Thanks,

Char

在 2018-12-15 05:19:50，"Alexey Bataev" <a.ba...@outlook.com> 写道：

[Eric Christopher via llvm-dev]

It'll be in within a week :)

[treinz via llvm-dev]

Hi Alex, Eric and Valentin,

Thanks for the information. I don't mean to push this but I'm in desperate need of debugging some cuda code. I'm not familiar with the llvm internal but it sounds like there's at least line info now, right? If so, can you point me to a branch of llvm that can help tracing the bug down to certain line of code. I believe my bug is simply a write/read out of array boundary.

Thanks,

Char

在 2018-12-05 07:15:28，"Alexey Bataev" <a.ba...@hotmail.com> 写道：

[treinz via llvm-dev]

Are you planning to release this as soon as it's ready or you want to make it into a major release? Is it possible to let me know (maybe by replying to this thread) once the code is ready? I know sometimes it takes a while to get things in the major release. I greatly appreciate your work on this!

Thanks,

Char

在 2018-12-15 05:19:50，"Alexey Bataev" <a.ba...@outlook.com> 写道：

[treinz via llvm-dev]

Hi Valentin,

I haven't been able consistently to get line information of device code with `-g -O0`, sometimes `cuda-memcheck` can tell which line triggers the error but not always. `cuda-gdb` can only trace back to the kernel launch. Is `--cuda-noopt-device-debug` the option you mentioned to turn off ptxas optimisation? What's the difference between that and simply `-O0`?

Thanks for helping me out!

[treinz via llvm-dev]

Hi Alexey,

I tried the b7195a6 from the llvm github mirror, which does include your commit D46189 (see below). But it can't compile the example program on https://llvm.org/docs/CompileCudaWithLLVM.html#invoking-clang -- the error is:

[aznb@asp testcuda]$ clang++ axpy.cu -g -o axpy --cuda-gpu-arch=sm_61 -lcudart_static -ldl -lrt -pthread -v

clang version 9.0.0 (https://git.llvm.org/git/clang.git 4566ed0d7c076ad061168f7c6cb5fa3a793614a8) (https://git.llvm.org/git/llvm.git b7195a6d8d9064c4fac6a7b91826de80df394f69)

Target: x86_64-unknown-linux-gnu

Thread model: posix

InstalledDir: /home/aznb/.linuxbrew/bin

Found candidate GCC installation: /home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0

Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.4

Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.7

Selected GCC installation: /home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0

Candidate multilib: .;@m64

Selected multilib: .;@m64

Found CUDA installation: /net/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64, version 10.0

 "/home/aznb/mycodes/testcuda -ferror-limit 19 -fmessage-length 0 -pthread -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -o /tmp/59224042.1.login.q/axpy-560f45.s -x cuda axpy.cu

clang -cc1 version 9.0.0 based upon LLVM 9.0.0svn default target x86_64-unknown-linux-gnu

ignoring nonexistent directory "/include"

ignoring nonexistent directory "/include"

ignoring duplicate directory "/home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0"

ignoring duplicate directory "/home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/x86_64-unknown-linux-gnu"

ignoring duplicate directory "/home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/backward"

ignoring duplicate directory "/usr/local/include"

ignoring duplicate directory "/home/aznb/.linuxbrew/Cellar/llvm/HEAD-b7195a6/lib/clang/9.0.0/include"

ignoring duplicate directory "/usr/include"

#include "..." search starts here:

#include <...> search starts here:

 /home/aznb/.linuxbrew/Cellar/llvm/HEAD-b7195a6/lib/clang/9.0.0/include/cuda_wrappers

 /net/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/include

 /home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0

 /home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/x86_64-unknown-linux-gnu

 /home/aznb/.linuxbrew/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/backward

 /usr/local/include

 /home/aznb/.linuxbrew/Cellar/llvm/HEAD-b7195a6/lib/clang/9.0.0/include

 /usr/include

End of search list.

 "/net/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/bin/ptxas" -m64 -g --dont-merge-basicblocks --return-at-end -v --gpu-name sm_61 --output-file /tmp/59224042.1.login.q/axpy-1ebb06.o /tmp/59224042.1.login.q/axpy-560f45.s

ptxas /tmp/59224042.1.login.q/axpy-560f45.s, line 290; fatal   : Parsing error near '{': syntax error

ptxas fatal   : Ptx assembly aborted due to errors

clang-9: error: ptxas command failed with exit code 255 (use -v to see invocation)

Maybe it's related to the illegal assembly code mentioned in https://reviews.llvm.org/D46189 Any idea what's going on? I'm including other people to see if they have any suggestion. https://reviews.llvm.org/D46189 also mentioned that this is fixed but has the fixed been merged in the trunk yet?

BTW, github mirror's b7195a6 does include your commit about the debug info on nvptx:

commit d77902a49a06890b97e33515d346c6e4df6b756b

Author: Alexey Bataev <a.ba...@hotmail.com>

Date:   Tue Jan 22 17:43:37 2019 +0000

    [DEBUGINFO, NVPTX] Enable support for the debug info on NVPTX target.

    Summary: Enable full support for the debug info.

    Reviewers: echristo

    Subscribers: jholewinski, aprantl, JDevlieghere, llvm-commits

    Differential Revision: https://reviews.llvm.org/D46189

    git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351846 91177308-0d34-0410-b5e6-96231b3b80d8

Thanks,

Char

在 2019-01-23 07:59:12，"Alexey Bataev" <a.ba...@outlook.com> 写道：

Hi Char, I don't know how often it is updated, but most probably it is already updated.

Best regards,

Alexey Bataev

22 янв. 2019 г., в 17:46, treinz <tre...@163.com> написал(а):

Hi Alexey,

Thank you for letting me know. Can I assume the the github mirror is bumped to 9.0 and should contain the commit you mentioned? Just want to make sure I'm looking at the right repo.

Thanks,

Char

在 2019-01-23 01:46:44，"Alexey Bataev" <a.ba...@outlook.com> 写道：

Hi, debug info for the NVPTX is committed. It is not a part of 8.0 RC1, unfortunately, only in 9.0 trunk.

-------------
Best regards,
Alexey Bataev

14.12.2018 17:08, Alexey Bataev пишет:

[Alexey Bataev via llvm-dev]

Hi Char, could you send the PTX file?

-------------
Best regards,
Alexey Bataev

23.01.2019 13:37, treinz пишет:

[Alexey Bataev via llvm-dev]

Ok, I found the problem. The patch, that enables debug info, was not applied completely. Will fix this soon.

-------------
Best regards,
Alexey Bataev

23.01.2019 13:37, treinz пишет:

Hi Alexey,

[Alexey Bataev via llvm-dev]

Hi Char, I found the problem, for some reason the last patch was applied correctly. Just committed the fixed version. Tried to compile axpy.cu, everything works.

-------------
Best regards,
Alexey Bataev

23.01.2019 13:37, treinz пишет:

Hi Alexey,

[treinz via llvm-dev]

Hi Alexey,

Thanks for working on this. I assume the last commit you mentioned is:

commit 48aefa8877713616802d9c90b63f86fe50b9a101 (HEAD -> master, origin/master, origin/HEAD)

Author: Alexey Bataev <a.ba...@hotmail.com>

Date:   Wed Jan 23 18:59:54 2019 +0000

    [DEBUGINFO, NVPTX] Enable support for the debug info on NVPTX target.

    Enable full support for the debug info.

    Differential revision: https://reviews.llvm.org/D46189

    git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@351974 91177308-0d34-0410-b5e6-96231b3b80d8

I'll check this one out. 

Thanks,

Char

[Alexey Bataev via llvm-dev]

Yes, this one.

-------------
Best regards,
Alexey Bataev

23.01.2019 14:17, treinz via llvm-dev пишет:

[Alexey Bataev via llvm-dev]

Hi Char, unfortunately I cannot help you with these crashes, because they caused not by clang /LLVM, but the NVidia's PTX-to-SASS translator ptxas. It is ptxas who crashes and I can't do anything about this. You need to report the bug to NVidia, but you need to manually run the ptxas tool with the generated  .s file. Sometimes it helps to compile the code sequentially, not using the parallel builds. Plus, those crashes may disappear from time to time.

Best regards,

Alexey Bataev

25 февр. 2019 г., в 19:59, treinz <tre...@163.com> написал(а):

Hi Alexey,

Thanks for the great work! The version I checked out works most of the time. But I do encounter crashes sometimes. I can't file a bug report on https://bugs.llvm.org/ because I don't have an account. I sent an email to bugs-...@lists.llvm.org for an account already but I haven't heard back. Meanwhile, can you take a look at the issue? I'm attaching the bug report files here -- the clang_error.log has the verbose output from clang and the other files are generated automatically by clang upon the crash. I'm using the above-mentioned commit 48aefa8877713616802d9c90b63f86fe50b9a101 from github mirror. The bug is reproducible on my machine with a nvidia RTX2080 card (sm=Turing75).

Thanks,

Char

 

<bugreport.tar.bz2>

[treinz via llvm-dev]

Hi Alexey,

Just want to make sure I understand what you said because I'm not familiar with the llvm pipeline, it's this line:

/net/gs/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/bin/ptxas" -m64 -g --dont-merge-basicblocks --return-at-end -v --gpu-name sm_75 --output-file /tmp/60663577.1.login.q/testparticles-4fd988.o /tmp/60663577.1.login.q/testparticles-1d20c4.s

that segfaults, where the testparticles-1d20c4.s is generated by the command:

/home/aznb/.linuxbrew/Cellar/llvm/HEAD-48aefa8/bin/clang-9" -cc1 -triple nvptx64-nvidia-cuda -aux-triple x86_64-unknown-linux-gnu -S -disable-free -disable-llvm-verifier -discard-value-names -main-file-name testparticles.cpp -mrelocation-model static -mthread-model posix -mdisable-fp-elim -no-integrated-as -fuse-init-array -fcuda-is-device -mlink-builtin-bitcode /net/gs/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/nvvm/libdevice/libdevice.10.bc -target-feature +ptx61 -target-cpu sm_75 -dwarf-column-info -debug-info-kind=limited -dwarf-version=2 -debugger-tuning=gdb -v -resource-dir /home/aznb/.linuxbrew/Cellar/llvm/HEAD-48aefa8/lib/clang/9.0.0 -internal-isystem /home/aznb/.linuxbrew/Cellar/llvm/HEAD-48aefa8/lib/clang/9.0.0/include/cuda_wrappers -internal-isystem /net/gs/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/include -include __clang_cuda_runtime_wrapper.h -isystem /home/aznb/.linuxbrew/Cellar/boost/1.68.0_clang/include/boost-1_68 -I /home/aznb/mycodes/scbmc -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/kokkos-src/core/src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/kokkos-src/containers/src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/kokkos-src/algorithms/src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/kokkos-build -I /home/aznb/mycodes/scbmc/../thrust -I /net/gs/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/include -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/kokkosutils-src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/alias-src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/ziggurat-src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/distribution-src -I /home/aznb/mycodes/scbmc/build_cuda_clang_debug/_deps/generic-src -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0 -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/x86_64-unknown-linux-gnu -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/backward -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0 -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/x86_64-unknown-linux-gnu -internal-isystem /home/aznb/.linuxbrew/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../include/c++/5.5.0/backward -internal-isystem /usr/local/include -internal-isystem /home/aznb/.linuxbrew/Cellar/llvm/HEAD-48aefa8/lib/clang/9.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/aznb/.linuxbrew/Cellar/llvm/HEAD-48aefa8/lib/clang/9.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O0 -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /home/aznb/mycodes/scbmc/build_cuda_clang_debug -ferror-limit 19 -fmessage-length 0 -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -o /tmp/60663577.1.login.q/testparticles-1d20c4.s -x cuda /home/aznb/mycodes/scbmc/testparticles.cpp

Both of these commands were shown in the error log. How do I replace my original cpp files with the two *.cu files automatically generated by llvm in the above command to reproduce the bug?

Thanks,

Char

[Alexey Bataev via llvm-dev]

Hi Char, yes, this line causes the crash. You don't need .cu files (I don't know what files are you talking about, clang does not generate .cu files). You need just this .s file to reproduce the error and nothing else.

-------------
Best regards,
Alexey Bataev

26.02.2019 13:06, treinz пишет:

[treinz via llvm-dev]

Hi Alexey,

Thanks for the great work! The version I checked out works most of the time. But I do encounter crashes sometimes. I can't file a bug report on https://bugs.llvm.org/ because I don't have an account. I sent an email to bugs-...@lists.llvm.org for an account already but I haven't heard back. Meanwhile, can you take a look at the issue? I'm attaching the bug report files here -- the clang_error.log has the verbose output from clang and the other files are generated automatically by clang upon the crash. I'm using the above-mentioned commit 48aefa8877713616802d9c90b63f86fe50b9a101 from github mirror. The bug is reproducible on my machine with a nvidia RTX2080 card (sm=Turing75).

Thanks,

Char

At 2019-01-24 03:25:39, "Alexey Bataev" <a.ba...@outlook.com> wrote:

[treinz via llvm-dev]

Hi Alexey,

I submitted the bug report to nvidia. While they are working on it, can you share some insight in what could potentially cause this? I just want to get a sense if such a bug require significant amount of work to fix, which can help me make some decision moving forward with my project.

Thanks,

Char

[Alexey Bataev via llvm-dev]

Hi Char, it looks like they have some kind of buffer overflow bug in their ptxas tool.

-------------
Best regards,
Alexey Bataev

27.02.2019 13:08, treinz пишет:

[treinz via llvm-dev]

Hi Alexey,

Is there any option for clang to turn on debug for the host code only but not the device code? I've been using something like -ggdb3 -O0 but this generate debug info for both host and device. I'm trying to work around the aforementioned ptxas bug.

Thanks,

Char

[Alexey Bataev via llvm-dev]

-Xopenmp-target -g0

-------------
Best regards,
Alexey Bataev

11.03.2019 14:35, treinz пишет:

[treinz via llvm-dev]

I tried that with the axpy example:

clang++ axpy.cu -Xopenmp-target -g0 -o axpy --cuda-gpu-arch=sm_75 \

-Wl,--dynamic-linker,$LINUXBREWHOME/lib/ld.so \

-L$LINUXBREWHOME/lib \

-L/net/gs/vol3/software/modules-sw/cuda/10.0/Linux/RHEL6/x86_64/lib64 \

-Wl,-rpath,$LINUXBREWHOME/lib \

-lcudart_static -ldl -lrt -pthread

but clang complains: clang-9: warning: argument unused during compilation: '-Xopenmp-target -g0' [-Wunused-command-line-argument]

[Alexey Bataev via llvm-dev]

This option is for OpenMP only. For cuda use something like this: --cuda-gpu-arch=sm_35 -Xarch_sm_35 -g0

-------------
Best regards,
Alexey Bataev

11.03.2019 14:55, treinz пишет:

[treinz via llvm-dev]

Hi Alexey,

Almost a year has passed and Nvidia finally fixes the ptxas issue in CUDA 10.2 according to: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-compiler-resolved-issues However, I can not yet use it with llvm 9.0.0 release because CUDA 10.2 is not supported yet. Is there other branches of the llvm repo that supports CUDA 10.2 now? Or do I need to wait for llvm 10 releaase.

Thanks,

Char

[Alexey Bataev via llvm-dev]

Hi Char, not sure if clang supports 10.2. You can ry to use trunk instead of release and check it yourself. Most probably clang 10 will support cuda 10. Also, you can use cuda 9 safely, just replace ptxas in the installation dir with the newest ptxas from cuda 10.2

-------------
Best regards,
Alexey Bataev

15.01.2020 9:17 AM, treinz пишет:

[Artem Belevich via llvm-dev]

On Wed, Jan 15, 2020 at 11:20 AM Alexey Bataev via llvm-dev <llvm...@lists.llvm.org> wrote:

Hi Char, not sure if clang supports 10.2. You can ry to use trunk instead of release and check it yourself. Most probably clang 10 will support cuda 10. Also, you can use cuda 9 safely, just replace ptxas in the installation dir with the newest ptxas from cuda 10.2

Clang does not have support for CUDA 10.2 yet. I'll try to get the patch for basic support out in upcoming weeks, 

unless someone beats me to it.

If all you need is to work around a bug in ptxas, replacing it is a reasonably 

safe approach (modulo things that may have been changed/deprecated in the new version).

--Artem

 

https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

--

--Artem Belevich