Julia + Polly ACC not targeting GPUs | All library dependencies satisfied

[llvmres...@iith.ac.in]

Hello,

I've manually set Target to TARGET_GPU, CudaVersion to sm_21 through cl::init() and linked PollyPPCG to libjulia to target a NVIDIA 820M GPU by default. But, Julia failed to run the code, especially kernel_gemm, on the GPU.

Observations,

• nvidia-smi didn't show any GPU processes
• nvprof ./julia didn't profile any kernels
• PPCGCodeGeneration's constructor was called only twice, right after Julia was started and before the REPL prompt.
• JULIA_LLVM_ARGS="-polly-acc-dump-code -polly-acc-dump-schedule" ./julia produced semantically equivalent C code for host and device and the schedule respectively.
  
• After making the following changes to GPUNodeBuilder::finalizeKernelFunction,
  

 std::string GPUNodeBuilder::finalizeKernelFunction() {
   if (verifyModule(*GPUModule,&(llvm::errs()))) {
         llvm::errs() << __func__ << ":BuildFailed\n";

   BuildSuccessful = false;

   return "";

• The function exited returning an empty string with the following output,
• DICompileUnit not listed in llvm.dbg.cu
  !11 = distinct !DICompileUnit(language: DW_LANG_C89, file: !3, producer: "julia", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !12)
  finalizeKernelFunction:BuildFailed
  
  

How can I resolve this ? Also, please point me to resource that'd explain this error.

Thank You,

Sanjay Srivallabh

[Tobias Grosser]

On Fri, Mar 10, 2017, at 12:37 PM, llvmres...@iith.ac.in wrote:
> Hello,
>
> I've manually set Target to TARGET_GPU, CudaVersion to sm_21 through
> cl::init() and linked PollyPPCG to libjulia to target a NVIDIA 820M GPU
> by
> default. But, Julia failed to run the code, especially kernel_gemm, on
> the
> GPU.
>
> Observations,
>

> - nvidia-smi didn't show any GPU processes
> - nvprof ./julia didn't profile any kernels
> - PPCGCodeGeneration's constructor was called only twice, right after

> Julia was started and before the REPL prompt.

> - JULIA_LLVM_ARGS="-polly-acc-dump-code -polly-acc-dump-schedule"

> ./julia produced semantically equivalent C code for host and device
> and the
> schedule respectively.

> - After making the following changes to

> GPUNodeBuilder::finalizeKernelFunction,
>
> std::string GPUNodeBuilder::finalizeKernelFunction() {

> if (verifyModule(*GPUModule*,&(llvm::errs**())*)) {
> *llvm**::errs() << __func__ << ":BuildFailed\n";*
> BuildSuccessful = false;
> return "";
>
> - The function exited returning an empty string with the following
> output,
> -

> !11 = distinct !DICompileUnit(language: DW_LANG_C89, file: !3,
> producer: "julia", isOptimized: true, runtimeVersion: 0,
> emissionKind:
> FullDebug, enums: !12)
> finalizeKernelFunction:BuildFailed
>
>
> How can I resolve this ? Also, please point me to resource that'd explain
> this error.

Run:

grep -R "DICompileUnit not listed in llvm.dbg.cu" <llvm_src>/

this should give you the code where the error message is emitted. It
seems that for some reason the debug-info metadata is invalid.

Best,
Tobias

[llvmres...@iith.ac.in]

Hello,

According to this commit and <llvm_src>/test/Verifier/dbg-orphaned-compileunit.ll, the "!11 = distinct !DICompileUnit(language: DW ... " DICompileUnit wasn't included in llvm.dbg.cu .

What does it mean by that ? I've asked this question on llvm-dev too.

Thank You,

Sanjay

[Tobias Grosser]

On Sat, Mar 11, 2017, at 12:15 PM, llvmres...@iith.ac.in wrote:
> Hello,
>

> According to this commit <https://reviews.llvm.org/D18518?id=51808> and

> <llvm_src>/test/Verifier/dbg-orphaned-compileunit.ll, the "!11 = distinct
> !DICompileUnit(language: DW ... " DICompileUnit wasn't included in
> llvm.dbg.cu .
>
> What does it mean by that ? I've asked this question on llvm-dev

> <http://lists.llvm.org/pipermail/llvm-dev/2017-March/110978.html> too.

I reply on llvm-dev.

Tobias

>
> Thank You,
> Sanjay
>
> On Friday, March 10, 2017 at 5:13:49 PM UTC+5:30, Tobias wrote:
> >
> >
> >

> > On Fri, Mar 10, 2017, at 12:37 PM, llvmres...@iith.ac.in <javascript:>

[SANJAY SRIVALLABH SINGAPURAM]

Hello,

Continuing from this post on llvm-dev,

Which part of the code is inserting debug info as call parameters ?

[Tobias Grosser]

Start from createLaunchParameters and check how SubTreeValues is filled.

Tobias

On Mon, Mar 13, 2017, at 05:45 PM, SANJAY SRIVALLABH SINGAPURAM wrote:
> Hello,
>
> Continuing from this post on llvm-dev

> <http://lists.llvm.org/pipermail/llvm-dev/2017-March/111007.html>,

[SANJAY SRIVALLABH SINGAPURAM]

Hello Tobias,

The SubtreeValues is a SetVector containing pointers to Value objects. According to <llvm_src>/include/llvm/IR/Value.h, a Value has a Type field and according to <llvm_src>/include/llvm/IR/Type.h a Value can be of one of many Types defined by an enum.

I made these changes to PPCGCodeGeneration to understand the Types we could isolate,

1165+ int iter=0;

1166   for (auto Val : SubtreeValues) {
1167+           llvm::errs() << iter++ << (Val->getType())->isPointerTy() << '\n' << (*Val) << '\n';
1168+           llvm::errs().flush();
1169      Instruction *Param = new AllocaInst(
1170         Val->getType(), Launch + "_param_" + std::to_string(Index),
1171         EntryBlock->getTerminator());

and got the following output,

11

; Function Attrs: nounwind readnone
declare void @llvm.dbg.value(metadata, i64, metadata, metadata) #2

20
i64 %1
30
i64 %0

It's clear that Value is a pointer, to a function @llvm.dbg.value. But, we can't exclude pointers, since they can be used to pass other information to kernel, like arrays. Although, it seems that for (long i = 0; i < Prog->n_array; i++) at line 1091 covers all the pointers to arrays. So, is it safe to exclude pointers from subtree values ?

Thank You,

Sanjay

PS :
1. How did a function call associated with debug information get into the subtree ?

2. How did you get Value::dump() working ? ( you seemed to have used it to generate the output  in your last llvm-dev post )

[Tobias Grosser]

On Mon, Mar 13, 2017, at 08:36 PM, SANJAY SRIVALLABH SINGAPURAM wrote:
> Hello Tobias,
>
> The SubtreeValues is a SetVector containing pointers to Value objects.
> According to <llvm_src>/include/llvm/IR/Value.h, a Value has a Type field
> and according to <llvm_src>/include/llvm/IR/Type.h a Value can be of one
> of
> many Types defined by an enum.
>
> I made these changes to PPCGCodeGeneration to understand the Types we
> could
> isolate,
>

> *1165+ int iter=0;*

> 1166 for (auto Val : SubtreeValues) {
>

> *1167+ llvm::errs() << iter++ <<
> (Val->getType())->isPointerTy()
> << '\n' << (*Val) << '\n';1168+ llvm::errs().flush();*

> 1169 Instruction *Param = new AllocaInst(
> 1170 Val->getType(), Launch + "_param_" + std::to_string(Index),
> 1171 EntryBlock->getTerminator());
>
> and got the following output,
>

> 1*1*

>
> ; Function Attrs: nounwind readnone
> declare void @llvm.dbg.value(metadata, i64, metadata, metadata) #2
>
> 20
> i64 %1
> 30
> i64 %0
>
> It's clear that Value is a pointer, to a function @llvm.dbg.value. But,
> we
> can't exclude pointers, since they can be used to pass other information
> to
> kernel, like arrays.

> Although, it seems that for (long i = 0; i <
> Prog->n_array; i++) at line 1091 covers all the pointers to arrays. So,
> is
> it safe to exclude pointers from subtree values ?

Have a look at polly::isIgnoredIntrinsic(). You likely want to use it in
addReferencesFromStmt to ignore debug value intrinsics.

> Thank You,
> Sanjay
>
> PS :
> 1. How did a function call associated with debug information get into the
> subtree ?

It was added. ;) I suggest to look through the code and see where the
array was filled. Was it difficult to find that location?

> 2. How did you get Value::dump() working ? ( you seemed to have used it
> to
> generate the output in your last llvm-dev post )

Val->dump().

In case it does not work, please describe what you see instead.

Best,

[SANJAY SRIVALLABH SINGAPURAM]

Hello Tobias,

It looks like a function to identify elements that could be trimmed out without losing the semantics and functionality of the program.

You likely want to use it in
addReferencesFromStmt to ignore debug value intrinsics.

Could you please elaborate ? Does it have anything to do with the following ?

lib/CodeGen/IslNodeBuilder.cpp
219 isl_stat addReferencesFromStmt(const ScopStmt *Stmt, void *UserPtr,
220                                bool CreateScalarRefs) {
[...]
243     if (CreateScalarRefs)
244       References.Values.insert(References.BlockGen.getOrCreateAlloca(*Access));

If so, it may not matter since,
lib/CodeGen/PPCGCodeGeneration.cpp:972:  addReferencesFromStmt(Stmt, User, false /* CreateScalarRefs */);

> Thank You,
> Sanjay
>
> PS :
> 1. How did a function call associated with debug information get into the
> subtree ?

It was added. ;) I suggest to look through the code and see where the
array was filled. Was it difficult to find that location?

I think it's filled here,

PPCGCodeGeneration.cpp

977 SetVector<Value *> GPUNodeBuilder::getReferencesInKernel(ppcg_kernel *Kernel) {
[...]
984   for (const auto &I : IDToValue)
985     SubtreeValues.insert(I.second);
 

We could catch these values using isIgnoredIntrinsic before they're inserted.

> 2. How did you get Value::dump() working ? ( you seemed to have used it
> to
> generate the output  in your last llvm-dev post )

Val->dump().

I did use this. The build failed stating "undefined reference to 'llvm::Value::dump() const".

In case it does not work, please describe what you see instead.

I've been using Val->print(llvm::errs()) indirectly through llvm::errs() << (*Val).

Thank You,

Sanjay

[Tobias Grosser]

Exactly.

> > You likely want to use it in
> > addReferencesFromStmt to ignore debug value intrinsics.
> >
> Could you please elaborate ? Does it have anything to do with the
> following
> ?
>
> lib/CodeGen/IslNodeBuilder.cpp
> 219 isl_stat addReferencesFromStmt(const ScopStmt *Stmt, void *UserPtr,
> 220 bool CreateScalarRefs) {
> [...]

> 243 *if (CreateScalarRefs)*
> 244
> *References.Values.insert(References.BlockGen.getOrCreateAlloca(*Access));*

>
> If so, it may not matter since,
> lib/CodeGen/PPCGCodeGeneration.cpp:972: addReferencesFromStmt(Stmt,
> User,

> *false* /* CreateScalarRefs */);

Maybe then findReferencesInBlock.

> > > Thank You,
> > > Sanjay
> > >
> > > PS :
> > > 1. How did a function call associated with debug information get into the
> > > subtree ?
> >
> > It was added. ;) I suggest to look through the code and see where the
> > array was filled. Was it difficult to find that location?
> >
> I think it's filled here,
> PPCGCodeGeneration.cpp
> 977 SetVector<Value *> GPUNodeBuilder::getReferencesInKernel(ppcg_kernel
> *Kernel) {
> [...]
> 984 for (const auto &I : IDToValue)
> 985 SubtreeValues.insert(I.second);

Yes, which via collectReferencesInGPUStmt calls addReferencesFromStmt.

> We could catch these values using isIgnoredIntrinsic before they're
> inserted.

I believe we should filter them out as early as possible.

> > > 2. How did you get Value::dump() working ? ( you seemed to have used it
> > > to
> > > generate the output in your last llvm-dev post )
> >
> > Val->dump().
> >
> I did use this. The build failed stating "undefined reference to
> 'llvm::Value::dump() const".
>
> >
> > In case it does not work, please describe what you see instead.
> >
> I've been using Val->print(llvm::errs()) indirectly through llvm::errs()
> <<
> (*Val).

Is this in a Julia or in a pure LLVM build? Maybe this applies:
http://stackoverflow.com/questions/22423120/using-llvmfunctiondump-linker-gives-undefined-reference-to-llvmvalue

> --
> You received this message because you are subscribed to the Google Groups
> "Polly Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to polly-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[SANJAY SRIVALLABH SINGAPURAM]

Hello Tobias,

I've done the following to IslNodeBuilder::getNumberOfIterations,
+   llvm::errs() << "Entre " << __func__ << '\n';       
     for (const Instruction &Inst : *BB)
+      {
+              if( polly::isIgnoredIntrinsic(&Inst) )
+              {
+                      llvm::errs() << (Inst) << '\n';
+                      continue;
+              }
[...]
+   llvm::errs() << "Exit " << __func__ << '\n';

It detects the calls to @llvm.debug.value but the calls still end up in the output file and opt reports an error,
$ ./llvm_build/bin/opt kernel_gemm-before.ll  -O3 -polly -S -o output.ll
PPCGCodeGeneration
Entre findReferencesInBlock
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#.021", i64 0, metadata !21, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#.021", i64 0, metadata !21, metadata !28), !dbg !27
Exit findReferencesInBlock
Entre findReferencesInBlock
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#2.022", i64 0, metadata !21, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#3.023", i64 0, metadata !22, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#3.023", i64 0, metadata !22, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#2.022", i64 0, metadata !21, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#2.022", i64 0, metadata !21, metadata !28), !dbg !27
Exit findReferencesInBlock
Entre findReferencesInBlock
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#.021", i64 0, metadata !21, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#4.024", i64 0, metadata !23, metadata !28), !dbg !27
  tail call void @llvm.dbg.value(metadata i64 %"#temp#.021", i64 0, metadata !21, metadata !28), !dbg !27
Exit findReferencesInBlock

DICompileUnit not listed in llvm.dbg.cu

!11 = distinct !DICompileUnit(language: DW_LANG_C89, file: !3, producer: "julia", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !12)
finalizeKernelFunction:BuildFailed

10
i64 %1
20
i64 %0

These function calls were already absent from the output file that was generated before the change. Just to be sure, I added a similar condition in  GPUNodeBuilder::getReferencesInKernel(ppcg_kernel *Kernel),

for (const auto &I : IDToValue)

  {
          if( polly::isIgnoredIntrinsic((I.second)) )
          {
                  llvm::errs() << I.second;
          }
          SubtreeValues.insert(I.second);
  }

But didn't produce any output.

Do you think Values are being added to SubtreeValues somewhere else ?

[Tobias Grosser]

Not sure. Can you check. In general, you can just look at how / where
the code is generated and see how and when debug values are processed.

Best,
Tobias

> Email had 1 attachment:
> + output.ll
> 27k (application/octet-stream)

[SANJAY SRIVALLABH SINGAPURAM]

Hello Tobias,

I shifted to using the Julia no-debug build. findReferencesInBlock couldn't find any debug information but generated a new error.

$ opt kernel_gemm-before.ll  -O3 -polly -S -o output.ll

DICompileUnit not listed in llvm.dbg.cu

!6 = distinct !DICompileUnit(language: DW_LANG_C89, file: !3, producer: "julia", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !5)

finalizeKernelFunction:BuildFailed
10
i64 %1
20
i64 %0

The following changes removed all debug info and generated the kernel.

<polly_src>/lib/CodeGen/PPCGCodeGeneration.cpp

@@ -1590,7 +1599,10 @@ std::string GPUNodeBuilder::createKernelASM() {
+#include "llvm/IR/DebugInfo.h"  // Can I do without this ?
 std::string GPUNodeBuilder::finalizeKernelFunction() {
+ 
+       llvm::StripDebugInfo((*GPUModule));

<llvm_src>/lib/Support/TargetRegistry.cpp
@@ -63,6 +63,7 @@ const Target *TargetRegistry::lookupTarget(const std::string &ArchName,
 const Target *TargetRegistry::lookupTarget(const std::string &TT,
                                            std::string &Error) {
   // Provide special warning when no targets are initialized.
+       llvm::errs() << "Target Triple:" << TT << '\n';

julia> @time kernel_gemm( 1, 0, s_c, s_a, s_b );
Target Triple:nvptx64-nvidia-cuda
No available targets are compatible with this triple. ## Does this mean it didn't generate the kernel ?

10
i64 %1
20
i64 %0

LLVM ERROR: Program used external function 'polly_getKernel' which could not be resolved!

Adding the rpath to the linker options didn't alleviate the problem,

<julia_src>/src/Makefile

@@ -57,7 +57,7 @@
-LLVMLINK += -lPolly -lPollyISL
+LLVMLINK += -lPolly -lPollyISL -lPollyPPCG -Wl,-rpath=/home/sanjay/Software/polly_julia/llvm_build/lib -lGPURuntime

It was strange to see that the kernel string was actually missing in kernel_gemm-after.ll and was present output.ll generated as follows,

$ ./llvm_build/bin/opt kernel_gemm-before.ll  -O3 -polly -S -o output.ll

Target Triple:x86_64-unknown-linux-gnu
Target Triple:nvptx64-nvidia-cuda    ## No issues with this target triple. !!

10
i64 %1
20
i64 %0

I've attached kernel_gemm-before.ll, output.ll and kernel_gemm-after.ll.

Please share your thoughts.

Thank You,

Sanjay

[Tobias Grosser]

On Thu, Mar 16, 2017, at 08:58 PM, SANJAY SRIVALLABH SINGAPURAM wrote:
> Hello Tobias,
>

> I shifted to using the Julia no-debug build.

OK. If it is easy, would be great if you could upstream a fix for the
debug version as well. (The one that ignores the intrinsics).

> findReferencesInBlock
> couldn't
> find any debug information but generated a new error.
>
> $ opt kernel_gemm-before.ll -O3 -polly -S -o output.ll
> DICompileUnit not listed in llvm.dbg.cu
> !6 = distinct !DICompileUnit(language: DW_LANG_C89, file: !3, producer:
> "julia", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug,
> enums: !5)
> finalizeKernelFunction:BuildFailed
> 10
> i64 %1
> 20
> i64 %0
>
> The following changes removed all debug info and generated the kernel.
>
> <polly_src>/lib/CodeGen/PPCGCodeGeneration.cpp
> @@ -1590,7 +1599,10 @@ std::string GPUNodeBuilder::createKernelASM() {

> *+#include "llvm/IR/DebugInfo.h" // Can I do without this ?*

Looks good, but maybe better at the top of the file?

> std::string GPUNodeBuilder::finalizeKernelFunction() {
> +

> *+ llvm::StripDebugInfo((*GPUModule));*

>
> <llvm_src>/lib/Support/TargetRegistry.cpp
> @@ -63,6 +63,7 @@ const Target *TargetRegistry::lookupTarget(const
> std::string &ArchName,
> const Target *TargetRegistry::lookupTarget(const std::string &TT,
> std::string &Error) {
> // Provide special warning when no targets are initialized.
> + llvm::errs() << "Target Triple:" << TT << '\n';
>
> julia> @time kernel_gemm( 1, 0, s_c, s_a, s_b );
> Target Triple:nvptx64-nvidia-cuda

> No available targets are compatible with this triple. *## Does this mean
> it
> didn't generate the kernel ?*

No. Most likely you do not have the NVPTX backend compiled into Julia.

> 10
> i64 %1
> 20
> i64 %0
> LLVM ERROR: Program used external function 'polly_getKernel' which could
> not be resolved!

OK. Seems that GPURuntime is not properly linked. Can you try to just
LD_PRELOAD it and see if that at least works?

> Adding the rpath to the linker options didn't alleviate the problem,
> <julia_src>/src/Makefile
> @@ -57,7 +57,7 @@
> -LLVMLINK += -lPolly -lPollyISL
> +LLVMLINK += -lPolly -lPollyISL -lPollyPPCG
> -Wl,-rpath=/home/sanjay/Software/polly_julia/llvm_build/lib -lGPURuntime
>
> It was strange to see that the kernel string was actually missing in
> kernel_gemm-after.ll and was present output.ll generated as follows,
> $ ./llvm_build/bin/opt kernel_gemm-before.ll -O3 -polly -S -o output.ll
> Target Triple:x86_64-unknown-linux-gnu

> Target Triple:nvptx64-nvidia-cuda *## No issues with this target
> triple.
> !!*

> 10
> i64 %1
> 20
> i64 %0
>
> I've attached kernel_gemm-before.ll, output.ll and kernel_gemm-after.ll.

Best,
Tobias

> Email had 3 attachments:
> + kernel_gemm-before.ll
> 10k (application/octet-stream)
> + kernel_gemm-after.ll
> 36k (application/octet-stream)
> + output.ll
> 36k (application/octet-stream)