CUDA Disabled after building?

[Honson Tran]

Hello Everyone!

I wanted to reach out and see if there's anyone that has ran into this similar problem. I managed to build PAPI  with cuda components, made sure I linked CUPTI during the build process, and can confirm it has been build by running `./papi_component_avail`

```

Available components and hardware information.
--------------------------------------------------------------------------------
PAPI version             : 6.0.0.0
Operating system         : Linux 5.11.0-41-generic
Vendor string and code   : GenuineIntel (1, 0x1)
Model string and code    : Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (158, 0x9e)
CPU revision             : 10.000000
CPUID                    : Family/Model/Stepping 6/158/10, 0x06/0x9e/0x0a
CPU Max MHz              : 4500
CPU Min MHz              : 800
Total cores              : 12
SMT threads per core     : 2
Cores per socket         : 6
Sockets                  : 1
Cores per NUMA region    : 12
NUMA regions             : 1
Running in a VM          : no
Number Hardware Counters : 10
Max Multiplex Counters   : 384
Fast counter read (rdpmc): yes
--------------------------------------------------------------------------------

Compiled-in components:
Name:   perf_event              Linux perf_event CPU counters
Name:   perf_event_uncore       Linux perf_event CPU uncore and northbridge
Name:   cuda                    CUDA events and metrics via NVIDIA CuPTI interfaces
   \-> Disabled:

Active components:
Name:   perf_event              Linux perf_event CPU counters
                                Native: 171, Preset: 59, Counters: 10
                                PMUs supported: ix86arch, perf, perf_raw, skl

Name:   perf_event_uncore       Linux perf_event CPU uncore and northbridge
                                Native: 4, Preset: 0, Counters: 3
                                PMUs supported: rapl


--------------------------------------------------------------------------------
```

I found a similar post here:

https://groups.google.com/a/icl.utk.edu/g/ptools-perfapi/c/Xh1I9zn6HBs

However, this post might be a bit outdated, as it mentions there are certain sm versions not covered from last year. I'm currently attempting this on a RTX 2060 (Laptop version).

One more important thing to note is that I'm running this inside a docker container. However, I have ran this container with the `--privileged` flag, and I can confirm that I have gotten PAPI to work with the CPU hardware counters, just not the GPU.

Is there any update or fix to enable the GPU counters? Any help would be appreciated!

Best,

Honson

[Giuseppe Congiu]

Hi Honson,

What version of CUDA are you using to configure PAPI? Your GPU should support compute capability 7.5. IIRC, this requires CUpti11 (which PAPI supports). If you are using an older CUDA version installation (check your PAPI_CUDA_ROOT environment variable) this might be causing the issue.

Please also have a look at https://bitbucket.org/icl/papi/src/master/src/components/cuda/README.md in case you have not already done so.

Thank you,

Giuseppe 

--
You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/21851d61-ddd4-4bb8-a5b8-23369673260en%40icl.utk.edu.

[Honson Tran]

Hello Giuseppe,

Thanks for the quick reply. I have painfully went through the installation again to double check everything.

>  What version of CUDA are you using to configure PAPI?

The Docker container I had was NVIDIA's TRT container v20.03 (this packaged CUDA 10.2 in the container). However, I have also exited the container, and tried a setup where I purged all Nvidia drivers on the local system and then installed CUDA 11.5. I made sure that I explicitly defined `PAPI_CUDA_ROOT` as well to point to `/usr/local/cuda`. I can confirm that for the docker environment as well as the local environment, I have tried the PAPI installation both on CUDA 10.2 and 11.5 with no luck.

> Please also have a look at https://bitbucket.org/icl/papi/src/master/src/components/cuda/README.md in case you have not already done so.

I took another look at this, and can also confirm that I can see all the necessary built libraries for PAPI during runtime (I even did this by making sure each directory containing the .so files were in `LD_LIBRARY_PATH`). One thing to note is that `PAPI_CUDA_ROOT/extras/CUPTI/include` is actually now installed in `/usr/include` and  `PAPI_CUDA_ROOT/extras/CUPTI/lib64` is actually in `/usr/local/cuda/lib64`. Still, I can see PAPI compiling the CUPTI component, but when you run `./papi_component_avail`, you're stuck with the following:

Compiled-in components:
Name:   perf_event              Linux perf_event CPU counters
Name:   perf_event_uncore       Linux perf_event CPU uncore and northbridge

   \-> Disabled: Insufficient permissions for uncore access.  Set /proc/sys/kernel/perf_event_paranoid to 0 or run as root.

Name:   cuda                    CUDA events and metrics via NVIDIA CuPTI interfaces
   \-> Disabled:

Active components:
Name:   perf_event              Linux perf_event CPU counters
                                Native: 171, Preset: 59, Counters: 10
                                PMUs supported: ix86arch, perf, perf_raw, skl

 Any helping would be much appreciated :)

[Giuseppe Congiu]

Can you also try the latest papi version (6.0.0.1)?

To view this discussion on the web visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/f8868593-2543-4718-90c1-0619224480fan%40icl.utk.edu.

[Honson Tran]

Hello,

Just pulled the latest 6.0.0.1 for papi, and the conclusion is still the same. Any other leads or troubleshooting I can do? Thanks for your time :)

-Honson

[Giuseppe Congiu]

Hi Honson,

It is hard to figure out what is going wrong without a “disabled_reason” string in the “papi_component_avail" output.

Could you please run “papi_component_avail” again, this time with debug enabled, and send me the output?

In order to do that reconfigure papi, adding the `--with-debug=yes` option to your configuration parameters. Then, "export PAPI_DEBUG=ALL" and run “papi_component_avail” again.

Note that when debug is enabled papi will output a lot of text so you might want to redirect that to a file.

Thanks,

Giuseppe

[Honson Tran]

Happy Holidays!

Attached is the full print out. Thanks again for the help!

-Honson

[Giuseppe Congiu]

Hi Honson,

I hope you had nice holidays.

From the log:

SUBSTRATE:components/cuda/linux-cuda.c:_cuda_init_component:834:172693 Entering with component idx: 2
SUBSTRATE:components/cuda/linux-cuda.c:_cuda_add_native_events:439:172693 Entering
SUBSTRATE:components/cuda/linux-cuda.c:_cuda_add_native_events:479:172693 Found 1 devices
SUBSTRATE:components/cuda/linux-cuda.c:_cuda_add_native_events:499:172693 error: function (*cuptiDeviceGetNumEventDomainsPtr) (mydevice->cuDev, &mydevice->maxDomains) failed with error CUPTI_ERROR_LEGACY_PROFILER_NOT_SUPPORTED.

Looks like you are using a legacy version of the cupti profiler. Please double check all your environment variables (including LD_LIBRARY_PATH) and make sure they are consistent.

I also found the following comment in the component's source code:

1548 // The routine cuptiDeviceGetNumEventDomains() is illegal for devices with compute
1549 // capability >= 7.5. From the online manual
1550 // (https://docs.nvidia.com/cupti/Cupti/modules.html): Legacy CUPTI Profiling is not
1551 // supported on devices with Compute Capability 7.5 or higher (Turing+). From
1552 // https://developer.nvidia.com/cuda-gpus#compute): We find the Quadro GTX 5000 (our first
1553 // failure) has a Compute Capability of 7.5.

which is consistent with the reported error in the log.

Best,
Giuseppe

> On 26 Dec 2021, at 05:20, Honson Tran <honso...@gmail.com> wrote:
>
> <papi-component-debug-log.txt>

[Honson Tran]

Hello Giuseppe,

Hope you had a good holiday break as well!

The GPU I'm using is a laptop RTX 2060, which would mean I am using CC 7.5. CUPTI should come installed with the nvidia toolkit that I have installed as well.

Given that the card is using CC 7.5, and the component's src code already mentions incompatibility with non-legacy devices, does PAPI have a fix for newer generation cards?

Best,

Honson

[Giuseppe Congiu]

Hi Honson,

The GPU I'm using is a laptop RTX 2060, which would mean I am using CC 7.5. CUPTI should come installed with the nvidia toolkit that I have installed as well.

Given that the card is using CC 7.5, and the component's src code already mentions incompatibility with non-legacy devices, does PAPI have a fix for newer generation cards?

PAPI supports both old and new cupti. From the log you sent back the problem seems to be with your environment.

The PAPI source code (from my previous email) simply says you ended up in a call you were not supposed to make because PAPI is using legacy cupti but your device does not support it.

I am not sure what is wrong with your environment but, unless you can find an example we can use to consistently reproduce the problem and show it is a PAPI related issue, we have no way to debug the problem.

Sorry I cannot be more helpful.

Best,

Giuseppe

[Honson Tran]

Hello Giuseppe,

I verified on my end on a Jetson AGX (sm_72), and CUPTI seems to be activated, but the CPU counters are not. Is there ARM support for PAPI for CPU counters?

Also, with regards to the situation with the laptop, I have confirmed via `nvcc -v` that I am using CUDA 11.5 on the RTX 2060 (sm_75). I have also made sure the CUPTI library on here is also 11.5. Prior to this installation, all previous Nvidia drivers and software relating to Nvidia was uninstalled. These were my steps to build and install PAPI:

1. wget http://icl.utk.edu/projects/papi/downloads/papi-6.0.0.tar.gz
2. tar -xvzf papi-6.0.0.tar.gz
3. cd papi-6.0.0/src
4. ./configure --prefix=$PWD/install --with-components=cuda
   
5. # It seems like PAPI keeps checking inside `/opt/cuda` for `cupti.h`, so I made a symlink to redirect it to where this header file resides in. This folder contains CUDA 11.5 ln -s /usr/local/cuda/ /opt/cuda
6. make && make install

I then went into `src/install/bin` and launched `./papi_component_avail`, which is where I see that cuda is disabled. Is there a way to replicate this on your end?

Best,

Honson

[Honson Tran]

Also, as a side note, please feel free to share your environment and installation details as well. Would like to compare and see if there is anything I missed.

-Honson