Debugging Error Code -1 from PAPI_add_event

[Wyatt Spear]

We're passing ,(PAPI_NATIVE_) coretemp:::hwmon2:temp3_input to TAU and running into PAPI Error: Error Code -1,Invalid argument from PAPI_add_event, somewhere inside _papi_hwi_find_native_event. We do pretty through error code checking and there are no issues with getting that metric name translated with PAPI_event_name_to_code and the resulting code returns ok from PAPI_query event. Other metrics are working as expected with the same test case. Any thoughts on how we can track down what's going on here? I was looking into activating APIDBG and INTDBG in PAPI but couldn't see how to do it. Configuring with --with-debug=yes didn't seem to activate those.

Thanks,

Wyatt Spear

[Vince Weaver]

On Wed, 23 Apr 2025, Wyatt Spear wrote:

> with the same test case. Any thoughts on how we can track down what's going
> on here? I was looking into activating APIDBG and INTDBG in PAPI but
> couldn't see how to do it. Configuring with --with-debug=yes didn't seem to
> activate those.

Once you compile with debugging, you enable the debugging messages with
environment variables. So something like

env PAPI_DEBUG=PAPI_API ./papi_native_avail
or
env PAPI_DEBUG=PAPI_ALL ./papi_native_avail


Vince Weaver
vincent...@maine.edu

[Wyatt Spear]

Thanks, that helps.

I've found that the error only occurs when we enable multiplexing. (in TAU that means we run PAPI_multiplex_init and then PAPI_assign_eventset_component and PAPI_set_multiplex on the EventSet)

The verbose PAPI output (plus some debug statements I added to PAPI_add_event) for the failure looks like this:

INTERNAL:papi_internal.c:_papi_hwi_query_native_event:2459:2264632:0 EXIT: ret: 0
API:papi.c:PAPI_query_event:1256:2264632:0 EXIT: return: 0
About to do PAPI_add_event..
API:papi.c:PAPI_add_event:2207:2264632:0 Entry: EventSet: 3, EventCode: 0x40000020
PAPI_add_event lookup
PAPI_add_event arg check
PAPI_add_event stop check
PAPI_add_event magic
INTERNAL:papi_internal.c:_papi_hwi_add_event:1339:2264632:0 ENTER: ESI: 0x1d204e0 (3), EventCode: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:523:2264632:0 ENTER: event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:554:2264632:0 EXIT: Found cidx: 3 event_index: 32, event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_add_event:1364:2264632:0 EXIT: Event is not valid for component index 3
API:papi.c:PAPI_add_event:2229:2264632:0 EXIT: return: -1
SUBSTRATE:components/appio/appio.c:fwrite:393:2264632:0 appio: intercepted fwrite(0x7f6c05ea0e21,1,12,0x7f6c0436a600)

PAPI Error: Error Code -1,Invalid argument

! rc: -1 comp: 3 lEa: 3 counter: 1073741856 -- -1
TAU: Error adding PAPI events: Invalid argument

The equivalent output without engaging multiplexing looks like this:

INTERNAL:papi_internal.c:_papi_hwi_query_native_event:2459:2266150:0 EXIT: ret: 0
API:papi.c:PAPI_query_event:1256:2266150:0 EXIT: return: 0
About to do PAPI_add_event..
API:papi.c:PAPI_add_event:2207:2266150:0 Entry: EventSet: 3, EventCode: 0x40000020
PAPI_add_event lookup
PAPI_add_event arg check
PAPI_add_event stop check
PAPI_add_event magic
INTERNAL:papi_internal.c:_papi_hwi_add_event:1339:2266150:0 ENTER: ESI: 0x1ce14e0 (3), EventCode: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:523:2266150:0 ENTER: event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:554:2266150:0 EXIT: Found cidx: 3 event_index: 32, event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_assign_eventset:764:2266150:0 ENTER: ESI: 0x1ce14e0 (3), cidx: 3
INTERNAL:papi_internal.c:_papi_hwi_add_event:1375:2266150:0 Adding event to slot 0 of EventSet 3
INTERNAL:papi_internal.c:_papi_hwi_query_native_event:2438:2266150:0 ENTER: EventCode: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:523:2266150:0 ENTER: event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_component_index:554:2266150:0 EXIT: Found cidx: 3 event_index: 32, event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_set_papi_event_code:120:2266150:0 new event_code: 0x40000020, update_flag: 0, previous event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_eventcode_to_native:584:2266150:0 ENTER: event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_eventcode_to_native:597:2266150:0 EXIT: result: 0x1
INTERNAL:papi_internal.c:_papi_hwi_query_native_event:2459:2266150:0 EXIT: ret: 0
INTERNAL:papi_internal.c:add_native_events:1237:2266150:0 ENTER: ESI: 0x1ce14e0, nevt: 0x7ffc72352e84, size: 1, out: 0x1ce46d0
INTERNAL:papi_internal.c:event_already_in_eventset:1055:2266150:0 ENTER: ESI: 0x1ce14e0, papi_event: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_eventcode_to_native:584:2266150:0 ENTER: event_code: 0x40000020
INTERNAL:papi_internal.c:_papi_hwi_eventcode_to_native:597:2266150:0 EXIT: result: 0x1
INTERNAL:papi_internal.c:event_already_in_eventset:1072:2266150:0 SUBSTRATE:components/appio/appio.c:fwrite:393:2266150:0 appio: intercepted fwrite(0x7f6c5c9fa765,1,19,0x7f6c5aec4600)
EXIT: PAPI_ENOEVNT

...continues to success.

Is there anything else I should try in terms of shaking out a fix or getting more diagnostic information?

[Treece Burgess]

Hello Wyatt,

I have been able to recreate your issue on an Intel Xeon Gold 6140. I have looked further into the issue and as stands PAPI does not support multiplexing for the Coretemp component. I have created PR #357 that enables support for multiplexing with the Coretemp component. 

I tested PR #357 with the application code that I have linked in the PR and this resolved the error for myself. If you could please test and let me know if it does for you. That would be greatly appreciated.

Best wishes,

Treece

--
You received this message because you are subscribed to the Google Groups "ptools-perfapi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ptools-perfap...@icl.utk.edu.
To view this discussion visit https://groups.google.com/a/icl.utk.edu/d/msgid/ptools-perfapi/4db4c16c-e1ba-4979-9565-0f5c9916aa0en%40icl.utk.edu.

[Wyatt Spear]

This didn't seem to work for me; I'm still seeing a failure with this component and multiplexing after introducing the change. I'm happy to do more diagnostics on my side if that will help. Is there a way to check this support directly with a utility binary provided by papi? I only know how to engage the metric and multiplexing through TAU.

[Treece Burgess]

Hello Wyatt,

If you mean a utility binary to check if a component supports multiplexing then no, there is not one as of now. Based on your email it appears you are using PAPI through TAU. To see if this is an issue on the PAPI side we need a reproducer with only PAPI code.

I am attaching a basic application code that can start as a reproducer, if you need to modify it to be more inline with your code structure in TAU please do.

Could you please also send me the following information:

• The configure you use to build PAPI (e.g. ./configure --prefix=$PWD/test-install --with-components="appio coretemp XYZ" where XYZ is any other components you are configuring with)
• System and architecture of your machine (uname -a)
• CPU architecture (cat /proc/cpuinfo | grep "name" | head -n 1)
• GCC version

Best wishes,

Treece

[Wyatt Spear]

Greetings.  So I revisited this recently with the official papi 7.2.0 release and I'm still seeing the issue with this counter in TAU. With multiplexing on, it fails but with multiplexing off it works. Your verify_multiplex example works so whatever is going on it isn't replicating what is happening in TAU.

Relevant papi debug output:

API:papi.c:PAPI_get_event_component:7307:3653948:0 Entry: EventCode: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_component_index:529:3653948:0 ENTER: event_code: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_component_index:560:3653948:0 EXIT: Found cidx: 3 event_index: 33, event_code: 0x40000021
API:papi.c:PAPI_add_event:2384:3653948:0 Entry: EventSet: 3, EventCode: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_add_event:1544:3653948:0 ENTER: ESI: 0xe55190 (3), EventCode: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_component_index:529:3653948:0 ENTER: event_code: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_component_index:560:3653948:0 EXIT: Found cidx: 3 event_index: 33, event_code: 0x40000021
INTERNAL:papi_internal.c:_papi_hwi_add_event:1569:3653948:0 EXIT: Event is not valid for component index 3
API:papi.c:PAPI_add_event:2406:3653948:0 EXIT: return: -1

TAU: Error adding PAPI events: Invalid argument

Here is the whole papi verbose output from a failing run: https://home.nic.uoregon.edu/~wspear/papi-7.2.0/papi.log

Papi configuration: ./configure --prefix=/home/users/wspear/bin/tau-pdt/roberta/papi-7.2.0-install-gcc --with-components="appio coretemp infiniband io net"  --with-debug=yes

uname -a: Linux roberta 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 1 04:16:12 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

CPU Arch: model name      : AMD EPYC 9654 96-Core Processor

GCC: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-22)

If you want to take a look at this on the tau side this runs it from start to finish (with adjustments for your papi location, and your papi lib in LD_LIBRARY_PATH if needed). It will work with TAU_PAPI_MULTIPLEXING removed.:

wget tau.uoregon.edu/tau.tgz

 tar -zxf tau.tgz

cd tau-2.34.1/

./configure  -bfd=download -useropt=-g\ -O3\ -DTAU_MAX_METRICS=200 -papi=/home/users/wspear/bin/tau-pdt/roberta/papi-7.2.0-install-gcc/

make clean install -j

export TAU_MAKEFILE=`cat .active_stub`

export PATH=${TAU_MAKEFILE/lib*/bin}:$PATH

cd examples/papi

make

TAU_METRICS=TIME,PAPI_FP_OPS,PAPI_L2_DCM,PAPI_NATIVE_perf::PERF_COUNT_HW_CACHE_L1D:MISS,PAPI_NATIVE_coretemp:::hwmon2:temp3_input TAU_PAPI_MULTIPLEXING=1 PAPI_DEBUG=PAPI_ALL ./simple

Thanks,

Wyatt Spear

[Treece Burgess]

Hello Wyatt,

Thanks for reaching back out and I hope you are enjoying the new features of PAPI 7.2.0.

I took the steps that you provided and followed them for both PAPI and TAU. Along with this I have access to the machine Roberta and used that during testing. After working with the TAU code I came to the conclusion this is not a PAPI issue.

To explain I simplified the problem to just be:

TAU_METRICS=TIME,PAPI_NATIVE_coretemp:::hwmon2:temp3_input TAU_PAPI_MULTIPLEXING=1 ./simple

This still produced the error: TAU: Error adding PAPI events: Invalid Argument.

At some point during the TAU workflow. The function initializeThread is entered and there is a conditional block that checks if a user has set the TAU_PAPI_MULTIPLEXING environment variable. We do, so we enter this conditional block.

Within the aforementioned conditional block, the workflow is: PAPI_multiplex_init() -> PAPI_assign_eventset_component( localThreadValue->EventSet[i], 0). Due to this, the eventset is being assigned to component 0 or what is normally the perf_event component. This is causing the issue. If I change 0 to 3, where 3 corresponds to the coretemp component index. I do not run into the above error. A better fix would be to not hardcode component index 0 and instead assign it based on the events being added.

Best wishes,

Treece

On Tue, Jul 22, 2025 at 3:11 PM Wyatt Spear <wjs...@gmail.com> wrote:

Our public directory paths changed. The log is now located here: https://nic.uoregon.edu/~wspear/papi-7.2.0/papi.log

[Wyatt Spear]

Our public directory paths changed. The log is now located here: https://nic.uoregon.edu/~wspear/papi-7.2.0/papi.log

On Friday, July 11, 2025 at 11:57:52 AM UTC-7 Wyatt Spear wrote:

[Wyatt Spear]

Thank You very much for investigating that, Treece! I've just pushed a commit to TAU that I think fixes this without breaking anything else. If so it will be in our next release and we should have fully functional multiplexing with components.

Regards,

Wyatt Spear