Modify opcodes.c example to use 64-bit counters

[José Morgado]

Hi,

I have been playing around with the opcodes.c example that is provided in the samples folder, however in some situations my applications reach an opcode count higher than the limit of a regular uint, I was looking into how to change this example client to use 64-bit uint instead of 32-bit uint. I did notice this comment in the opcodes.c client itself:

 * XXX: we're using 32-bit counters.  64-bit counters are more challenging: they're
 * harder to make atomic on 32-bit x86, and drx does not yet support them on ARM.

So I suspect this might be impossible for ARM systems? But maybe for x86 systems? I have tried changing the data types used, however it seems the limitation is related with the function drx_insert_counter_update, is there someway to have this function use 64-bit uints?

Best regards,
José Morgado

[Abhinav Sharma]

Hi,

> I have tried changing the data types used, however it seems the limitation is related with the function drx_insert_counter_update, is there someway to have this function use 64-bit uints?

What was the outcome?

Have you tried DRX_COUNTER_64BIT (https://dynamorio.org/group__drx.html#gga4790f45dcc812c1b00184a2edccdddf5a507636af18a8858c214066682d013a1a)?

Also see the notes at drx_insert_counter_update (https://dynamorio.org/group__drx.html#gab5f7ae9128e953fdf1f8c2ac3292f07a)

I see DRX_COUNTER_64BIT is not supported on 32-bit ARM (the issue linked in code is https://github.com/DynamoRIO/dynamorio/issues/1551) but I think it is supported on AArch64. Did you try that and encounter any issues?

Hope this helps,

Abhinav

[José Morgado]

Hi again,

The outcome of just changing the data types was that it still lead to wrong results when counts went over the 32 bit limit, however I have inserted the DRX_COUNTER_64BIT flag, alongside the already existing lock flag, and now its working, both on x86 and AARCH64, however on AARCH64 it fails to count all the opcodes when multiple threads are involved, it just counts the opcodes for one thread, while on x86 it seems to be able to count the total from all threads, but I think this is already a known issue.

[Abhinav Sharma]

Hi,

> it just counts the opcodes for one thread, while on x86 it seems to be able to count the total from all threads, but I think this is already a known issue.

Do you mean because of the following known issue documented in code?

 * XXX: our counters are racy on ARM.  We use DRX_COUNTER_LOCK to make them atomic
 * (at a perf cost) on x86.

It says the counters are racy on ARM, so the results don't really represent any one thread; they're supposed to be global but they're wrong probably because of the race.

I see that we have DRX_COUNTER_REL_ACQ (https://dynamorio.org/group__drx.html#gga4790f45dcc812c1b00184a2edccdddf5aa50fccbd9d5dc9f300d6033c971f95e4) that works on AArch64. The opcodes.c sample is old and was probably never updated to use DRX_COUNTER_REL_ACQ.

Try modifying the drx_insert_counter_update call in opcodes.c (https://github.com/DynamoRIO/dynamorio/blob/092b4f2678ee5e9b7578aaf93a099a57e2e7d968/api/samples/opcodes.c#L245) to add DRX_COUNTER_REL_ACQ for AArch64. If it works, feel free to send a PR with this change.

Abhinav

[José Morgado]

Hi,

I tried your suggestion of using DRX_COUNTER_REL_ACQ, however with and without DRX_COUNTER_64BIT, the measured opcodes are not accurate with any number of threads (besides 1), the counted instructions always under count the real number of executed instructions, and as I increase the number of threads the results get farther and farther from what is expected. I am running some in-line GCC assembly (volatile) so I am pretty sure the instructions are being executed.
I have an equivalent set of assembly code for x86 that I tested in the same way as the AARCH64 one, and in this case the results are accurate even when scaling up the number of threads (and using 64 bit counters), so it seems it is an issue with the AARCH64 counting.
Any idea on how I could fix this (could it be my a problem on my side?), or should I just open an issue on github?

Thank you for your time,
José Morgado

[Abhinav Sharma]

Hi,

Apologies for the delayed response.

Yes, please file an issue on Github with steps to reproduce the issue.

Regards,

Abhinav