Private code cache for Aarch64

[Artem Shcherbak]

Hi, @derekbruening and @AssadHashmi.

Is there any way to enable private code cache for Aarch64?

My question is related to what I am now trying to do detach in the case when several threads are executing the same code and periodically unlink doesn`t come through(probably race condition) and I want to check the reproducibility of the problem with private code caches.

BR,

Artem

[Derek Bruening]

Thread-private code caches were unfortunately never implemented for AArch64: https://github.com/DynamoRIO/dynamorio/issues/1884.  Help adding such support would be appreciated.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/60219a73-850c-4253-9ce0-22e80ea910dcn%40googlegroups.com.

[Derek Bruening]

What type of detach are you looking at: is it internally-triggered?  We have been successfully using such detach on large AArch64 applications.

[Artem Shcherbak]

We use an external signal mechanism (nudge) for detaching and for sharing fragments this can lead to a situation where we do not go to the dispatcher although unlink fragment is called and patch stab occurs. Adding a call to flush_fragment_unlink_shared at the end of the unlink_fragment_for_signal helps us ensure exit to the dispatcher every time. I hope it's a good solution but I'm not sure.

среда, 10 июля 2024 г. в 07:18:06 UTC+8, Derek Bruening:

[Derek Bruening]

On Tue, Jul 9, 2024 at 10:16 PM Artem Shcherbak <artem.sh...@gmail.com> wrote:

We use an external signal mechanism (nudge) for detaching and for sharing fragments this can lead to a situation where we do not go to the dispatcher although unlink fragment is called and patch stab occurs. Adding a call to flush_fragment_unlink_shared at the end of the unlink_fragment_for_signal helps us ensure exit to the dispatcher every time. I hope it's a good solution but I'm not sure.

No, flush_fragments_unlink_shared() prepares for removal from the cache with steps like removing from the indirect branch tables and removing from vmareas, which can cause problems later.  That should not be called for fragments that are not about to be deleted.  I would delve into why a thread is not exiting: use the debugger to see where it is.  Is this with added instrumentation and the instrumentation is making blocking syscalls mid-fragment?  Does this happen without any client?

 

среда, 10 июля 2024 г. в 07:18:06 UTC+8, Derek Bruening:

What type of detach are you looking at: is it internally-triggered?  We have been successfully using such detach on large AArch64 applications.

On Tue, Jul 9, 2024 at 7:16 PM Derek Bruening <brue...@google.com> wrote:

Thread-private code caches were unfortunately never implemented for AArch64: https://github.com/DynamoRIO/dynamorio/issues/1884.  Help adding such support would be appreciated.

On Mon, Jul 8, 2024 at 6:37 AM Artem Shcherbak <artem.sh...@gmail.com> wrote:

Hi, @derekbruening and @AssadHashmi.

Is there any way to enable private code cache for Aarch64?

My question is related to what I am now trying to do detach in the case when several threads are executing the same code and periodically unlink doesn`t come through(probably race condition) and I want to check the reproducibility of the problem with private code caches.

BR,

Artem

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/60219a73-850c-4253-9ce0-22e80ea910dcn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/9a131b2b-f89b-402b-b332-6ba1fb0e125bn%40googlegroups.com.

[Artem Shcherbak]

The thread is in client code. This happens with the drcachesim client, without the client this problem does not occur. This issue occurs when an application executes the same fragment into multiple threads. If the fragments are different, then there are no problems with exit to dispatcher and make detach.

среда, 10 июля 2024 г. в 20:00:54 UTC+3, Derek Bruening:

[Derek Bruening]

• What is the precise AArch64 processor hitting the failure to unlink?
• What kind of fragment-final branch is involved?  Is it a direct-branch-reachable target where the exit stub is direct, or is it too far away and the exit stub is indirect (see https://dynamorio.org/page_aarch64_far.html#autotoc_md11)?  Presumably it is the former, as the latter should be a data change and not a code change and there should be no corner case issues.
• If you force it to use a far indirect link does the problem go away?  (We had proposed adding an off-by-default option to force this for all branches but I don't think that ever happened so this would require a code change.)
• If you add a membarrier system call passing MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE on the unlink (maybe inside clear_icache()?) does the problem go away?
• Xref https://github.com/DynamoRIO/dynamorio/issues/1911

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/2e56f555-1a02-41a0-9389-44429d1a0f7en%40googlegroups.com.

[Artem Shcherbak]

• We use kunpeng 920 machine.
• Yes,  direct-branch happens in our case.
• I redirected the stub change to a indirect branch via call unlink_inderect_exit() instead unlink_direct_exit(). It didn't change the picture, moreover, for the case where the fragments in the threads different detach stopped working with this change, so maybe I did something wrong.
•  Adding the -enable_traces option solved the problem.
• And for stable reproduction of the problem I pin the application with several threads(executed the same code) on the one core. 

понедельник, 15 июля 2024 г. в 18:23:45 UTC+3, Derek Bruening:

[Derek Bruening]

What is the explanation for -enable_traces helping?  You had the threads pinned so it hit the same unlink-multiple-fragment situation?  Why would the unlink be any different?  Is the final branch in the trace fragment far or indirect and that's the difference (and your earlier attempt to make everything far maybe had something wrong)?

Also note that -enable_traces is not fully supported on arm: https://github.com/DynamoRIO/dynamorio/issues/2974.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/3132af05-e67e-4b53-af88-3b7dde72ad23n%40googlegroups.com.