Deadlock at check_wait_at_safe_spot
55 views
Skip to first unread message
Mohammad Ewais
Dec 3, 2023, 4:44:30 PM12/3/23
to DynamoRIO Users
Hi,
I am running DR on an app with 8 threads. In my DR side of things, the 8 threads need to synchronize every now and then through a barrier. I take care so the barrier wouldn't conlfict or deadlock with threads waiting on a futex and so. Now I have the following situation, 7 of the 8 threads are waiting on my barrier. As far as I can tell, the 8th is fresh (or at least recently) out of a futex syscall, but does not arrive and wait at it yet (so the barrier has only 7 of 8 waiting, I verified that count with gdb). Instead, it enters a function `check_wait_at_safe_spot` where it gets stuck!
Here is my backtrace:
```
#0 check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
#1 0x00007f3621fbd453 in dispatch_enter_fcache (targetf=0x7f355e5eb768, dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:436
#2 d_r_dispatch (dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:239
#3 0x00007f35de3ac445 in ?? ()
#4 0x0000000000000000 in ?? ()
```
It keeps alternating between lines 723 and 735 in synch.c (DR 8.0.0-1). The code for which is here: https://github.com/DynamoRIO/dynamorio/blob/release_8.0.0-1/core/synch.c#L735
I was able to print these values from gdb (these are used at line 723 above):
```
(gdb) p tsd->synch_perm
$4 = THREAD_SYNCH_VALID_MCONTEXT
(gdb) p tsd->pending_synch_count
$5 = 1
```
It seems the 8th thread is waiting on some condition to be fulfilled by another thread, at the same time the other thread is waiting on the barrier for thread 8. What sort of dependency is this? And how may I avoid it or get rid of it?
I am running DR on an app with 8 threads. In my DR side of things, the 8 threads need to synchronize every now and then through a barrier. I take care so the barrier wouldn't conlfict or deadlock with threads waiting on a futex and so. Now I have the following situation, 7 of the 8 threads are waiting on my barrier. As far as I can tell, the 8th is fresh (or at least recently) out of a futex syscall, but does not arrive and wait at it yet (so the barrier has only 7 of 8 waiting, I verified that count with gdb). Instead, it enters a function `check_wait_at_safe_spot` where it gets stuck!
Here is my backtrace:
```
#0 check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
#1 0x00007f3621fbd453 in dispatch_enter_fcache (targetf=0x7f355e5eb768, dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:436
#2 d_r_dispatch (dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:239
#3 0x00007f35de3ac445 in ?? ()
#4 0x0000000000000000 in ?? ()
```
It keeps alternating between lines 723 and 735 in synch.c (DR 8.0.0-1). The code for which is here: https://github.com/DynamoRIO/dynamorio/blob/release_8.0.0-1/core/synch.c#L735
I was able to print these values from gdb (these are used at line 723 above):
```
(gdb) p tsd->synch_perm
$4 = THREAD_SYNCH_VALID_MCONTEXT
(gdb) p tsd->pending_synch_count
$5 = 1
```
It seems the 8th thread is waiting on some condition to be fulfilled by another thread, at the same time the other thread is waiting on the barrier for thread 8. What sort of dependency is this? And how may I avoid it or get rid of it?
Derek Bruening
Dec 4, 2023, 3:35:23 PM12/4/23
to Mohammad Ewais, DynamoRIO Users
The callstack of the thread that is trying to synch with the one at check_wait_at_safe_spot would help. The one initiating the synch has some reason like a client-requested flush or client-requested suspend-the-world, or a DR-initiated reset. The DR-initiated ones though are triggered by app actions. A separation of app and client/DR locks is what keeps things safe. What triggers your client's synchronization?
--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/04deae0c-fd39-4248-ac6a-f75c6267b11dn%40googlegroups.com.
Mohammad Ewais
Dec 4, 2023, 4:12:28 PM12/4/23
to DynamoRIO Users
Mybarrier/sync happens every specific count of instructions for all threads. So the callstack of the other thread would show nothing other than a BB inserted function, going through a bunch of my calls till it hits the barrier and waits:
```
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355deffac0 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dfebdb0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dfec210, thread=<optimized out>, address=<optimized out>, bb=0x7f355df4ede0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0921ad in ?? ()
#8 0x0000000000000007 in ?? ()
#9 0x00007f361df57010 in ?? ()
#10 0x00007ffe8f01e0e0 in ?? ()
#11 0x00007f355df4a720 in ?? ()
#12 0x0000000000000000 in ?? ()
#1 0x00007f355deffac0 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dfebdb0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dfec210, thread=<optimized out>, address=<optimized out>, bb=0x7f355df4ede0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0921ad in ?? ()
#8 0x0000000000000007 in ?? ()
#9 0x00007f361df57010 in ?? ()
#10 0x00007ffe8f01e0e0 in ?? ()
#11 0x00007f355df4a720 in ?? ()
#12 0x0000000000000000 in ?? ()
```
The sequence I am seeing here is the following:
Thread 0: Clears Code Cache for everyone -> Instruments new BB -> Waits at barrier
Thread 0: Clears Code Cache for everyone -> Instruments new BB -> Waits at barrier
Thread 8 (the one in question): Gets futex syscall -> Exits syscall -> Gets stuck
Based on this and your response above, it might be the client-requested flush causing the issues. But then what is a safe way to flush without deadlocking threads like so? Should I use dr_delay_flush_region instead?
PS: The barrier I am using is implemented with atomics, no locks there. I do use mutexes elsewhere so will check if any of them are locked while this happens. But I don't fully understand whether the cause is the flush or mutexes being held?!
PS: The barrier I am using is implemented with atomics, no locks there. I do use mutexes elsewhere so will check if any of them are locked while this happens. But I don't fully understand whether the cause is the flush or mutexes being held?!
Derek Bruening
Dec 4, 2023, 4:21:57 PM12/4/23
to Mohammad Ewais, DynamoRIO Users
But if one thread is waiting at check_wait_at_safe_spot, then some other thread is actively trying to synch with that thread: so that other thread should have a callstack in DR code. Understanding the request from that other thread for this target one to wait is key to understanding what is happening.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/9413334c-2872-468b-8b3e-92d13e94246bn%40googlegroups.com.
Mohammad Ewais
Dec 4, 2023, 4:42:09 PM12/4/23
to DynamoRIO Users
I see your point, but something is up. All other 7 threads are at the exact stacktrace I showed above, all at the Enter function of my barrier. Only the 8th is waiting for this synch!!!
Here's everything:
```
GNU gdb (Ubuntu 10.1-2ubuntu2) 10.1.90.20210411-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 168806
[New LWP 168807]
[New LWP 168808]
[New LWP 168809]
[New LWP 168810]
[New LWP 168811]
[New LWP 168812]
[New LWP 168813]
0x00007f3622094237 in syscall_ready ()
0x7f362236da00 <gdb_priv_cmds>: "add-symbol-file '/work/mewais/DCArch/DCSim/Build//libDCSClient.so' 0x00007f35de07eb40\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so' 0x00007f3621f979e0\nadd-sy"...
0x7f362236dac8 <gdb_priv_cmds+200>: "mbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrwrap.so' 0x00007f35de3cbd90\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrmgr.so' 0x00007f"...
0x7f362236db90 <gdb_priv_cmds+400>: "35de5d84c0\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrutil.so' 0x00007f35de7eb4e0\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libd"...
0x7f362236dc58 <gdb_priv_cmds+600>: "rx.so' 0x00007f35de9f2c80\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrreg.so' 0x00007f35debff770\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib6"...
0x7f362236dd20 <gdb_priv_cmds+800>: "4/release/libdrsyms.so' 0x00007f35dee12650\nadd-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007f3621b74650\nadd-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007f3621eea090\nadd-symbol-file '/l"...
0x7f362236dde8 <gdb_priv_cmds+1000>: "ib/x86_64-linux-gnu/libstdc++.so.6' 0x00007f36217c02e0\nadd-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007f3621da93c0\nadd-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007f3621d815e0\n"
0x7f362236deae <gdb_priv_cmds+1198>: ""
0x7f362236deaf <gdb_priv_cmds+1199>: ""
Loading gdb scripts for debugging DynamoRIO...
(gdb) add-symbol-file '/work/mewais/DCArch/DCSim/Build//libDCSClient.so' 0x00007f35de07eb40
add symbol table from file "/work/mewais/DCArch/DCSim/Build//libDCSClient.so" at
.text_addr = 0x7f35de07eb40
Reading symbols from /work/mewais/DCArch/DCSim/Build//libDCSClient.so...
(gdb) add-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so' 0x00007f3621f979e0
add symbol table from file "/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so" at
.text_addr = 0x7f3621f979e0
Reading symbols from /tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so...
Reading symbols from /tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so.debug...
Loading gdb scripts for debugging DynamoRIO...
(gdb) thread 1
[Switching to thread 1 (LWP 168806)]
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355deffac0 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dfebdb0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dfec210, thread=<optimized out>, address=<optimized out>, bb=0x7f355df4ede0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0921ad in ?? ()
#8 0x0000000000000007 in ?? ()
#9 0x00007f361df57010 in ?? ()
#10 0x00007ffe8f01e0e0 in ?? ()
#11 0x00007f355df4a720 in ?? ()
#12 0x0000000000000000 in ?? ()
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#0 0x00007f3622094237 in syscall_ready ()
#0 0x00007f3622094237 in syscall_ready ()
check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
#0 check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
#1 0x00007f3621fbd453 in dispatch_enter_fcache (targetf=0x7f355e5eb768, dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:436
#2 d_r_dispatch (dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:239
#3 0x00007f35de3ac445 in ?? ()
#4 0x0000000000000000 in ?? ()
Here's everything:
```
GNU gdb (Ubuntu 10.1-2ubuntu2) 10.1.90.20210411-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 168806
[New LWP 168807]
[New LWP 168808]
[New LWP 168809]
[New LWP 168810]
[New LWP 168811]
[New LWP 168812]
[New LWP 168813]
0x00007f3622094237 in syscall_ready ()
0x7f362236da00 <gdb_priv_cmds>: "add-symbol-file '/work/mewais/DCArch/DCSim/Build//libDCSClient.so' 0x00007f35de07eb40\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so' 0x00007f3621f979e0\nadd-sy"...
0x7f362236dac8 <gdb_priv_cmds+200>: "mbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrwrap.so' 0x00007f35de3cbd90\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrmgr.so' 0x00007f"...
0x7f362236db90 <gdb_priv_cmds+400>: "35de5d84c0\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrutil.so' 0x00007f35de7eb4e0\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libd"...
0x7f362236dc58 <gdb_priv_cmds+600>: "rx.so' 0x00007f35de9f2c80\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib64/release/libdrreg.so' 0x00007f35debff770\nadd-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/ext/lib6"...
0x7f362236dd20 <gdb_priv_cmds+800>: "4/release/libdrsyms.so' 0x00007f35dee12650\nadd-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007f3621b74650\nadd-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007f3621eea090\nadd-symbol-file '/l"...
0x7f362236dde8 <gdb_priv_cmds+1000>: "ib/x86_64-linux-gnu/libstdc++.so.6' 0x00007f36217c02e0\nadd-symbol-file '/lib/x86_64-linux-gnu/libm.so.6' 0x00007f3621da93c0\nadd-symbol-file '/lib/x86_64-linux-gnu/libgcc_s.so.1' 0x00007f3621d815e0\n"
0x7f362236deae <gdb_priv_cmds+1198>: ""
0x7f362236deaf <gdb_priv_cmds+1199>: ""
Loading gdb scripts for debugging DynamoRIO...
(gdb) add-symbol-file '/work/mewais/DCArch/DCSim/Build//libDCSClient.so' 0x00007f35de07eb40
add symbol table from file "/work/mewais/DCArch/DCSim/Build//libDCSClient.so" at
.text_addr = 0x7f35de07eb40
Reading symbols from /work/mewais/DCArch/DCSim/Build//libDCSClient.so...
(gdb) add-symbol-file '/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so' 0x00007f3621f979e0
add symbol table from file "/tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so" at
.text_addr = 0x7f3621f979e0
Reading symbols from /tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so...
Reading symbols from /tools/DynamoRIO/DynamoRIO-Linux-8.0.0-1/lib64/release/libdynamorio.so.debug...
Loading gdb scripts for debugging DynamoRIO...
(gdb) thread 1
[Switching to thread 1 (LWP 168806)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355deffac0 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dfebdb0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dfec210, thread=<optimized out>, address=<optimized out>, bb=0x7f355df4ede0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0921ad in ?? ()
#8 0x0000000000000007 in ?? ()
#9 0x00007f361df57010 in ?? ()
#10 0x00007ffe8f01e0e0 in ?? ()
#11 0x00007f355df4a720 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (LWP 168807)]
[Switching to thread 2 (LWP 168807)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355df9c280 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355df8edf0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355df976a0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e823bf8) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0e3b4d in ?? ()
#8 0x00007f361df520c4 in ?? ()
#9 0x0000000000000001 in ?? ()
#10 0x00007f3621318dc0 in ?? ()
#11 0x00007f355e536720 in ?? ()
#12 0x00007f361df6b140 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (LWP 168808)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355df976a0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e823bf8) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df0e3b4d in ?? ()
#8 0x00007f361df520c4 in ?? ()
#9 0x0000000000000001 in ?? ()
#10 0x00007f3621318dc0 in ?? ()
#11 0x00007f355e536720 in ?? ()
#12 0x00007f361df6b140 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (LWP 168808)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355e4faa80 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dff6e70) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff74f0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e5fd640) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df1132bd in ?? ()
#8 0x00007f361df6b0e0 in ?? ()
#9 0x0000000000000008 in ?? ()
#10 0x00007f3620b17e30 in ?? ()
#11 0x00007f355e573720 in ?? ()
#12 0x00007f361df6b138 in ?? ()
#13 0x00007f361dfe5800 in ?? ()
#14 0x0000000000000000 in ?? ()
(gdb) thread 4
[Switching to thread 4 (LWP 168809)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff74f0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e5fd640) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df1132bd in ?? ()
#8 0x00007f361df6b0e0 in ?? ()
#9 0x0000000000000008 in ?? ()
#10 0x00007f3620b17e30 in ?? ()
#11 0x00007f355e573720 in ?? ()
#12 0x00007f361df6b138 in ?? ()
#13 0x00007f361dfe5800 in ?? ()
#14 0x0000000000000000 in ?? ()
(gdb) thread 4
[Switching to thread 4 (LWP 168809)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355e504180 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dff9600) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff9c80, thread=<optimized out>, address=<optimized out>, bb=0x7f355e686568) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df15ae11 in ?? ()
#8 0x00007f3620317948 in ?? ()
#9 0x0000000000000000 in ?? ()
(gdb) thread 5
[Switching to thread 5 (LWP 168810)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff9c80, thread=<optimized out>, address=<optimized out>, bb=0x7f355e686568) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df15ae11 in ?? ()
#8 0x00007f3620317948 in ?? ()
#9 0x0000000000000000 in ?? ()
(gdb) thread 5
[Switching to thread 5 (LWP 168810)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355e507640 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dffa330) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dffa9b0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e6b2898) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df172a39 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb) thread 6
[Switching to thread 6 (LWP 168811)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dffa9b0, thread=<optimized out>, address=<optimized out>, bb=0x7f355e6b2898) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df172a39 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb) thread 6
[Switching to thread 6 (LWP 168811)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355e501400 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dff88d0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff8f50, thread=<optimized out>, address=<optimized out>, bb=0x7f355e657258) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df142a0d in ?? ()
#8 0x00007f361df6b140 in ?? ()
#9 0x000000000000004e in ?? ()
#10 0x00007f361f314dc0 in ?? ()
#11 0x00007f355e5a3720 in ?? ()
#12 0x00007f361df6b140 in ?? ()
#13 0x000000000000004e in ?? ()
#14 0x00007f35de3acd02 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb) thread 7
[Switching to thread 7 (LWP 168812)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff8f50, thread=<optimized out>, address=<optimized out>, bb=0x7f355e657258) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df142a0d in ?? ()
#8 0x00007f361df6b140 in ?? ()
#9 0x000000000000004e in ?? ()
#10 0x00007f361f314dc0 in ?? ()
#11 0x00007f355e5a3720 in ?? ()
#12 0x00007f361df6b140 in ?? ()
#13 0x000000000000004e in ?? ()
#14 0x00007f35de3acd02 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb) thread 7
[Switching to thread 7 (LWP 168812)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x00007f355e4fe340 in ?? ()
#2 0x00007f362200f239 in dr_thread_yield () at /home/travis/build/DynamoRIO/dynamorio/core/lib/instrument.c:5053
#3 0x00007f35de199afd in DCS::EVENT::DynamicEntryExitBarrier::Enter (this=this@entry=0x7f35de3a06a0) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:102
#4 0x00007f35de19ac49 in DCS::EVENT::SimulateEvents (phase_end_cycle=2000) at /work/mewais/DCArch/DCSim/Sources/DCS/Event/Simulation.cpp:138
#5 0x00007f35de19f631 in DCS::CORE::Scheduler::SimulatePhase (this=0x7f355dff7ba0) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Scheduler.cpp:104
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff8220, thread=<optimized out>, address=<optimized out>, bb=0x7f355e62b898) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df12aa39 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb) thread 8
[Switching to thread 8 (LWP 168813)]
#6 0x00007f35de1a6b88 in DCS::CORE::Simple::BasicBlock (this=0x7f355dff8220, thread=<optimized out>, address=<optimized out>, bb=0x7f355e62b898) at /work/mewais/DCArch/DCSim/Sources/DCS/Cores/Simple.cpp:67
#7 0x00007f35df12aa39 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb) thread 8
[Switching to thread 8 (LWP 168813)]
#0 0x00007f3622094237 in syscall_ready ()
(gdb) where
#0 0x00007f3622094237 in syscall_ready ()
#1 0x0000000000000000 in ?? ()
(gdb) n
Single stepping until exit from function syscall_ready,
which has no line number information.
os_thread_sleep (milliseconds=<optimized out>) at /home/travis/build/DynamoRIO/dynamorio/core/unix/os.c:3495
3495 /home/travis/build/DynamoRIO/dynamorio/core/unix/os.c: No such file or directory.
(gdb) n
(gdb) n
Single stepping until exit from function syscall_ready,
which has no line number information.
os_thread_sleep (milliseconds=<optimized out>) at /home/travis/build/DynamoRIO/dynamorio/core/unix/os.c:3495
3495 /home/travis/build/DynamoRIO/dynamorio/core/unix/os.c: No such file or directory.
(gdb) n
check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
723 /home/travis/build/DynamoRIO/dynamorio/core/synch.c: No such file or directory.
(gdb) where
(gdb) where
#0 check_wait_at_safe_spot (dcontext=dcontext@entry=0x7f355e47c2c0, cur_state=cur_state@entry=THREAD_SYNCH_VALID_MCONTEXT) at /home/travis/build/DynamoRIO/dynamorio/core/synch.c:723
#1 0x00007f3621fbd453 in dispatch_enter_fcache (targetf=0x7f355e5eb768, dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:436
#2 d_r_dispatch (dcontext=0x7f355e47c2c0) at /home/travis/build/DynamoRIO/dynamorio/core/dispatch.c:239
#3 0x00007f35de3ac445 in ?? ()
#4 0x0000000000000000 in ?? ()
(gdb)
```
```
Message has been deleted
Mohammad Ewais
Dec 7, 2023, 2:46:32 PM12/7/23
to DynamoRIO Users
I think I found the locks causing the issues. Simply marking them as app locks solved it for me.
Sorry for the trouble.
Derek Bruening
Dec 7, 2023, 3:05:34 PM12/7/23
to Mohammad Ewais, DynamoRIO Users
Thanks for reporting the solution, glad you've figured it out. I was about to ask about locks you're using, going back to the primary invariant of keeping app and client/DR locks disjoint.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/c2a9e9fd-10c1-4528-92ce-aecdf31b850an%40googlegroups.com.
Mohammad Ewais
Dec 7, 2023, 3:07:30 PM12/7/23
to DynamoRIO Users
Thanks a lot for your help.
Mohammad Ewais
Dec 12, 2023, 9:47:06 AM12/12/23
to DynamoRIO Users
Very sorry to reopen this, but "when" exactly does this synch happen? Can it happen anytime or only at specific checkpoints?
I am having the same issue again. This time I am sure that no mutexes are being locked at all when the issue happens. So looking for other causes, I started enabling/disabling pieces of code to find out what's causing the deadlock. Interestingly, my barrier is the one causing the deadlock, if I comment it out everything works smoothly (functionally incorrect, but no deadlock).
The problem is the barrier has no locks or mutexes at all, not std or DR ones. It is only built with atomics and loops. I understand that a compiler can sometimes generate locks instead of atomics, but the barrier is using cmpxchg16 instructions in inline assembly so no chance of that happenning. The barrier can be entered from a clean call at the beginning of a BB, or from a pre/post system call function (futex syscall only). Naturally, as a barrier, it could cause the threads to wait at these locations for a while (hence my question above about when the synch can happen)
I am having the same issue again. This time I am sure that no mutexes are being locked at all when the issue happens. So looking for other causes, I started enabling/disabling pieces of code to find out what's causing the deadlock. Interestingly, my barrier is the one causing the deadlock, if I comment it out everything works smoothly (functionally incorrect, but no deadlock).
The problem is the barrier has no locks or mutexes at all, not std or DR ones. It is only built with atomics and loops. I understand that a compiler can sometimes generate locks instead of atomics, but the barrier is using cmpxchg16 instructions in inline assembly so no chance of that happenning. The barrier can be entered from a clean call at the beginning of a BB, or from a pre/post system call function (futex syscall only). Naturally, as a barrier, it could cause the threads to wait at these locations for a while (hence my question above about when the synch can happen)
Reply all
Reply to author
Forward
0 new messages