Running the Tests Locally
33 views
Skip to first unread message
Palmer Dabbelt
Nov 18, 2025, 12:32:24 AM (11 days ago) Nov 18
to DynamoRIO Users
I've been trying to use DynamoRIO and have run into a bunch of stability issues, I'm trying to figure out if it's an issue with my setup or if it's something that's broken for everyone. Unfortunately the issues manifest as just sort of add behavior when running workloads -- simple stuff works, but trying to run any of our benchmarks results in issues ranging from incorrect trace files to application crashes/hangs.
One thing I noticed is that I get quite a few test failures when running locally. On AArch64 (my target platform) I get 83 test failures, and on x86_64 I get 65. I'm not sure if that's the expected behavior?
Here's my test failure lists:
[aarch64]
The following tests FAILED:
1 - core_unit_tests (SEGFAULT)
7 - tool.drcachesim.core_sharded (SEGFAULT)
22 - drmemtrace_proj (Failed)
101 - code_api|linux.vfork (Failed)
108 - code_api|pthreads.pthreads_fork_FLAKY (Failed)
110 - code_api|linux.reset (Failed)
111 - code_api|linux.rseq (Failed)
112 - code_api|linux.rseq_table (Failed)
113 - code_api|linux.rseq_noarray (Failed)
123 - code_api|security-common.TestMemProtChg_FLAKY (Failed)
134 - code_api|client.exception (Failed)
138 - code_api|client.attach_test (Failed)
139 - code_api|client.detach_test (Timeout)
140 - code_api|client.attach_blocking (Failed)
141 - code_api|client.attach_state (Failed)
142 - code_api|client.memory_dump_test (Failed)
143 - code_api|client.attach_memory_dump_test (Failed)
146 - code_api|client.nudge_test (Failed)
170 - code_api|client.gonative (Failed)
188 - code_api|client.low_on_memory (Failed)
207 - code_api|sample.bbcount (Failed)
208 - code_api|sample.bbsize (Failed)
209 - code_api|sample.div (Failed)
211 - code_api|sample.memtrace_simple (Failed)
212 - code_api|sample.memval_simple (Failed)
213 - code_api|sample.memval_simple_scattergather (Failed)
214 - code_api|sample.instrace_simple (Failed)
215 - code_api|sample.opcode_count (Failed)
217 - code_api|sample.instrcalls (Failed)
218 - code_api|sample.cbrtrace (Failed)
219 - code_api|sample.wrap (Failed)
220 - code_api|sample.signal (Failed)
221 - code_api|sample.syscall (Failed)
223 - code_api|sample.inline (Failed)
224 - code_api|sample.inscount (Failed)
225 - code_api|sample.inscount.cleancall (Failed)
226 - code_api|sample.inscount.prof-pcs.cleancall (Failed)
227 - code_api|sample.opcodes (Failed)
228 - code_api|sample.stl_test (Failed)
305 - code_api|tool.drcacheoff.rseq (Failed)
306 - code_api|tool.drcacheoff.rseq-filter (Failed)
307 - code_api|tool.drcacheoff.rseq-dfilter (Failed)
308 - code_api|tool.drcacheoff.burst_aarch64_sys (Failed)
309 - code_api|tool.drcacheoff.gencode (Failed)
310 - code_api|tool.drcacheoff.gencode_filtered (Failed)
311 - code_api|tool.drcacheoff.burst_static (Failed)
312 - code_api|tool.drcacheoff.burst_replace (Failed)
313 - code_api|tool.drcacheoff.burst_replaceall (Failed)
315 - code_api|tool.drcacheoff.burst_maps (Failed)
316 - code_api|tool.drcacheoff.burst_syscall_inject (Failed)
317 - code_api|tool.drcacheoff.burst_traceopts (Failed)
318 - code_api|tool.drcacheoff.windows-timestamps (Failed)
319 - code_api|tool.drcacheoff.delay-ts (Failed)
320 - code_api|tool.drcacheoff.burst_futex (Failed)
321 - code_api|tool.drcacheoff.scale_time (Failed)
322 - code_api|tool.drcacheoff.burst_sleep (Failed)
323 - code_api|tool.drcacheoff.burst_threads (Failed)
324 - code_api|tool.drcacheoff.synch_attach (Failed)
325 - code_api|tool.drcacheoff.burst_threads_counts (Failed)
326 - code_api|tool.drcacheoff.burst_malloc (Failed)
327 - code_api|tool.drcacheoff.burst_reattach (Failed)
328 - code_api|tool.drcacheoff.burst_threadfilter (Failed)
329 - code_api|tool.drcacheoff.burst_threadL0filter (Failed)
330 - code_api|tool.drcacheoff.burst_client (Failed)
331 - code_api|tool.drcacheoff.purestatic (Failed)
339 - code_api|tool.record_filter (Failed)
375 - code_api|client.drx_time_scale-test (Failed)
376 - code_api|client.drx_sleep_scale-test (Failed)
377 - code_api|client.drx_timeout_scale-test (Failed)
378 - code_api|client.drwrap-test-detach (Failed)
388 - code_api|api.startstop (Failed)
389 - code_api|api.detach (Failed)
390 - code_api|api.detach_state (Failed)
391 - code_api|api.detach_signal (Failed)
392 - code_api|api.detach_spawn (Failed)
393 - code_api|api.detach_spawn_stress_FLAKY (Failed)
394 - code_api|api.detach_spawn_quick_exit (Failed)
402 - code_api|api.static_signal (Failed)
403 - code_api|api.static_crash (Failed)
404 - code_api|api.static_sideline (Timeout)
407 - code_api|api.static_maps_mixup_novars_FLAKY (Failed)
413 - code_api|api.rseq (Failed)
419 - code_api,satisfy_w_xor_x|linux.vfork (Failed)
[aarch64]
The following tests FAILED:
1 - core_unit_tests (SEGFAULT)
7 - tool.drcachesim.core_sharded (SEGFAULT)
22 - drmemtrace_proj (Failed)
101 - code_api|linux.vfork (Failed)
108 - code_api|pthreads.pthreads_fork_FLAKY (Failed)
110 - code_api|linux.reset (Failed)
111 - code_api|linux.rseq (Failed)
112 - code_api|linux.rseq_table (Failed)
113 - code_api|linux.rseq_noarray (Failed)
123 - code_api|security-common.TestMemProtChg_FLAKY (Failed)
134 - code_api|client.exception (Failed)
138 - code_api|client.attach_test (Failed)
139 - code_api|client.detach_test (Timeout)
140 - code_api|client.attach_blocking (Failed)
141 - code_api|client.attach_state (Failed)
142 - code_api|client.memory_dump_test (Failed)
143 - code_api|client.attach_memory_dump_test (Failed)
146 - code_api|client.nudge_test (Failed)
170 - code_api|client.gonative (Failed)
188 - code_api|client.low_on_memory (Failed)
207 - code_api|sample.bbcount (Failed)
208 - code_api|sample.bbsize (Failed)
209 - code_api|sample.div (Failed)
211 - code_api|sample.memtrace_simple (Failed)
212 - code_api|sample.memval_simple (Failed)
213 - code_api|sample.memval_simple_scattergather (Failed)
214 - code_api|sample.instrace_simple (Failed)
215 - code_api|sample.opcode_count (Failed)
217 - code_api|sample.instrcalls (Failed)
218 - code_api|sample.cbrtrace (Failed)
219 - code_api|sample.wrap (Failed)
220 - code_api|sample.signal (Failed)
221 - code_api|sample.syscall (Failed)
223 - code_api|sample.inline (Failed)
224 - code_api|sample.inscount (Failed)
225 - code_api|sample.inscount.cleancall (Failed)
226 - code_api|sample.inscount.prof-pcs.cleancall (Failed)
227 - code_api|sample.opcodes (Failed)
228 - code_api|sample.stl_test (Failed)
305 - code_api|tool.drcacheoff.rseq (Failed)
306 - code_api|tool.drcacheoff.rseq-filter (Failed)
307 - code_api|tool.drcacheoff.rseq-dfilter (Failed)
308 - code_api|tool.drcacheoff.burst_aarch64_sys (Failed)
309 - code_api|tool.drcacheoff.gencode (Failed)
310 - code_api|tool.drcacheoff.gencode_filtered (Failed)
311 - code_api|tool.drcacheoff.burst_static (Failed)
312 - code_api|tool.drcacheoff.burst_replace (Failed)
313 - code_api|tool.drcacheoff.burst_replaceall (Failed)
315 - code_api|tool.drcacheoff.burst_maps (Failed)
316 - code_api|tool.drcacheoff.burst_syscall_inject (Failed)
317 - code_api|tool.drcacheoff.burst_traceopts (Failed)
318 - code_api|tool.drcacheoff.windows-timestamps (Failed)
319 - code_api|tool.drcacheoff.delay-ts (Failed)
320 - code_api|tool.drcacheoff.burst_futex (Failed)
321 - code_api|tool.drcacheoff.scale_time (Failed)
322 - code_api|tool.drcacheoff.burst_sleep (Failed)
323 - code_api|tool.drcacheoff.burst_threads (Failed)
324 - code_api|tool.drcacheoff.synch_attach (Failed)
325 - code_api|tool.drcacheoff.burst_threads_counts (Failed)
326 - code_api|tool.drcacheoff.burst_malloc (Failed)
327 - code_api|tool.drcacheoff.burst_reattach (Failed)
328 - code_api|tool.drcacheoff.burst_threadfilter (Failed)
329 - code_api|tool.drcacheoff.burst_threadL0filter (Failed)
330 - code_api|tool.drcacheoff.burst_client (Failed)
331 - code_api|tool.drcacheoff.purestatic (Failed)
339 - code_api|tool.record_filter (Failed)
375 - code_api|client.drx_time_scale-test (Failed)
376 - code_api|client.drx_sleep_scale-test (Failed)
377 - code_api|client.drx_timeout_scale-test (Failed)
378 - code_api|client.drwrap-test-detach (Failed)
388 - code_api|api.startstop (Failed)
389 - code_api|api.detach (Failed)
390 - code_api|api.detach_state (Failed)
391 - code_api|api.detach_signal (Failed)
392 - code_api|api.detach_spawn (Failed)
393 - code_api|api.detach_spawn_stress_FLAKY (Failed)
394 - code_api|api.detach_spawn_quick_exit (Failed)
402 - code_api|api.static_signal (Failed)
403 - code_api|api.static_crash (Failed)
404 - code_api|api.static_sideline (Timeout)
407 - code_api|api.static_maps_mixup_novars_FLAKY (Failed)
413 - code_api|api.rseq (Failed)
419 - code_api,satisfy_w_xor_x|linux.vfork (Failed)
[x86_64]
The following tests FAILED:
8 - tool.drcachesim.core_sharded (SEGFAULT)
30 - drmemtrace_proj (Failed)
31 - linux.xarch (Failed)
128 - code_api|linux.reset (Failed)
129 - code_api|linux.rseq (Failed)
130 - code_api|linux.rseq_table (Failed)
131 - code_api|linux.rseq_noarray (Failed)
145 - code_api|security-common.TestAllocWE (Failed)
146 - code_api|security-common.TestMemProtChg_FLAKY (Failed)
154 - code_api|client.alloc (Failed)
164 - code_api|client.exception (Failed)
192 - code_api|client.attach_test (Failed)
193 - code_api|client.detach_test (Failed)
194 - code_api|client.attach_blocking (Failed)
195 - code_api|client.memory_dump_test (Failed)
196 - code_api|client.attach_memory_dump_test (Failed)
199 - code_api|client.nudge_test (Failed)
247 - code_api|client.drx-scattergather (Failed)
248 - code_api|client.drx-scattergather-bbdup (Failed)
269 - code_api|sample.bbcount (Failed)
270 - code_api|sample.bbsize (Failed)
272 - code_api|sample.div (Failed)
274 - code_api|sample.memtrace_simple (Failed)
276 - code_api|sample.memval_simple (Failed)
277 - code_api|sample.memval_simple_scattergather (Failed)
278 - code_api|sample.instrace_simple (Failed)
279 - code_api|sample.opcode_count (Failed)
280 - code_api|sample.cbr (Failed)
281 - code_api|sample.countcalls (Failed)
282 - code_api|sample.inc2add (Failed)
283 - code_api|sample.memtrace_x86_binary (Failed)
284 - code_api|sample.memtrace_x86_text (Failed)
285 - code_api|sample.instrace_x86_binary (Failed)
286 - code_api|sample.instrace_x86_text (Failed)
288 - code_api|sample.ssljack (Failed)
289 - code_api|sample.modxfer (Failed)
290 - code_api|sample.modxfer_app2lib (Failed)
291 - code_api|sample.hot_bbcount (Failed)
292 - code_api|sample.instrcalls (Failed)
293 - code_api|sample.cbrtrace (Failed)
294 - code_api|sample.wrap (Failed)
295 - code_api|sample.signal (Failed)
296 - code_api|sample.syscall (Failed)
298 - code_api|sample.inline (Failed)
299 - code_api|sample.inscount (Failed)
300 - code_api|sample.inscount.cleancall (Failed)
301 - code_api|sample.inscount.prof-pcs.cleancall (Failed)
302 - code_api|sample.opcodes (Failed)
303 - code_api|sample.stl_test (Failed)
351 - code_api|tool.multi_indir (Failed)
357 - code_api|tool.drcachesim.scattergather-x86 (Failed)
407 - code_api|tool.drcacheoff.rseq (Failed)
408 - code_api|tool.drcacheoff.rseq-filter (Failed)
409 - code_api|tool.drcacheoff.rseq-dfilter (Failed)
424 - code_api|tool.drcacheoff.burst_threads (Failed)
445 - code_api|tool.record_filter (Failed)
465 - code_api|tool.drcpusim.cpuid-Prescott (Failed)
466 - code_api|tool.drcpusim.cpuid-Presler (Failed)
467 - code_api|tool.drcpusim.cpuid-Merom (Failed)
468 - code_api|tool.drcpusim.cpuid-Penryn (Failed)
490 - code_api|client.drx_time_scale-test (Failed)
492 - code_api|client.drx_timeout_scale-test (Failed)
510 - code_api|api.static_signal (Failed)
515 - code_api|api.static_maps_mixup_novars_FLAKY (Failed)
534 - code_api|api.rseq (Failed)
8 - tool.drcachesim.core_sharded (SEGFAULT)
30 - drmemtrace_proj (Failed)
31 - linux.xarch (Failed)
128 - code_api|linux.reset (Failed)
129 - code_api|linux.rseq (Failed)
130 - code_api|linux.rseq_table (Failed)
131 - code_api|linux.rseq_noarray (Failed)
145 - code_api|security-common.TestAllocWE (Failed)
146 - code_api|security-common.TestMemProtChg_FLAKY (Failed)
154 - code_api|client.alloc (Failed)
164 - code_api|client.exception (Failed)
192 - code_api|client.attach_test (Failed)
193 - code_api|client.detach_test (Failed)
194 - code_api|client.attach_blocking (Failed)
195 - code_api|client.memory_dump_test (Failed)
196 - code_api|client.attach_memory_dump_test (Failed)
199 - code_api|client.nudge_test (Failed)
247 - code_api|client.drx-scattergather (Failed)
248 - code_api|client.drx-scattergather-bbdup (Failed)
269 - code_api|sample.bbcount (Failed)
270 - code_api|sample.bbsize (Failed)
272 - code_api|sample.div (Failed)
274 - code_api|sample.memtrace_simple (Failed)
276 - code_api|sample.memval_simple (Failed)
277 - code_api|sample.memval_simple_scattergather (Failed)
278 - code_api|sample.instrace_simple (Failed)
279 - code_api|sample.opcode_count (Failed)
280 - code_api|sample.cbr (Failed)
281 - code_api|sample.countcalls (Failed)
282 - code_api|sample.inc2add (Failed)
283 - code_api|sample.memtrace_x86_binary (Failed)
284 - code_api|sample.memtrace_x86_text (Failed)
285 - code_api|sample.instrace_x86_binary (Failed)
286 - code_api|sample.instrace_x86_text (Failed)
288 - code_api|sample.ssljack (Failed)
289 - code_api|sample.modxfer (Failed)
290 - code_api|sample.modxfer_app2lib (Failed)
291 - code_api|sample.hot_bbcount (Failed)
292 - code_api|sample.instrcalls (Failed)
293 - code_api|sample.cbrtrace (Failed)
294 - code_api|sample.wrap (Failed)
295 - code_api|sample.signal (Failed)
296 - code_api|sample.syscall (Failed)
298 - code_api|sample.inline (Failed)
299 - code_api|sample.inscount (Failed)
300 - code_api|sample.inscount.cleancall (Failed)
301 - code_api|sample.inscount.prof-pcs.cleancall (Failed)
302 - code_api|sample.opcodes (Failed)
303 - code_api|sample.stl_test (Failed)
351 - code_api|tool.multi_indir (Failed)
357 - code_api|tool.drcachesim.scattergather-x86 (Failed)
407 - code_api|tool.drcacheoff.rseq (Failed)
408 - code_api|tool.drcacheoff.rseq-filter (Failed)
409 - code_api|tool.drcacheoff.rseq-dfilter (Failed)
424 - code_api|tool.drcacheoff.burst_threads (Failed)
445 - code_api|tool.record_filter (Failed)
465 - code_api|tool.drcpusim.cpuid-Prescott (Failed)
466 - code_api|tool.drcpusim.cpuid-Presler (Failed)
467 - code_api|tool.drcpusim.cpuid-Merom (Failed)
468 - code_api|tool.drcpusim.cpuid-Penryn (Failed)
490 - code_api|client.drx_time_scale-test (Failed)
492 - code_api|client.drx_timeout_scale-test (Failed)
510 - code_api|api.static_signal (Failed)
515 - code_api|api.static_maps_mixup_novars_FLAKY (Failed)
534 - code_api|api.rseq (Failed)
Derek Bruening
Nov 18, 2025, 12:44:34 AM (11 days ago) Nov 18
to Palmer Dabbelt, DynamoRIO Users
No, that is not expected. There is something unusual in your environments, at least compared to others that have been tested. What are the typical failure reasons? core_unit_tests segfault is very strange: what is the callstack there?
--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/6e7b2333-bd66-4b73-ba61-5e72a15090a5n%40googlegroups.com.
Palmer Dabbelt
Nov 18, 2025, 11:32:02 AM (11 days ago) Nov 18
to DynamoRIO Users
Here's a stacktrace from GDB -- I don't have debug packages debuginfod just hangs, so something's up there, but looks like it's in some TLS routines?
$ gdb bin64/core_unit_tests
GNU gdb (CentOS Stream) 16.3-2.el9
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin64/core_unit_tests...
Reading symbols from /home/palmerdabbelt/work/dynamorio/build/bin64/core_unit_tests.debug...
(gdb) r
Starting program: /home/palmerdabbelt/work/dynamorio/build/bin64/core_unit_tests
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.centos.org/>
Enable debuginfod for this session? (y or [n]) n
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
our_memcpy_time: size=1 time=0
libc_memcpy_time: size=1 time=0
our_memcpy_time: size=4 time=0
libc_memcpy_time: size=4 time=0
our_memcpy_time: size=128 time=5
libc_memcpy_time: size=128 time=1
our_memcpy_time: size=512 time=20
libc_memcpy_time: size=512 time=1
our_memcpy_time: size=8192 time=315
libc_memcpy_time: size=8192 time=12
our_memcpy_time: size=20480 time=733
libc_memcpy_time: size=20480 time=30
Program received signal SIGSEGV, Segmentation fault.
0x00000000711d4c34 in get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3124
3124 READ_TLS_SLOT_IMM(TLS_THREAD_ID_OFFSET, tid);
Missing rpms, try: dnf --enablerepo='*debug*' install glibc-debuginfo-2.34-232.el9.aarch64
(gdb) bt
#0 0x00000000711d4c34 in get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3124
#1 get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3119
#2 d_r_get_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3111
#3 0x000000007102914c in acquire_recursive_app_lock (mc=0x0, lock=0x712cf610 <global_alloc_lock>)
at /home/palmerdabbelt/work/dynamorio/core/utils.c:1009
#4 acquire_recursive_lock (lock=lock@entry=0x712cf610 <global_alloc_lock>) at /home/palmerdabbelt/work/dynamorio/core/utils.c:1021
#5 0x00000000710361f0 in common_global_heap_free (tu=0xfffd800200f8, p=0xfffd80038e80, size=<optimized out>)
at /home/palmerdabbelt/work/dynamorio/core/heap.c:3550
#6 0x0000000071065604 in our_memcpy_vs_libc () at /home/palmerdabbelt/work/dynamorio/core/io.c:1063
#7 unit_test_io () at /home/palmerdabbelt/work/dynamorio/core/io.c:1245
#8 0x0000000071000758 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
at /home/palmerdabbelt/work/dynamorio/core/unit_tests.c:77
GNU gdb (CentOS Stream) 16.3-2.el9
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin64/core_unit_tests...
Reading symbols from /home/palmerdabbelt/work/dynamorio/build/bin64/core_unit_tests.debug...
(gdb) r
Starting program: /home/palmerdabbelt/work/dynamorio/build/bin64/core_unit_tests
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.centos.org/>
Enable debuginfod for this session? (y or [n]) n
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
our_memcpy_time: size=1 time=0
libc_memcpy_time: size=1 time=0
our_memcpy_time: size=4 time=0
libc_memcpy_time: size=4 time=0
our_memcpy_time: size=128 time=5
libc_memcpy_time: size=128 time=1
our_memcpy_time: size=512 time=20
libc_memcpy_time: size=512 time=1
our_memcpy_time: size=8192 time=315
libc_memcpy_time: size=8192 time=12
our_memcpy_time: size=20480 time=733
libc_memcpy_time: size=20480 time=30
Program received signal SIGSEGV, Segmentation fault.
0x00000000711d4c34 in get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3124
3124 READ_TLS_SLOT_IMM(TLS_THREAD_ID_OFFSET, tid);
Missing rpms, try: dnf --enablerepo='*debug*' install glibc-debuginfo-2.34-232.el9.aarch64
(gdb) bt
#0 0x00000000711d4c34 in get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3124
#1 get_tls_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3119
#2 d_r_get_thread_id () at /home/palmerdabbelt/work/dynamorio/core/unix/os.c:3111
#3 0x000000007102914c in acquire_recursive_app_lock (mc=0x0, lock=0x712cf610 <global_alloc_lock>)
at /home/palmerdabbelt/work/dynamorio/core/utils.c:1009
#4 acquire_recursive_lock (lock=lock@entry=0x712cf610 <global_alloc_lock>) at /home/palmerdabbelt/work/dynamorio/core/utils.c:1021
#5 0x00000000710361f0 in common_global_heap_free (tu=0xfffd800200f8, p=0xfffd80038e80, size=<optimized out>)
at /home/palmerdabbelt/work/dynamorio/core/heap.c:3550
#6 0x0000000071065604 in our_memcpy_vs_libc () at /home/palmerdabbelt/work/dynamorio/core/io.c:1063
#7 unit_test_io () at /home/palmerdabbelt/work/dynamorio/core/io.c:1245
#8 0x0000000071000758 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
at /home/palmerdabbelt/work/dynamorio/core/unit_tests.c:77
Here's the logs, in case I'm missing something (`--output-on-failure` doesn't show anything extra):
$ ctest --stop-on-failure --verbose
UpdateCTestConfiguration from :/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Test project /home/palmerdabbelt/life/dynamorio/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: core_unit_tests
1: Test command: /home/palmerdabbelt/life/dynamorio/build/bin64/core_unit_tests
1: Working Directory: /home/palmerdabbelt/life/dynamorio/build/core
1: Test timeout computed to be: 1500
1: our_memcpy_time: size=1 time=0
1: libc_memcpy_time: size=1 time=0
1: our_memcpy_time: size=4 time=1
1: libc_memcpy_time: size=4 time=0
1: our_memcpy_time: size=128 time=4
1: libc_memcpy_time: size=128 time=0
1: our_memcpy_time: size=512 time=17
1: libc_memcpy_time: size=512 time=1
1: our_memcpy_time: size=8192 time=252
1: libc_memcpy_time: size=8192 time=12
1: our_memcpy_time: size=20480 time=629
1: libc_memcpy_time: size=20480 time=31
1/423 Test #1: core_unit_tests ..................................................***Exception: SegFault 1.14 sec
0% tests passed, 1 tests failed out of 1
Label Time Summary:
RUNS_ON_QEMU = 0.00 sec*proc (0 test)
RUN_IN_RELEASE = 0.00 sec*proc (0 test)
Total Test time (real) = 1.17 sec
UpdateCTestConfiguration from :/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Parse Config file:/home/palmerdabbelt/life/dynamorio/build/DartConfiguration.tcl
Test project /home/palmerdabbelt/life/dynamorio/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: core_unit_tests
1: Test command: /home/palmerdabbelt/life/dynamorio/build/bin64/core_unit_tests
1: Working Directory: /home/palmerdabbelt/life/dynamorio/build/core
1: Test timeout computed to be: 1500
1: our_memcpy_time: size=1 time=0
1: libc_memcpy_time: size=1 time=0
1: our_memcpy_time: size=4 time=1
1: libc_memcpy_time: size=4 time=0
1: our_memcpy_time: size=128 time=4
1: libc_memcpy_time: size=128 time=0
1: our_memcpy_time: size=512 time=17
1: libc_memcpy_time: size=512 time=1
1: our_memcpy_time: size=8192 time=252
1: libc_memcpy_time: size=8192 time=12
1: our_memcpy_time: size=20480 time=629
1: libc_memcpy_time: size=20480 time=31
1/423 Test #1: core_unit_tests ..................................................***Exception: SegFault 1.14 sec
0% tests passed, 1 tests failed out of 1
Label Time Summary:
RUNS_ON_QEMU = 0.00 sec*proc (0 test)
RUN_IN_RELEASE = 0.00 sec*proc (0 test)
Total Test time (real) = 1.17 sec
The following tests FAILED:
1 - core_unit_tests (SEGFAULT)
Errors while running CTest
Output from these tests are in: /home/palmerdabbelt/life/dynamorio/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
$ cat Testing/Temporary/LastTest.log
Start testing: Nov 18 08:21 PST
----------------------------------------------------------
1/423 Testing: core_unit_tests
1/423 Test: core_unit_tests
Command: "/home/palmerdabbelt/life/dynamorio/build/bin64/core_unit_tests"
Directory: /home/palmerdabbelt/life/dynamorio/build/core
"core_unit_tests" start time: Nov 18 08:21 PST
Output:
----------------------------------------------------------
our_memcpy_time: size=1 time=0
libc_memcpy_time: size=1 time=0
our_memcpy_time: size=4 time=1
libc_memcpy_time: size=4 time=0
our_memcpy_time: size=128 time=4
libc_memcpy_time: size=128 time=0
our_memcpy_time: size=512 time=17
libc_memcpy_time: size=512 time=1
our_memcpy_time: size=8192 time=252
libc_memcpy_time: size=8192 time=12
our_memcpy_time: size=20480 time=629
libc_memcpy_time: size=20480 time=31
<end of output>
Test time = 1.14 sec
----------------------------------------------------------
Test Failed.
"core_unit_tests" end time: Nov 18 08:21 PST
"core_unit_tests" time elapsed: 00:00:01
----------------------------------------------------------
End testing: Nov 18 08:21 PST
RUNS_ON_QEMU = 0.00 sec*proc
RUN_IN_RELEASE = 0.00 sec*proc
Output from these tests are in: /home/palmerdabbelt/life/dynamorio/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
$ cat Testing/Temporary/LastTest.log
Start testing: Nov 18 08:21 PST
----------------------------------------------------------
1/423 Testing: core_unit_tests
1/423 Test: core_unit_tests
Command: "/home/palmerdabbelt/life/dynamorio/build/bin64/core_unit_tests"
Directory: /home/palmerdabbelt/life/dynamorio/build/core
"core_unit_tests" start time: Nov 18 08:21 PST
Output:
----------------------------------------------------------
our_memcpy_time: size=1 time=0
libc_memcpy_time: size=1 time=0
our_memcpy_time: size=4 time=1
libc_memcpy_time: size=4 time=0
our_memcpy_time: size=128 time=4
libc_memcpy_time: size=128 time=0
our_memcpy_time: size=512 time=17
libc_memcpy_time: size=512 time=1
our_memcpy_time: size=8192 time=252
libc_memcpy_time: size=8192 time=12
our_memcpy_time: size=20480 time=629
libc_memcpy_time: size=20480 time=31
<end of output>
Test time = 1.14 sec
----------------------------------------------------------
Test Failed.
"core_unit_tests" end time: Nov 18 08:21 PST
"core_unit_tests" time elapsed: 00:00:01
----------------------------------------------------------
End testing: Nov 18 08:21 PST
RUNS_ON_QEMU = 0.00 sec*proc
RUN_IN_RELEASE = 0.00 sec*proc
Derek Bruening
Nov 18, 2025, 1:54:57 PM (11 days ago) Nov 18
to Palmer Dabbelt, DynamoRIO Users
Test logs are in Testing/Temporary/LastTest.log. With so many failures there likely is a single type that accounts for a majority of them which may point to the key issue in this environment.
Best to switch to debug build to get more easily understandable errors.
The aarch64 unit_tests crash on TLS access on a lock on free but not on alloc does not seem to make sense. If there is something unusual about the tpid* registers on this machine you'd think it would affect the earlier code, and every single test rather than a subset of them.
To view this discussion visit https://groups.google.com/d/msgid/dynamorio-users/2efa1c4d-37c1-41c8-b767-194aedf09d0fn%40googlegroups.com.
Palmer Dabbelt
Nov 18, 2025, 3:57:37 PM (11 days ago) Nov 18
to Derek Bruening, DynamoRIO Users
[replying inline, as there's multiple questions in here]
On Tue, Nov 18, 2025 at 10:54 AM Derek Bruening <brue...@google.com> wrote:
>
> Test logs are in Testing/Temporary/LastTest.log. With so many failures there likely is a single type that accounts for a majority of them which may point to the key issue in this environment.
I've attached the whole log. Just poking poking around a bit, I'm seeing:
Two of these libdrx build issues,
gmake[3]: *** No rule to make target
'/home/palmerdabbelt/work/dynamorio/build/ext/lib64/release/libdrx.so',
needed by 'bin/libbbbuf.s
o'. Stop.
I think this rseq assert is an issue, too (though there might also be
a regex mismatch):
/home/palmerdabbelt/work/dynamorio/suite/tests/linux/rseq.cpp:167:
bool register_rseq(): Assertion `res == 0 || (res == -1 && errno ==
ENOSYS)' failed.
tool.drcacheoff.gencode has the same SEGV in the TLS code.
A bunch of others report a SEGV too, but the debugger isn't catching
it. Not sure what's going on there, but the SEGVs are early like the
TLS ones are.
> Best to switch to debug build to get more easily understandable errors.
This new stuff is from a debug build, I'd thought I had one before but
I must have just lost it in a cmake rerun.
> The aarch64 unit_tests crash on TLS access on a lock on free but not on alloc does not seem to make sense. If there is something unusual about the tpid* registers on this machine you'd think it would affect the earlier code, and every single test rather than a subset of them.
I don't know if we're doing anything special in TLS land, I'll try and
ask around but not sure how successful that'll be. It feels like this
is a good one to look at first, as it's happening early, happening in
multiple places, and it looks like mangling up TLS could cause all
sorts of issues -- as far as I can tell DynamoRIO is trying to store
some of its own stuff in TLS.
That said, I'm not at all familiar with DynamoRIO so any pointers
would be appreciated...
On Tue, Nov 18, 2025 at 10:54 AM Derek Bruening <brue...@google.com> wrote:
>
> Test logs are in Testing/Temporary/LastTest.log. With so many failures there likely is a single type that accounts for a majority of them which may point to the key issue in this environment.
Two of these libdrx build issues,
gmake[3]: *** No rule to make target
'/home/palmerdabbelt/work/dynamorio/build/ext/lib64/release/libdrx.so',
needed by 'bin/libbbbuf.s
o'. Stop.
I think this rseq assert is an issue, too (though there might also be
a regex mismatch):
/home/palmerdabbelt/work/dynamorio/suite/tests/linux/rseq.cpp:167:
bool register_rseq(): Assertion `res == 0 || (res == -1 && errno ==
ENOSYS)' failed.
tool.drcacheoff.gencode has the same SEGV in the TLS code.
A bunch of others report a SEGV too, but the debugger isn't catching
it. Not sure what's going on there, but the SEGVs are early like the
TLS ones are.
> Best to switch to debug build to get more easily understandable errors.
I must have just lost it in a cmake rerun.
> The aarch64 unit_tests crash on TLS access on a lock on free but not on alloc does not seem to make sense. If there is something unusual about the tpid* registers on this machine you'd think it would affect the earlier code, and every single test rather than a subset of them.
ask around but not sure how successful that'll be. It feels like this
is a good one to look at first, as it's happening early, happening in
multiple places, and it looks like mangling up TLS could cause all
sorts of issues -- as far as I can tell DynamoRIO is trying to store
some of its own stuff in TLS.
That said, I'm not at all familiar with DynamoRIO so any pointers
would be appreciated...
Palmer Dabbelt
Nov 18, 2025, 5:48:38 PM (11 days ago) Nov 18
to DynamoRIO Users
Also just a quick update: these TLS failures seem to only manifest on our internal systems, or at least they're not showing up on an AWS machine. The AWS machine I have is much smaller (2 cores vs 72 cores) so it's possible something's just masking the bug, I'm trying to get a bigger AWS machine so I can run more stuff and see.
Suyash Mahar
Nov 25, 2025, 2:56:05 PM (4 days ago) Nov 25
to DynamoRIO Users
Responding on behalf of Palmer.
If I use an older commit (pre c4618dd9), I see about 35 tests fail on our Grace machine vs about 45 tests on an AWS graviton machine. With the latest master, I see about 64 tests fail on the Grace machine.
If I use an older commit (pre c4618dd9), I see about 35 tests fail on our Grace machine vs about 45 tests on an AWS graviton machine. With the latest master, I see about 64 tests fail on the Grace machine.
Should we expect all tests to pass?
As for the TLS issue, with git bisect, I was able to confirm the commit (tip from @assad) that seem to have broken TLS on our machine:
As for the TLS issue, with git bisect, I was able to confirm the commit (tip from @assad) that seem to have broken TLS on our machine:
```
c4618dd93e8334c437ddf4db4b33caebf2742fca is the first bad commit
commit c4618dd93e8334c437ddf4db4b33caebf2742fca (HEAD)
Author: Derek Bruening <brue...@google.com>
Date: Wed Oct 15 20:10:51 2025 -0400
i#7676: Refactor aarchxx private TLS to match the actual layout (#7679)
```
commit c4618dd93e8334c437ddf4db4b33caebf2742fca (HEAD)
Author: Derek Bruening <brue...@google.com>
Date: Wed Oct 15 20:10:51 2025 -0400
i#7676: Refactor aarchxx private TLS to match the actual layout (#7679)
```
-Suyash
Reply all
Reply to author
Forward
0 new messages