Cannot run any DynamoRIO tool without crashing

Derek

unread,

Mar 8, 2022, 8:18:00 AM3/8/22

to DynamoRIO Users

Hello,

I have a strange issue while trying to get DynamoRIO working.

I just cannot use any tool without crashing on my distro. Trivial examples like `./drrun -t drcov -- grep` result in:

```

```

I tried both the official release version and built DynamoRIO also on my own and both did not work. Also, other DBI tools like QBDI do also not work for some reason. They crash with a SEGV too.

In a quick test in a virtual machine running Linux Mint showed that DynamoRIO and also QBDI work flawlessly. So my intuition is that something with my distro prevents DBI tools from working. Does anyone have an idea what that could be?

Thank you!

Operating System: Manjaro Linux
KDE Plasma Version: 5.24.2
KDE Frameworks Version: 5.91.0
Qt Version: 5.15.2
Kernel Version: 5.16.11-2-MANJARO (64-bit)
Graphics Platform: X11
Processors: 8 × Intel® Core™ i5-8250U CPU @ 1.60GHz
Memory: 7.6 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics 620

assad.hashm...@gmail.com

unread,

Mar 8, 2022, 1:30:47 PM3/8/22

to DynamoRIO Users

Can you run with logging and debug to get some clues to find out what's going wrong?
Use the -debug and -loglevel options as described at https://dynamorio.org/page_logging.html
Look through the logfiles for warnings, assert messages and any other events which hint at a cause of the crash.

Derek

unread,

Mar 8, 2022, 2:54:27 PM3/8/22

to DynamoRIO Users

Thank you for your answer.

I ran the `./drrun -debug -loglevel 2 -t drcov -- grep` command had a look into the log file. Though I did not find much hints about the cause of the crash.

I attached the logs to this post. The SEGV is at around line 100784 in the thread log.

One observation I made that might be very important: I just downgraded my installed glibc version (glibc-2.35-2) to an older version (glibc-2.33-5). With the downgraded version, the crash does not occur!

Altough I dont want to stay at the downgraded version because a lot of programs broke but this might be a very interesting hint anyways.

grep.0.32805.html

log.0.32805.html

Derek

unread,

Mar 11, 2022, 5:48:24 AM3/11/22

to DynamoRIO Users

Just tested it in a fresh Arch Linux environment and I can reproduce the issue there too.

There was a change in the Arch Linux glibc packgage where the real libc library was called "libc-2.xx.so" in glibc-2.33-5, but with glibc-2.35-2 it's called "libc.so.6".

Could that be a possible cause of the bug? At least that was the problem in QBDI and was now fixed.

Also, by looking in the logs, the crash always happens at "shared_delete_lock(mutex)@/home/runner/work/dynamorio/dynamorio/core/vmareas.c:414". This could also be a hint.

Derek Bruening

unread,

Mar 21, 2022, 11:33:29 PM3/21/22

to Derek, DynamoRIO Users

Does it crash without any client at all?

Generally with a crash outside the code cache I would start with a symbolized callstack. Run under gdb or attach at the crash point, load the symbols for DR and any private libraries (if it only repros with a client), and get a callstack.

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/81b9c8ea-4d1c-42da-b882-67b07750ae8dn%40googlegroups.com.

Derek

unread,

Mar 22, 2022, 4:44:08 AM3/22/22

to DynamoRIO Users

Hello Derek,

it also crashes without any client. However, simple applications like ls do work but something like grep results in a crash.

I have attached the gdb backtrace at the point of the crash.

Thanks!

'bt' and 'bt full'.txt

Derek Bruening

unread,

Mar 22, 2022, 3:04:01 PM3/22/22

to Derek, DynamoRIO Users

That is not the callstack of the crash. Please see https://dynamorio.org/page_debugging.html#autotoc_md140 on expected signals at safe_read*; just continue past.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/1ebde65a-7f85-4316-886c-23765d1d1111n%40googlegroups.com.

Derek

unread,

Mar 22, 2022, 4:00:37 PM3/22/22

to DynamoRIO Users

Now it should be correct.

'bt' and 'bt full'.txt

Derek Bruening

unread,

Mar 22, 2022, 4:49:12 PM3/22/22

to Derek, DynamoRIO Users

So it involves rseq? Are these apps using rseq, via some shared library on your system? What do the logs say about rseq? Unfortunately there are rseq usages that simply cannot be supported: but as many as possible have checks. Debug build had no complaints? I would search the logs for rseq. See https://dynamorio.org/page_rseq.html for rseq info.

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/497ddf5e-1543-47eb-b9fa-91faefbdaa80n%40googlegroups.com.

Derek

unread,

Mar 23, 2022, 11:29:37 AM3/23/22

to DynamoRIO Users

- Are these apps using rseq, via some shared library on your system?

Maybe. This article https://www.phoronix.com/scan.php?page=news_item&px=Glibc-RSEQ-Returns-2021 states that with glibc 2.35 RSEQ support was added to glibc. This may explain why the crash does not occur when glibc 2.33 is installed.

- Debug build had no complaints?

No, everything builds fine, without related warnings.

- What do the logs say about rseq?

I think the only occurence of rseq in my posted logs is around line 27591, tough I'm not sure if it is related to the crash.

Derek Bruening

unread,

Mar 24, 2022, 2:53:05 PM3/24/22

to Derek, DynamoRIO Users

Could you file a bug in our tracker on this? We have put a lot of effort into rseq support and routinely run large internal applications using rseq; this may be some usage that does not match the conventions set forth by the rseq authors (unfortunately this rseq kernel feature makes it infeasible to handle every possible app behavior...). (As for other DBI systems failing: last I checked none of them had rseq handling at all.)

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/a7e67555-8699-435b-b640-988e2eae1423n%40googlegroups.com.

Reply all

Reply to author

Forward