drrun crashes with any client due to SIGFPE

[couyoh]

Hello,
I can't run drrun with any client due to SIGFPE even with a sample client.
Without any client, it works.

Tested on commit 5e13602, Arch Linux x86_64 (glibc package version is 2.35-3) and Ubuntu 21.10 x86_64 (glibc-bin package version is 2.34-0ubuntu3.2).
Even if I use pre-compiled binary (DynamoRIO-Linux-9.0.1.tar.gz), no works.

It looks the problem is related to <https://github.com/DynamoRIO/dynamorio/pull/5134>.
When I use commit 26b5fb (cherry-picked from commit 1dec190 for fixing build error), it's worked but when I use commit f3d907d (cherry-picked from commit 1dec190) it's crashed.

The below output is commit 5e13602 on Ubuntu 21.10.

```
$ uname -a
Linux ubuntu-VirtualBox 5.13.0-35-generic #40-Ubuntu SMP Mon Mar 7 08:03:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.34-0ubuntu3.2) stable release version 2.34.
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 10.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.

$ ./bin64/drrun -version
drrun version 9.0.19077 -- build 0

$ ./bin64/drrun -verbose -c api/bin/libbbcount.so -- ls
INFO: client 0 path: /home/ubuntu/Desktop/git/build/api/bin/libbbcount.so
INFO: targeting application: "/usr/bin/ls"
INFO: app cmdline:  "ls"
INFO: configuration directory is "/home/ubuntu/.dynamorio"
INFO: will exec /usr/bin/ls
INFO: registering client with id=0 path=|/home/ubuntu/Desktop/git/build/api/bin/libbbcount.so| ops=||
Floating point exception (core dumped)

$ gdb -q --args ./bin64/drrun -debug -c api/samples/bin/libbbcount.so -- ls
Reading symbols from ./bin64/drrun...
Reading symbols from /home/ubuntu/Desktop/git/build/bin64/drrun.debug...
(gdb) r
Starting program: /home/ubuntu/Desktop/git/build/bin64/drrun -debug -c api/samples/bin/libbbcount.so -- ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
process 48064 is executing new program: /home/ubuntu/Desktop/git/build/lib64/debug/libdynamorio.so
<Starting application /usr/bin/ls (48064)>

Program received signal SIGILL, Illegal instruction.
syscall_ready () at ../core/drlibc/drlibc_x86.asm:184
184                pop      REG_XBX
(gdb) b core/unix/loader.c:693
Breakpoint 1 at 0x7ffff7e7a507: file ../core/unix/loader.c, line 693.
(gdb) c
Continuing.
<Initial options = -no_dynamic_options -client_lib '/home/ubuntu/Desktop/git/build/api/samples/bin/libbbcount.so;0;' -client_lib64 '/home/ubuntu/Desktop/git/build/api/samples/bin/libbbcount.so;0;' -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >

Breakpoint 1, privload_os_finalize (privmod=0x7ffdb3ba77b8) at ../core/unix/loader.c:693
693                (*libc_early_init)(true);
(gdb) c
Continuing.

Program received signal SIGFPE, Arithmetic exception.
0x00007ffff783065e in ?? ()
(gdb) x/3s gdb_priv_cmds
0x7ffff7ff1c20 <gdb_priv_cmds>:        "add-symbol-file '/home/ubuntu/Desktop/git/build/api/samples/bin/libbbcount.so' 0x00007fffb3b1d2a0\nadd-symbol-file '/home/ubuntu/Desktop/git/build/lib64/debug/libdynamorio.so' 0x00007ffff7bb0000\nadd-sy"...
0x7ffff7ff1ce8 <gdb_priv_cmds+200>:        "mbol-file '/home/ubuntu/Desktop/git/build/ext/lib64/debug/libdrx.so' 0x00007fffb3b644b0\nadd-symbol-file '/home/ubuntu/Desktop/git/build/ext/lib64/debug/libdrreg.so' 0x00007fffb3b78df0\nadd-symbol-file "...
0x7ffff7ff1db0 <gdb_priv_cmds+400>:        "'/home/ubuntu/Desktop/git/build/ext/lib64/debug/libdrmgr.so' 0x00007fffb3b8bcb0\nadd-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007ffff76e46c0\nadd-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0"...
(gdb) add-symbol-file '/lib/x86_64-linux-gnu/libc.so.6' 0x00007ffff76e46c0
add symbol table from file "/lib/x86_64-linux-gnu/libc.so.6" at
        .text_addr = 0x7ffff76e46c0
(y or n) y
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
Reading symbols from /usr/lib/debug/.build-id/f0/fc29165cbe6088c0e1adf03b0048fbecbc003a.debug...
warning: td_ta_new failed: generic error
warning: File "/usr/lib/x86_64-linux-gnu/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/lib/x86_64-linux-gnu/libthread_db.so.1
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
(gdb) bt
#0  0x00007ffff783065e in __nptl_tls_static_size_for_stack () at ../nptl/nptl-stack.h:59
#1  __pthread_early_init () at ../sysdeps/nptl/pthread_early_init.h:46
#2  __libc_early_init (initial=<optimized out>) at libc_early_init.c:44
#3  0x00007ffff7e7a512 in privload_os_finalize (privmod=0x7ffdb3ba77b8) at ../core/unix/loader.c:693
#4  0x00007ffff7d55dd3 in privload_load_process (privmod=0x7ffdb3ba77b8) at ../core/loader_shared.c:818
#5  0x00007ffff7d555ad in privload_load (filename=0x7fffffffbd30 "/lib/x86_64-linux-gnu/libc.so.6", dependent=0x7ffdb3b71fb8, client=false) at ../core/loader_shared.c:683
#6  0x00007ffff7e7a5ad in privload_locate_and_load (impname=0x7fffb3b1c927 "libc.so.6", dependent=0x7ffdb3b71fb8, reachable=false) at ../core/unix/loader.c:710
#7  0x00007ffff7e79dae in privload_process_imports (mod=0x7ffdb3b71fb8) at ../core/unix/loader.c:566
#8  0x00007ffff7d55d32 in privload_load_process (privmod=0x7ffdb3b71fb8) at ../core/loader_shared.c:811
#9  0x00007ffff7d53d9a in privload_process_early_mods () at ../core/loader_shared.c:139
#10 0x00007ffff7d53f8c in loader_init_epilogue (dcontext=0x7ffdb3ba0080) at ../core/loader_shared.c:203
#11 0x00007ffff7bc224a in dynamorio_app_init_part_two_finalize () at ../core/dynamo.c:670
#12 0x00007ffff7e7dd20 in privload_early_inject (sp=0x7fffffffdf10, old_libdr_base=0x0, old_libdr_size=140737488346448) at ../core/unix/loader.c:2154
#13 0x00007ffff7e2619d in reloaded_xfer () at ../core/arch/x86/x86.asm:1179
#14 0x0000000000000001 in ?? ()
#15 0x00007fffffffe28a in ?? ()
#16 0x0000000000000000 in ?? ()
```

Thank you.

[Derek Bruening]

What instruction is raising the SIGFPE (i.e., please disassemble the instructions at that point)?

Wondering why no one else has hit this: others are using glibc 2.35 (xref https://github.com/DynamoRIO/dynamorio/issues/5431) and not hitting this (in that issue, with -disable_rseq, everything works fine).

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/7574aca1-47bc-4ebc-834a-f9f9e52a9bf9n%40googlegroups.com.

[couyoh]

Thank you for your reply.

I forgot to say that I tested on fresh (but all packages are up to date) Ubuntu 21.10.

>What instruction is raising the SIGFPE (i.e., please disassemble the instructions at that point)?

```
(gdb) x/i $pc
=> 0x7ffff7aae65e <__libc_early_init+142>:        div    rsi
(gdb) p $rsi
$2 = 0
```

Probably dl_tls_static_size and dl_tls_static_align (on glibc/nptl/nptl-stack.h:58 of 2.34-0ubuntu3.2's glibc source) are zero.

```
(gdb) x/10i __libc_early_init+98
   0x7ffff7aae632 <__libc_early_init+98>:        mov    rax,QWORD PTR [rip+0xa488f]        # 0x7ffff7b52ec8
   0x7ffff7aae639 <__libc_early_init+105>:        xor    edx,edx
   0x7ffff7aae63b <__libc_early_init+107>:        mov    rsi,QWORD PTR [rax+0x2b0]
=> 0x7ffff7aae642 <__libc_early_init+114>:        mov    rbx,QWORD PTR [rax+0x2a8]
   0x7ffff7aae649 <__libc_early_init+121>:        mov    rcx,QWORD PTR [rax+0x18]
   0x7ffff7aae64d <__libc_early_init+125>:        add    rbx,rsi
   0x7ffff7aae650 <__libc_early_init+128>:        mov    rax,rbx
   0x7ffff7aae653 <__libc_early_init+131>:        mov    QWORD PTR [rip+0xab4d6],rcx        # 0x7ffff7b59b30
   0x7ffff7aae65a <__libc_early_init+138>:        sub    rax,0x1
   0x7ffff7aae65e <__libc_early_init+142>:        div    rsi
(gdb) x/g $rax+0x2b0
0x7ffff7d7cef0:        0x0000000000000000
(gdb) x/g $rax+0x2a8
0x7ffff7d7cee8:        0x0000000000000000
```

>Wondering why no one else has hit this: others are using glibc 2.35 (xref https://github.com/DynamoRIO/dynamorio/issues/5431) and not hitting this (in that issue, with -disable_rseq, everything works fine).

The environment of the xref looks Manjaro Linux, which is Arch-based, and I've already tested on Arch.

Just in case, I downloaded Manjaro KDE Linux 21.2.5, and then I tested on it with re-compiled binary (DynamoRIO-Linux-9.0.1.tar.gz), but drrun gives SIGFPE.
Manjaro's glibc is version 2.35.

When I tested on fresh (but all packages are up to date) Ubuntu 21.04, it works. Glibc is 2.33-0ubuntu5.
It looks not to work from 2.34.

2022年3月27日日曜日 13:00:54 UTC+9 Derek Bruening:

[Derek Bruening]

Hmm, so without the __libc_early_init call, glibc 2.32 crashes; but with it 2.34 crashes?  Can't win.  There must be some other magic hardcoded initialization done specially for libc in 2.34 by ld.so??

Maybe the best course is if you could file an issue to track this and further discussion will go there?

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/c2bd6549-c5f0-4b66-8f8a-30431a32a32an%40googlegroups.com.

[Derek Bruening]

Someone else hit this and filed https://github.com/DynamoRIO/dynamorio/issues/5437 so that will serve as the tracking issue.