honggfuzz crashing (segfault) on libunwind. Multiple times, same place

Tomasz Bukowski

unread,

Jun 2, 2021, 6:20:34 AM6/2/21

to honggfuzz

While running honggfuzz, It crashes in libunwind.

Tested both on stock ubuntu and on newest, freshly compiled lib,

Is there any well-tested way to debug that issue ?

Classic gdb approach failed.

Setting kernel option to keep cores also fails (cause is saves core for every crash of fuzzed app ... )

I was not able to stable reproduce that - but it is being regularly triggered.

Artifacts below.

[1222618.313893] honggfuzz[3207897]: segfault at 7f2d7d6ad000 ip 00007f2d7d1ffcbb sp 00007f2d720c9390 error 4 in libunwind-x86_64.so.8.0.1[7f2d7d1fe000+10000]

[1222740.122044] honggfuzz[3214987]: segfault at 7ffff7e5b000 ip 00007ffff79b1cbb sp 00007fffecc7d390 error 4 in libunwind-x86_64.so.8.0.1[7ffff79b0000+10000]

[1223221.998324] honggfuzz[3217848]: segfault at 7ffff7e5b000 ip 00007ffff79b1cbb sp 00007fffece7e390 error 4 in libunwind-x86_64.so.8.0.1[7ffff79b0000+10000]

[1223459.537005] honggfuzz[3217920]: segfault at 7ffff7e60000 ip 00007ffff79b1cbb sp 00007fffece7e390 error 4 in libunwind-x86_64.so.8.0.1[7ffff79b0000+10000]

[1223221.998329] Code: 3b f8 ff ff 89 df 49 89 07 e8 61 f8 ff ff 49 8b 3f 4c 8b 54 24 08 48 83 ff ff 0f 84 63 01 00 00 49 8b 77 08 48 83 fe 06 76 12 <81> 3f 7f 45 4c

46 75 0a 80 7f 04 02 0f 84 0d 01 00 00 4c 89 54 24

Parsed version:

>> BEGIN

PROC : honggfuzz

PID : 3547495

MEM @ : 0x00007ffff7e4b000 [......]

EIP : 0x00007ffff79b1cbb [........]

ESP : 0x00007fffecc7db90 [........]

ECODE : 4 (READ from unmapped)

CRASH-LIB: libunwind-x86_64.so.8.0.1

CRASH-MEM: 0x00007ffff79b0000 ... 0x00007ffff79c0000 (size 0x0000000000010000)

EIP-BASE : 0x0000000000001cbb == EIP - BASE

>> BINARY

FILE PATH : /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8.0.1

-- disasm -- 8< --

1cb9: 76 12 jbe 1ccd <_Ux86_64_get_elf_image@@Base+0x66d>

1cbb: 81 3f 7f 45 4c 46 cmp DWORD PTR [rdi],0x464c457f

1cc1: 75 0a jne 1ccd <_Ux86_64_get_elf_image@@Base+0x66d>

-- disasm -- 8< --

<< BINARY

<< SEGFAULT

Robert Święcki

unread,

Jun 4, 2021, 3:48:55 AM6/4/21

to Tomasz Bukowski, honggfuzz

Hi,

If it crashes in libunwind, it crashes the whole honggfuzz process, right?

In general gdb debugging should work, because gdb doesn't attached to new spawned processes, and you can enforce it with

set follow-fork-mode parent

which is probably default anyway. Or, is it the problem that it's hard to debug it with gdb anyway?

As for cores, if you set core limit to a non-zero value, and then use

--rlimit_core 0

then cores for honggfuzz itself should be saved, but not for fuzzed processes.

HTH

--
You received this message because you are subscribed to the Google Groups "honggfuzz" group.
To unsubscribe from this group and stop receiving emails from it, send an email to honggfuzz+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/honggfuzz/61363630-6561-4eee-8532-fb5fc0e8f677n%40googlegroups.com.

--

Robert Święcki

Tomasz Bukowski

unread,

Jun 6, 2021, 1:21:10 PM6/6/21

to Robert Święcki, honggfuzz

Yes, it crashes the whole honggfuzz process.

After RTFM how to use gdb it actually worked :-)

#0 0x00007ffff79b1cbb in _Ux86_64_get_elf_image () from /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
#1 0x00007ffff7bd08fe in ?? () from /usr/lib/x86_64-linux-gnu/libunwind-ptrace.so.0
#2 0x00007ffff7bd0a09 in _UPT_find_proc_info () from /usr/lib/x86_64-linux-gnu/libunwind-ptrace.so.0
#3 0x00007ffff79b8e7b in ?? () from /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
#4 0x00007ffff79bae96 in ?? () from /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
#5 0x00007ffff79bc409 in ?? () from /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
#6 0x00007ffff79b46e1 in _Ux86_64_step () from /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
#7 0x0000555555570ec6 in arch_unwindStack ()
#8 0x000055555556f00e in arch_traceSaveData ()
#9 0x000055555556fd65 in arch_traceAnalyze ()
#10 0x000055555556c420 in arch_reapChild ()
#11 0x000055555556ad30 in subproc_Run ()
#12 0x000055555555dd73 in fuzz_threadNew ()
#13 0x00007ffff6e9f6db in start_thread (arg=0x7fffece9b700) at pthread_create.c:463
#14 0x00007ffff6bc871f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

pwndbg> info registers
rax 0x0 0
rbx 0x15 21
rcx 0x7ffff6ea9537 140737335956791
rdx 0x0 0
rsi 0x270d4 1599d56
rdi 0x7ffff7e60000 140737352433664
rbp 0x7ffff7ff4000 0x7ffff7ff4000
rsp 0x7fffece7e390 0x7fffece7e390
r8 0x15 21
r9 0x0 0
r10 0x7ffff7ff4edf 140737354092255
r11 0x0 0
r12 0x7fffece7e4c0 140737168008384
r13 0x7fffece7e4c8 140737168008392
r14 0x1000 4096
r15 0x7fffe003e638 140736951739960
rip 0x7ffff79b1cbb 0x7ffff79b1cbb <_Ux86_64_get_elf_image+1627>
eflags 0x10212 [ AF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0

context:

0x00007ffff79b1ca7 <+1607>: cmp rdi,0xffffffffffffffff
0x00007ffff79b1cab <+1611>: je 0x7ffff79b1e14 <_Ux86_64_get_elf_image+1972>
0x00007ffff79b1cb1 <+1617>: mov rsi,QWORD PTR [r15+0x8]
0x00007ffff79b1cb5 <+1621>: cmp rsi,0x6
0x00007ffff79b1cb9 <+1625>: jbe 0x7ffff79b1ccd <_Ux86_64_get_elf_image+1645>
=> 0x00007ffff79b1cbb <+1627>: cmp DWORD PTR [rdi],0x464c457f
0x00007ffff79b1cc1 <+1633>: jne 0x7ffff79b1ccd <_Ux86_64_get_elf_image+1645>
0x00007ffff79b1cc3 <+1635>: cmp BYTE PTR [rdi+0x4],0x2
0x00007ffff79b1cc7 <+1639>: je 0x7ffff79b1dda <_Ux86_64_get_elf_image+1914>
0x00007ffff79b1ccd <+1645>: mov QWORD PTR [rsp+0x8],r10
0x00007ffff79b1cd2 <+1650>: or ebx,0xffffffff
0x00007ffff79b1cd5 <+1653>: call 0x7ffff79b15c0 <munmap@plt>

....

0x555555785000 0x5555559ac000 rw-p 227000 0 [heap]

....

0x7ffff7dd1000 0x7ffff7dd2000 r--p 1000 2000 /usr/lib/x86_64-linux-gnu/libunwind-ptrace.so.0.0.0
0x7ffff7dd2000 0x7ffff7dd3000 rw-p 1000 3000 /usr/lib/x86_64-linux-gnu/libunwind-ptrace.so.0.0.0
0x7ffff7dd3000 0x7ffff7dfc000 r-xp 29000 0 /lib/x86_64-linux-gnu/ld-2.27.so
0x7ffff7e88000 0x7ffff7f48000 rw-p c0000 0 /memfd:hf-0-input (deleted)
0x7ffff7f48000 0x7ffff7fd9000 rw-p 91000 0 /memfd:hf-cmpfeedback (deleted)
0x7ffff7fd9000 0x7ffff7fe4000 rw-p b000 0

RDI looks legit, but it is not mapped. It is far away from the heap ;/

Going deeper into libunwind, the problem is in:

static inline int elf_map_image (struct elf_image *ei, const char *path)

SEE: https://github.com/libunwind/libunwind/blob/ec171c9ba7ea3abb2a1383cee2988a7abd483a1f/src/elfxx.h#L73

(actual decompiled assembly code, similar to source):

ret_elf_image->size = file_stats.st_size;
ptr_mem = mmap(0x0,file_stats.st_size,1,2,file_handle,0);
ret_elf_image->image = ptr_mem;
close(file_handle);
dataPtr = ret_elf_image->image;
if (dataPtr == 0xffffffffffffffff) goto LAB_00102604;
if (((ret_elf_image->size < 0x7) || (*dataPtr != 0x464c457f)) ||
((dataPtr[4] != 2 || (dataPtr[6] != 1)))) {
/* ^---- SEGFAULT HERE */
uVar18 = 0xffffffff;
munmap(dataPtr,ret_elf_image->size);
}

Looks like memory returned by mmap is not available somehow.

That is weird.

- close(fd) should not invalidate mmap'ed memory.

- The returned pointer looks legit (not null, not -1, not anything weird)

- EBX contains valid (at moment of mmap) file handle

For now got no idea if and how to chase the root cause ;/

--

Tomasz Bukowski,
keidii(at)gmail(dot)com

Robert Święcki

unread,

Jun 7, 2021, 5:37:23 AM6/7/21

to Tomasz Bukowski, honggfuzz

Yeah, this code look ok'ish. I mean the code should check if file_stats.st_size is bigger than some minimal values (say, size of ELF header), but it'd end up in SIGBUS and not SIGSEGV if it was the case.

I can only encourage further debugging, if this problem seems interesting personally to you.

That said, maybe it's about locking? The libunwind (as opposed to libbfd) seems to be MT-Safe. but maybe it's not. Could you quickly add the following code to the beginning of arch_unwindStack and check again?

static pthread_mutex_t arch_unwind_mutex = PTHREAD_MUTEX_INITIALIZER;

MX_SCOPED_LOCK(&arch_unwind_mutex);

--

Robert Święcki

Tomasz Bukowski

unread,

Jun 7, 2021, 6:48:28 AM6/7/21

to Robert Święcki, honggfuzz

It is clearly a libunwind problem - but it affects honggfuzz ;)

I will try to debug this, but it sounds like a lot of resources :)

- as mentioned it is completely not checking file size against minimal ELF size (that is already a finding :-)

I will try to submit that to libunwind repo ;)

- also it smells like TOCTOU - file being trimmed/deleted/etc - especially that vulnerable open /proc/X/maps , and after that trying to open corresponding binary