Crash in operator new - what could be happening here?

[Johannes S.]

Hi all,
Some years ago I found a bug in a glibc function while fuzzing my own code (which added much to the confusion while debugging), and I have a feeling I might have run into something similar again.
First off, I have just switched to afl++'s new LTO mode with Clang 11 and installed it on my Debian Buster system exactly as explained in the manual. Since it has been running, it has found one new crash, but with a very strange backtrace:

#0  0x00007fa5bbf10be8 in __gxx_personality_v0 (version=<optimized out>, actions=<optimized out>, exceptionClass=<optimized out>, unwind_exception=0x7fa5bbf1f4ac <(anonymous namespace)::heap+444>, context=0x7ffded19b870) at ./libcxxabi/src/cxa_personality.cpp:977
#1  0x00007fa5bb79922b in _Unwind_RaiseException (exc=0x7fa5bbf1f4ac <(anonymous namespace)::heap+444>) at ../../../src/libgcc/unwind.inc:118
#2  0x00007fa5bbf105a5 in __cxa_throw (thrown_object=0x7fa5bbf1f4cc <(anonymous namespace)::heap+476>, tinfo=0x4c2740 <typeinfo for std::bad_alloc>, dest=<optimized out>) at ./libcxxabi/src/cxa_exception.cpp:279
#3  0x00007fa5bbfa7d58 in operator new (size=<optimized out>) at ./libcxx/src/new.cpp:76
#4  0x0000000000453f26 in __gnu_cxx::new_allocator<OpenMPT::ModCommand>::allocate (this=<optimized out>, __n=27) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/ext/new_allocator.h:111
#5  std::allocator_traits<std::allocator<OpenMPT::ModCommand> >::allocate (__a=..., __n=27) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/alloc_traits.h:436
#6  std::_Vector_base<OpenMPT::ModCommand, std::allocator<OpenMPT::ModCommand> >::_M_allocate (this=<optimized out>, __n=27) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:296
#7  std::_Vector_base<OpenMPT::ModCommand, std::allocator<OpenMPT::ModCommand> >::_M_create_storage (this=<optimized out>, __n=27) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:311
#8  std::_Vector_base<OpenMPT::ModCommand, std::allocator<OpenMPT::ModCommand> >::_Vector_base (this=<optimized out>, __n=27, __a=...) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:260
#9  std::vector<OpenMPT::ModCommand, std::allocator<OpenMPT::ModCommand> >::vector (this=<optimized out>, __n=27, __value=..., __a=...) at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:429
#10 OpenMPT::CPattern::AllocatePattern (this=0x7fa5baeeda38, rows=<optimized out>) at soundlib/pattern.cpp:125
#11 OpenMPT::CPatternContainer::Insert (this=0x7fa5bb49d430, index=<optimized out>, rows=<optimized out>) at soundlib/patternContainer.cpp:86
#12 0x0000000000302c9b in OpenMPT::CSoundFile::ReadMO3 (this=<optimized out>, file=..., loadFlags=OpenMPT::CSoundFile::loadCompleteModule) at soundlib/Load_mo3.cpp:1049
...
Program terminated with signal SIGSEGV, Segmentation fault.

Frame #12 ends up allocating a lot of memory so we can be relativly sure that the memory is really being exhausted.
In frame #3 you can see `operator new`. new.cpp:76 is `throw std::bad_alloc()`, as expected (after malloc returned 0). My code would normally catch this exception, but instead... the standard library's exception handler segfaults?
Of course the first thought in this situation is "heap corruption!" - however, on first sight I don't really see anything in __gxx_personality_v0 that could rely on a piece of memory that I corrupted before this call to operator new.
I also manually reviewed my own code leading to this code path but so far I cannot find anything obvious that could lead to a heap corruption (and this code has been fuzzed for the last 4-5 years). Rebuilding with asan (but not running through afl) also didn't show anything suspicious.

Now, as I have recently switched to afl++ LTO - do you think this could contribute to the issue? Could it be a real bug in the standard library in low-memory situations? Do you have any other hints how I could proceed with debugging this crash?
I'm also somewhat confused by the apparent mixture of the GNU standard library and the LLVM standard library (libcxxabi). Could this be related?

I have been staring at this for the last few days but I still have no idea what's going on, so any help would be appreciated.

Cheers,
Johannes

[Marc]

Hello Johannes,

this is hard to say just looking at the backtrace.
I would recomend compiling the target with asan, this will make the root
cause analysis easier.

Note that I do not recommend to fuzz with an asan compiled target. the
slowdown is huge, and usually libdislocator finds the same bugs with
much less overhead.

LTO is unlikely to be the issue btw.

Regards,
Marc

> --
> You received this message because you are subscribed to the Google
> Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to afl-users+...@googlegroups.com
> <mailto:afl-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/afl-users/2153ee09-febb-45f8-8044-0c785ad91760n%40googlegroups.com
> <https://groups.google.com/d/msgid/afl-users/2153ee09-febb-45f8-8044-0c785ad91760n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Marc Heuse
www.mh-sec.de
PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573

[Johannes S.]

Hi all,

Okay, it's been a while since I have been looking at this issue but as I keep seeing these crashes in my fuzzing instances, I wanted to dig deeper. I was relatively sure that the code surrounding the crashes was sane, so I tried to set up a smaller test program and see if I can reproduce the issue that way. I think while doing so I actually found a different issue with libdislocator - or at least the strack trace is sufficiently different.

Step 1: The program

fuzz.cpp is attached to this message. All it does is taking the file passed to the program, and interprets the bytes inside the file as numbers of elements to allocate in a vector. Of course as a first step it's crucial to rule out that this program itself is sane (as far as fuzzing is concerned). The program allocates a couple of items on the heap and as soon as that fails with std::bad_alloc, it will bail out of the loop.

The program may of course get killed by the system if it runs out of memory, which would result in a SIGKILL (which did happen a couple of times while testing). However, what I was observing was a SIGABRT. From my understanding the code shouldn't cause any SIGABRTs, as a failure to allocate memory should rather result in std::bad_alloc being thrown in C++.

Step 2: Compilation

Using afl++3.10c compiled against clang-12 (from https://apt.llvm.org/ on a fresh Debian Stable VM), and the following command line:

afl-clang-lto++ ./fuzz.cpp -std=c++17 -lm -fPIC -fno-strict-aliasing -fvisibility=hidden -O3 -ffast-math -fno-omit-frame-pointer -o fuzz

Step 3: Fuzzing

I'm invoking afl-fuzz as follows:

export AFL_PRELOAD=/path/to/libdislocator.so
/path/to/afl-fuzz -p coe -f infile -t 5000 -i /path/to/initial -o /path/to/findings -S fuzzer01 /path/to/fuzz infile

I kept it running for a couple of minutes in a 64-bit Debian VM with 4GB of memory.

Eventually, I got some crashes with this backtrace (with example files that are attached to this message and can be used as initial input):

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50    ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffaeb65c535 in __GI_abort () at abort.c:79
#2  0x00007ffaebc3f6f3 in __dislocator_alloc (len=len@entry=2048) at libdislocator.so.c:240
#3  0x00007ffaebc3f912 in malloc (len=2048) at libdislocator.so.c:321
#4  0x00007ffaebb46918 in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libc++abi.so.1
#5  0x0000000000205eaa in __gnu_cxx::new_allocator<std::shared_ptr<char> >::allocate (
    this=0x7ffabb305fe8, __n=<optimized out>)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/ext/new_allocator.h:111
#6  std::allocator_traits<std::allocator<std::shared_ptr<char> > >::allocate (__a=...,
    __n=<optimized out>)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/alloc_traits.h:436
#7  std::_Vector_base<std::shared_ptr<char>, std::allocator<std::shared_ptr<char> > >::_M_allocate (
    this=0x7ffabb305fe8, __n=<optimized out>)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:296
#8  std::_Vector_base<std::shared_ptr<char>, std::allocator<std::shared_ptr<char> > >::_M_create_storage (this=0x7ffabb305fe8, __n=<optimized out>)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:311
#9  std::_Vector_base<std::shared_ptr<char>, std::allocator<std::shared_ptr<char> > >::_Vector_base (
    this=0x7ffabb305fe8, __n=<optimized out>, __a=...)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:260
#10 std::vector<std::shared_ptr<char>, std::allocator<std::shared_ptr<char> > >::vector (
    this=0x7ffabb305fe8, __n=<optimized out>, __a=...)
    at /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/stl_vector.h:416
#11 main (argc=<optimized out>, argv=<optimized out>) at ./fuzz.c:29

You'll notice that line 240 in libdislocator.so.c is tied to hard_fail being true, which is tied to an environment variable being set that I didn't know of until today - so naturally, AFL_LD_HARD_FAIL is *not* set in my environment. I double-checked and "echo $AFL_LD_HARD_FAIL" just returns an empty line.

So, what could be happening here? Could libdislocator ironically somehow be overwriting its own memory?

- Johannes

[Marc]

so that libdislocater is able to detect use-after-free vulnerabilities
it does not free the memory, just marking it as inaccessible instead.
Could it be that the fuzzed process is running out of memory?
if the crashes are not happening without libdislocator this is the
likely cause.

you can also compile with ASAN and use that instead of libdislocator if
the issue persists.

Regards,
Marc

[Johannes S.]

Hi Marc,

if the process ran out of memory, there should be a SIGKILL from my understanding, or malloc should return NULL if hard_fail is not set in libdislocator. What's strange here is that the process is instead killed with a SIGABRT due to hitting ` if (hard_fail) FATAL("mmap() failed on alloc (OOM?)");` even though hard_fail should clearly be 0 (I didn't set the environment variable). I haven't seen any crashes when using ASAN instead by the way, so it does seem to be an issue with libdislocator. I'd really like to be able to use both of them, because simulating low-memory environments has unearthed issues in the past, and from my understanding that's not possible due to the way ASAN works.

- Johannes