Inconsistent Signal for Out of Memory Error

[Hendra Gunadi]

Dear afl-users,

Not sure if this is a double post, but I couldn't find my previous post anywhere. During our trial fuzzing campaign on pdftotext tool from poppler (ver 0.64), we encountered some of the crashes that are signaled with 11 by AFL. Since generally SIGSEGV is interesting from the security perspective, we tried to reproduce it outside of AFL and analyze them further. To our surprise, we couldn't reproduce SIGSEGV outside of AFL even though we already set the memory limit using ulimit -Sv. We figured out that the problem may be linked to the way that memory is limited within afl-fuzz and outside, where afl-fuzz is limiting the memory using setrlimit on resource RLIMIT_AS:

RLIMIT_AS

This is the maximum size of the process’s virtual memory (address space). The limit is specified in bytes, and is rounded down to the system page size. This limit affects calls to brk(2), mmap(2), and mremap(2), which fail with the error ENOMEM upon exceeding this limit. In addition, automatic stack expansion fails (and generates a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

which introduce the possibility of SIGSEGV and ulimit does not seem to behave exactly the same. Not sure if this is an interesting problem in itself, but this obviously make signal 11 not necessarily tempting.

Regards,

Hendra

[Jakub Wilk]

* Hendra Gunadi <h3n...@gmail.com>, 2018-05-20, 16:00:

>Not sure if this is a double post, but I couldn't find my previous post
>anywhere.

FWIW, I didn't receive your previous post.

>Since generally SIGSEGV is interesting from the security perspective,
>we tried to reproduce it outside of AFL and analyze them further. To
>our surprise, we couldn't reproduce SIGSEGV outside of AFL even though
>we already set the memory limit using ulimit -Sv. We figured out that
>the problem may be linked to the way that memory is limited within
>afl-fuzz and outside, where afl-fuzz is limiting the memory using
>setrlimit on resource RLIMIT_AS:

But "ulimit -Sv" uses RLIMIT_AS, too, so what's you're seeing can't be
explained just by semantics of RLIMIT_AS. There must be something else
going on.

(Yes, exhausting the stack will cause segmentation fault. This is not
specific to RLIMIT_AS, but of course it's slightly more likely when
virtual memory limited.)

--
Jakub Wilk

[Hendra Gunadi]

Hi Jakub, thanks for your reply. Yeah indeed I should have checked the source code of

ulimit before making any judgement, my bad. However, the question of inconsistent signal

is still troubling me. I produced the core dump which of the run of the input file which signals

segfault, but I couldn't reverse trace it because an inspection to the instruction pointer

just gave me some rubbish.

[Hendra Gunadi]

Just throwing it up there, could this be related?

https://stackoverflow.com/questions/3980687/strange-segfaults-using-fprintf

However I'm not entirely convinced due to the difference in nature between fork and thread.

Or maybe it is similar in this particular case.

[Hendra Gunadi]

Ah I should have given more context, sorry. I did a backtrace on the coredump and it gave

me these lines :

#0  0x00007ffff6e4c18e in _IO_vfprintf_internal (s=0x7ffffffd7460, format=0x7ffff7b12a88 "Out of memory\n", ap=0x7ffffffd9b18) at vfprintf.c:1267
#1  0x00007ffff6e4eef1 in buffered_vfprintf (s=0x7ffff71c4540 <_IO_2_1_stderr_>, format=<optimised out>, args=<optimised out>) at vfprintf.c:2320
#2  0x00007ffff6e4c32d in _IO_vfprintf_internal (s=s@entry=0x7ffff71c4540 <_IO_2_1_stderr_>, format=format@entry=0x7ffff7b12a88 "Out of memory\n", ap=ap@entry=0x7ffffffd9b18) at vfprintf.c:1293
#3  0x00007ffff6f15bc9 in ___fprintf_chk (fp=0x7ffff71c4540 <_IO_2_1_stderr_>, flag=flag@entry=1, format=format@entry=0x7ffff7b12a88 "Out of memory\n") at fprintf_chk.c:35
#4  0x00007ffff76206d5 in fprintf (__fmt=0x7ffff7b12a88 "Out of memory\n", __stream=<optimised out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:98
#5  gmalloc (checkoverflow=false, size=<optimised out>) at /home/users/hengunadi/Downloads/poppler-0.64.0/goo/gmem.cc:164

Which looks like it is related to fprintf to stderr and also in a "spawned" environment.

[Hendra Gunadi]

Just in case anyone is interested to play around, find attached for the input file and a simple

program wrapper which exhibited this behavior. This program is taken from afl-tmin.c and I cut

down the unrelated things to the "minimal working case". The pdftotext is obtained from poppler-0.64.0

In my computer, to compile I will need the alloc-inl.h, config.h, debug.h, types.h, and  from afl

in the same directory, then

gcc -o test test.c

./test pdftotext crash_input

- Hendra

[Jakub Wilk]

* Hendra Gunadi <h3n...@gmail.com>, 2018-05-24, 22:12:

>Just in case anyone is interested to play around, find attached for the
>input file and a simple program wrapper which exhibited this behavior.
>This program is taken from afl-tmin.c and I cut down the unrelated
>things to the "minimal working case". The pdftotext is obtained from
>poppler-0.64.0 In my computer, to compile I will need the alloc-inl.h,
>config.h, debug.h, types.h, and from afl in the same directory, then
>
>gcc -o test test.c
>./test pdftotext crash_input

I can't reproduce it here:

$ ./test pdftotext crash_input
Syntax Error (7031): Dictionary key must be a name object
Syntax Error (7044): Dictionary key must be a name object
Syntax Error (7046): Dictionary key must be a name object
[snip - more errors]
Syntax Error (2487): Dictionary key must be a name object
Syntax Error (2504): Dictionary key must be a name object
Syntax Error (2638): Unknown filter ''
Out of memory
exit status 256 16642

This was tested with:

* Debian 9.4 (stretch) on amd64;
* Poppler 0.64.0 built from source (but against distro libs) with
afl-gcc;
* AFL 2.36b from distro package.

--
Jakub Wilk

[Hendra Gunadi]

Oh no, this is annoying when a problem can't be reproduced on antoher computer.

I'm not that familiar with various Linux distro either to understand how it can be

different. However, my computer runs Linux Ubuntu 16.04.4 LTS (Xenial Xerus) on

model name      : Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz

cache size      : 8192 KB

as a side note, in case I haven't made it clear, this problem is not specific to

AFL but running it under "spawned" environment in general, i.e., running normal

build of pdftotext (without AFL-instrumented) without the "test" context gives

normal error message while running it under "test" context returns SIGSEGV.

Just throwing random ideas here, does the depth of callstack has anything to do

with it? Because the backtrace on the core dump gives me depth of 1000+ calls.

[Hendra Gunadi]

Sorry about my previous comment on general problem, it seems there is even a difference

between in behaviour of instrumented and un-instrumented binary, but that is separate issue.

Btw here is what I did in my computer and you can see that the error messages are different.