afl++ says the program crashes, but it doesn't when run from bash

[Pierre Abbat]

I'm trying to fuzz quadlods's (https://github.com/phma/quadlods) interactive
mode. I've previously done this using version 2.52b, but since I've added
multithreading, afl-gcc 2.52b doesn't compile it. With 2.60d (these are on two
different computers), it tells me the following:

$ afl-fuzz -m 200 -i in -o out -M fuzz0 ./quadlods interact
afl-fuzz++2.60d based on afl by Michal Zalewski and a big online community
[+] afl++ is maintained by Marc "van Hauser" Heuse, Heiko "hexcoder" Eissfeldt
and Andrea Fioraldi
[+] afl++ is open source, get it at https://github.com/vanhauser-thc/
AFLplusplus
[+] Power schedules from github.com/mboehme/aflfast
[+] Python Mutator and llvm_mode whitelisting from github.com/choller/afl
[+] afl-tmin fork server patch from github.com/nccgroup/TriforceAFL
[+] MOpt Mutator from github.com/puppet-meteor/MOpt-AFL
[*] Getting to work...
[+] Using exploration-based constant power schedule (EXPLORE)
[+] You have 12 CPU cores and 8 runnable tasks (utilization: 67%).
[+] Try parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning 'in'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,time:0,orig:input'...
[*] Spinning up the fork server...
[+] All right - fork server is up.

[-] Oops, the program crashed with one of the test cases provided. There are
several possible explanations:

- The test case causes known crashes under normal working conditions. If
so, please remove it. The fuzzer should be seeded with interesting
inputs - but not ones that cause an outright crash.

- The current memory limit (200 MB) is too low for this program, causing
it to die due to OOM when parsing valid files. To fix this, try
bumping it up with the -m setting in the command line. If in doubt,
try something along the lines of:

( ulimit -Sd $[199 << 10]; /path/to/binary [...] <testcase )

Tip: you can use http://jwilk.net/software/recidivm to quickly
estimate the required amount of virtual memory for the binary. Also,
if you are using ASAN, see /usr/local/share/doc/afl/notes_for_asan.txt.

- Least likely, there is a horrible bug in the fuzzer. If other options
fail, poke <afl-...@googlegroups.com> for troubleshooting tips.

[-] PROGRAM ABORT : Test case 'id:000000,time:0,orig:input' results in a crash
Location : perform_dry_run(), src/afl-fuzz-init.c:603

Running ( ulimit -Sd $[149 << 10]; ./quadlods interact <in/input ) from bash
results in clean output. (My usual memory limit is 150; I upped it in case
memory was the problem.) Same for the other two files, including test.data,
which I recently added to the repo. (The divide by zero is one of the things I
test.) When I run it with only test.data in in/, it still crashes in afl++.
What's wrong?

Pierre
--
The gostak pelled at the fostin lutt for darfs for her martle plave.
The darfs had smibbed, the lutt was thale, and the pilter had nothing snave.

[Marc Heuse]

It could be that increasing the memory with -m none will prevent it crashing. It could just be a malloc failing. (Which can be an exploitable bug)

(2.60 is pretty old btw)

[Pierre Abbat]

On Friday, September 25, 2020 3:11:05 PM EDT Marc Heuse wrote:
> It could be that increasing the memory with -m none will prevent it
> crashing. It could just be a malloc failing. (Which can be an exploitable
> bug)

-m none does make it succeed.

> (2.60 is pretty old btw)

I've pulled the latest and will try it.

Pierre
--
When a barnacle settles down, its brain disintegrates.
Já não percebe nada, já não percebe nada.

[Pierre Abbat]

I don't like fuzzing a program with no memory limit, but -m 1000 works. It may
need that much because it's reading a 400 MB data file, which wasn't there when
I released the previous version. It reads only a small part of the file, but
maybe the fuzzer is causing it to memory-map the file or something.

Pierre

--
I believe in Yellow when I'm in Sweden and in Black when I'm in Wales.

[Pierre Abbat]

I left the fuzzer running for not quite an hour. When I came back, it said it
found five hangs and three crashes. The hangs are very long runtimes (the fuzz
file told it to generate over 200,000 tuples instead of less than 100). But the
crashes don't seem to be valid crashes. The files contain bit flips in the
commands. When I ran normally-compiled Quadlods on them, it said "Parse error"
on two and "Invalid command" on the third. Quadlods compiled for fuzzing did
the same; it didn't crash. Yet AFL++ says it died with signal 11 (segfault).
What gives?

[dome...@gmail.com]

Hi,

Souds a bit like it's hitting your mem limit again, and, in turn, tries to access the null page? 

Are you running your target in persistent mode?  There it could also have assembled some state from prior testcases that ultimately leads to the crash. 

Best 

Dominik

[Pierre Abbat]

On Saturday, September 26, 2020 2:46:18 AM EDT dome...@gmail.com wrote:
> Hi,
>
> Souds a bit like it's hitting your mem limit again, and, in turn, tries to
> access the null page?
> Are you running your target in persistent mode? There it could also have
> assembled some state from prior testcases that ultimately leads to the
> crash.

I'm not using persistent mode. Can you try fuzzing it? I've attached the three
files that supposedly make it crash, along with two of the seed files (the third
is test.data in the repo).

Should I file a bug report?

[Marc]

If it crashes it crashes. if it is difficult to reproduce it still means
it is a crash. there is no bug.

does it also crash with afl-showmap? and if you put the crashes into an
input directory and start a new fuzz run, is afl-fuzz complaining that
there are crashing seeds?

On 27.09.20 03:44, Pierre Abbat wrote:
> On Saturday, September 26, 2020 2:46:18 AM EDT dome...@gmail.com wrote:
>> Hi,
>>
>> Souds a bit like it's hitting your mem limit again, and, in turn, tries to
>> access the null page?
>> Are you running your target in persistent mode? There it could also have
>> assembled some state from prior testcases that ultimately leads to the
>> crash.
>
> I'm not using persistent mode. Can you try fuzzing it? I've attached the three
> files that supposedly make it crash, along with two of the seed files (the third
> is test.data in the repo).
>
> Should I file a bug report?

--
Marc Heuse
www.mh-sec.de

PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573

[Pierre Abbat]

On Sunday, September 27, 2020 3:22:14 AM EDT Marc wrote:
> If it crashes it crashes. if it is difficult to reproduce it still means
> it is a crash. there is no bug.
>
> does it also crash with afl-showmap? and if you put the crashes into an
> input directory and start a new fuzz run, is afl-fuzz complaining that
> there are crashing seeds?

I ran afl-showmap; it gave me a file of numbers for each, and did not say that
they crashed. afl-fuzz does not complain about crashing seeds. It does, though,
say that the instrumentation varies across runs. How do I find where it is
varying?

Pierre

--
Lanthanidia deliciosa: What the kiwifruit would be
if it weren't so radioactive.

[Marc]

Hi,

On 27.09.20 13:28, Pierre Abbat wrote:
> On Sunday, September 27, 2020 3:22:14 AM EDT Marc wrote:
>> If it crashes it crashes. if it is difficult to reproduce it still means
>> it is a crash. there is no bug.
>>
>> does it also crash with afl-showmap? and if you put the crashes into an
>> input directory and start a new fuzz run, is afl-fuzz complaining that
>> there are crashing seeds?
>
> I ran afl-showmap; it gave me a file of numbers for each, and did not say that
> they crashed. afl-fuzz does not complain about crashing seeds. It does, though,
> say that the instrumentation varies across runs. How do I find where it is
> varying?

I sometimes have similar issues. that target crashes but It is not
possible to reproduce. For me this is is with persistent harnesses
though where this can easily happen because of library state that is
there. in normal forkserver fuzzing I havent seen that so far. try to
set the same -m settings etc.

For instability, why, how to see where and maybe how to get rid of it:
https://github.com/AFLplusplus/AFLplusplus/blob/dev/docs/FAQ.md#why-is-my-stability-below-100

Regards,
Marc

[Pierre Abbat]

On Sunday, September 27, 2020 7:35:09 AM EDT Marc wrote:
> I sometimes have similar issues. that target crashes but It is not
> possible to reproduce. For me this is is with persistent harnesses
> though where this can easily happen because of library state that is
> there. in normal forkserver fuzzing I havent seen that so far. try to
> set the same -m settings etc.

I'm using the forkserver.

> For instability, why, how to see where and maybe how to get rid of it:
> https://github.com/AFLplusplus/AFLplusplus/blob/dev/docs/FAQ.md#why-is-my-st
> ability-below-100

The stability is 90% when run on the files that crashed it. I didn't notice the
stability when I ran it with the original set of files; one of them seeds the
generators with randomness, but the crash files are derived from a test file
that should be deterministic.

I used this command to build for fuzzing:
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=~ -
DCMAKE_C_COMPILER=afl-clang -DCMAKE_CXX_COMPILER=afl-clang++ -DNO_INSTALL=1 ~/
src/$1

Does that use PCGUARD or LTO?

[Marc]

Hi Pierre,

> I used this command to build for fuzzing:
> cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=~ -
> DCMAKE_C_COMPILER=afl-clang -DCMAKE_CXX_COMPILER=afl-clang++ -DNO_INSTALL=1 ~/
> src/$1
>
> Does that use PCGUARD or LTO?

that is hard to say, as afl-clang will point to afl-clang-fast if
llvm_mode was compiled and installed, and to afl-gcc if not.

Use afl-clang-fast or afl-clang-lto directly, always.

[Pierre Abbat]

I think I've figured it out. I recently added multithreading to the program for
computing the discrepancy of a sequence. The varying instrumentation comes
from threads running different numbers of times before they're told to stop.
The crash is a bug in pthreads or the C++ thread library, in which if two
threads exit at exactly the same time, something gets corrupted. The
interactive mode, which is what I'm fuzzing, does not use threads, so I can
turn them off for fuzzing.

Pierre

--
sei do'anai mi'a djuno puze'e noroi nalselganse srera