Feature request: sending input files straight to crashes/

[Jakub Wilk]

Hi Michal, hi AFL users!

When afl-fuzz notices that one of the input files causes the target
program to crash, it aborts instructing the user to fix the problem.
And this is normally the right thing to do.

However, if you have a big corpus of interesting files using tool A, and
then you're feeding to a different tool B, then it would be much more
convenient if afl-fuzz simply copied crashy input files to crashes/ and
moved on with remaining input files.

Could you add an option for that? :-)

(As a work around, one could probably create fake worker directory
seeded with the corpus, and let afl-fuzz sync from it. But it's
cumbersome to set it all up.)

--
Jakub Wilk

[Steven Vittitoe]

+1 to this feature. I spent a bunch of time last week pruning an input corpus. Many of us probably have some scripts that automate this already.

--
Jakub Wilk

--
You received this message because you are subscribed to the Google Groups "afl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michal Zalewski]

> +1 to this feature. I spent a bunch of time last week pruning an input
> corpus. Many of us probably have some scripts that automate this already.

Would it be simpler to add a setting for afl-cmin to remove crashing
test cases? Any circumstances where you'd be taking a random
collection of input files but not running afl-cmin on it?

/mz

[Steven Vittitoe]

I have no use cases where running afl-cmin on a corpus would cause an issue. Sounds like a fine solution.

--
You received this message because you are subscribed to the Google Groups "afl-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2015-02-08, 12:55:

>>+1 to this feature. I spent a bunch of time last week pruning an input
>>corpus. Many of us probably have some scripts that automate this
>>already.
>
>Would it be simpler to add a setting for afl-cmin to remove crashing
>test cases?

I want to know which files triggered crashes.
But with s/remove/move elsewhere/, sounds good to me.

--
Jakub Wilk

[Michal Zalewski]

> I have no use cases where running afl-cmin on a corpus would cause an issue.
> Sounds like a fine solution.

Okie, added, will ship in the next release.

/mz

[Sami Liedes]

Not a random collection, but for me this has been a problem when
resuming. If the machine is otherwise under load or the behavior of
the program being tested varies, typically one of the test cases (of
which there may be thousands...) fails and then afl just refuses to
start.

Sami

[Michal Zalewski]

>> Would it be simpler to add a setting for afl-cmin to remove crashing
>> test cases? Any circumstances where you'd be taking a random
>> collection of input files but not running afl-cmin on it?
>
> Not a random collection, but for me this has been a problem when
> resuming. If the machine is otherwise under load or the behavior of
> the program being tested varies, typically one of the test cases (of
> which there may be thousands...) fails and then afl just refuses to
> start.

Huh? Test cases start to randomly crash under load?

Or do you mean, time out? If the latter, I think that skipping them is
long-supported at this point (just use -t nn+ instead of -t nn).

/mz

[Michal Zalewski]

> I want to know which files triggered crashes.
> But with s/remove/move elsewhere/, sounds good to me.

This would make afl-cmin slower. I need to rewrite it in C, but for
now, I just want it to be as fast as it can while still being a shell
script. Ignoring crashes requires no extra code in afl-cmin (just a
couple lines in afl-showmap in -C mode). Moving them around and (and
preferably also minimizing that corpus!) is more involved and will
slow down one of the performance-critical loops.

Now, to elaborate on why I wanted to go with afl-cmin instead of
adding a check in afl-fuzz:

1) It's problematic to allow afl-cmin to do minimization while looking
at crashing inputs as first-class citizens, only to have them rejected
by the fuzzer. This makes it possible for afl-cmin to decide that a
crashing input is the best candidate for a bunch of tuples that are
also covered by several non-crashing inputs, and reject the latter
bunch. Then, afl-fuzz will reject the crashing candidate selected by
afl-cmin, and your loaded corpus will have glaring gaps.

2) The current design of afl-fuzz isn't very conductive to moving test
cases out of the queue. The "dry run" takes place only after the files
are sorted, loaded into memory, renamed, gived filename-tied IDs that
make them traceable to other inputs, etc. This design can be changed,
but it has many other benefits; for example, it means that you don't
leave the output directory in some half-baked state if you hit Ctrl-C
before the startup is complete (especially important for in-place
resume).

At this point, my gut feeling is that supporting AFL_SKIP_CRASHES=1 is
worthwhile; but putting the crashes in a separate bucket will have to
wait until the rewrite of afl-cmin. I can add exit codes to
afl-showmap.c to make it easier to write a simple for loop to detect,
crashes, though. Something like:

for i in corpus/*; do ./afl-showmap -o /dev/null -t 123 -m 456
/path/to/foo $i; test "$?" = "2" && mv $i my_crashes/; done

/mz

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2015-02-08, 14:57:

>At this point, my gut feeling is that supporting AFL_SKIP_CRASHES=1 is
>worthwhile; but putting the crashes in a separate bucket will have to
>wait until the rewrite of afl-cmin. I can add exit codes to
>afl-showmap.c to make it easier to write a simple for loop to detect,
>crashes, though. Something like:
>
>for i in corpus/*; do ./afl-showmap -o /dev/null -t 123 -m 456
>/path/to/foo $i; test "$?" = "2" && mv $i my_crashes/; done

How about adding also AFL_ONLY_CRASHES?

--
Jakub Wilk

[Michal Zalewski]

> How about adding also AFL_ONLY_CRASHES?

Can do.

/mz

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2015-02-09, 08:08:

>>How about adding also AFL_ONLY_CRASHES?
>Can do.

Or perhaps, for consistency with afl-fuzz, it should be a command-line
option (-C)?

--
Jakub Wilk

[Smith, Thomas G. (NIH/NLM/NCBI) [C]]

> Not a random collection, but for me this has been a problem when
> resuming. If the machine is otherwise under load or the behavior of
> the program being tested varies, typically one of the test cases (of
> which there may be thousands...) fails and then afl just refuses to
> start.

I had a similar problem, where the program under test is less deterministic than one might hope, and a few of the file in the queue crash when afl-fuzz tries to resume. The program is also horribly slow, so it took all day to resume (I eventually just deleted most of the corpus so it would start).

I'm fuzzing a program that does a bunch of really involved validation on a file's semantics - should I have started by fuzzing a simpler, faster program that just understands the syntax but doesn't care about the meaning?

Thanks for everything :)
-Thomas

[Michal Zalewski]

>>> How about adding also AFL_ONLY_CRASHES?
>> Can do.
> Or perhaps, for consistency with afl-fuzz, it should be a command-line
> option (-C)?

Yup, that's exactly what I have pending :-)

/mz