Ignore crashing input when resuming

[Leo Barnes]

I have a small enhancement request for a problem I have been running into lately:

I am running several hundred fuzzers that sync every now and then. Some of the new paths found by some of the fuzzers have found a small memory corruption bug that only happens sometimes (which is why it got classified as a new path rather than a crash). These paths are then imported by the other fuzzers. Every 24 hours I have to restart the fuzzing jobs. I do this by specifying -i- to get them to resume. The problem is that some of the fuzzing jobs die on the initial run of all the input since some of the inputs crash.

I would say that the most logical way for the fuzzer to handle this is to not quit on crashing inputs when you resume, but rather copy/move it to the crash output and ignore it in the future.

Thanks!

[Jakub Wilk]

* Leo Barnes <barne...@gmail.com>, 2015-08-06, 08:48:

>I am running several hundred fuzzers that sync every now and then. Some
>of the new paths found by some of the fuzzers have found a small memory
>corruption bug that only happens sometimes (which is why it got
>classified as a new path rather than a crash). These paths are then
>imported by the other fuzzers. Every 24 hours I have to restart the
>fuzzing jobs. I do this by specifying -i- to get them to resume. The
>problem is that some of the fuzzing jobs die on the initial run of all
>the input since some of the inputs crash.
>
>I would say that the most logical way for the fuzzer to handle this is
>to not quit on crashing inputs when you resume, but rather copy/move it
>to the crash output and ignore it in the future.

https://groups.google.com/d/topic/afl-users/ue3u3MKYJQU
Apparently it's difficult to implement. :(

--
Jakub Wilk

[Leo Barnes]

Ah, too bad. Running afl-cmin on all the fuzzer input queues is going to be very time consuming and may not work since the only way of reliably getting the crashing files to crash is to run with ASan enabled (which is problematic when running afl-fuzz/cmin since I have to run in 64-bit mode).

What does the fuzzer need/expect when it resumes? Can I remove files manually from the queue directory before starting the fuzzer without it going haywire, or do all the files have to have sequential names?

If I can remove files, I might just create some kind of pre-filter script that runs before starting the fuzzer and simply removes all crashing files from the queue directory.

Cheers,

//Leo

 

--
Jakub Wilk

[Michal Zalewski]

>> https://groups.google.com/d/topic/afl-users/ue3u3MKYJQU
>> Apparently it's difficult to implement. :(

Since it's a recurring theme, I'll try to put together a solution.
It's not going to be perfect, but at least for session resumption, we
can put something together. Sorry for the slow pace of updates of
recent, but I just had a kid born and wasn't getting as much sleep /
free time as before =)

> What does the fuzzer need/expect when it resumes?

The most important part are queue/.state/ entries, which are used to
figure out which files have gone through deterministic fuzzing to
avoid wasting CPU time. You can add or delete queue/ entries if you
want.

/mz

[Leo Barnes]

On Friday, August 7, 2015 at 10:49:32 AM UTC-7, Michal Zalewski wrote:

>> https://groups.google.com/d/topic/afl-users/ue3u3MKYJQU
>> Apparently it's difficult to implement. :(

Since it's a recurring theme, I'll try to put together a solution.
It's not going to be perfect, but at least for session resumption, we
can put something together. Sorry for the slow pace of updates of
recent, but I just had a kid born and wasn't getting as much sleep /
free time as before =)

No worries, that is totally understandable :)

 

> What does the fuzzer need/expect when it resumes?

The most important part are queue/.state/ entries, which are used to
figure out which files have gone through deterministic fuzzing to
avoid wasting CPU time. You can add or delete queue/ entries if you
want.

If I'm only running slave sessions (that don't run the deterministic fuzzing if I understand it correct) can the .state entries safely be deleted?

Thanks!

//Leo

 

/mz

[Michal Zalewski]

> If I'm only running slave sessions (that don't run the deterministic fuzzing
> if I understand it correct) can the .state entries safely be deleted?

You can delete it, but nothing bad will happen if you keep it.

/mz

[Michal Zalewski]

OK, version 1.86b will support AFL_SKIP_CRASHES. This causes crashes
to be treated somewhat similarly to timeouts. They don't go away, but
are not fuzzed. The setting is hacky and will inhibit many of the
usual sanity checks (say, the ASAN / memory limit diagnostics, etc).

/mz

[Ben Nagy]

* Michal Zalewski <lca...@gmail.com> [2015-08-08 21:29:35 -0700]:

Wait.. so are saying that they will be copied in from the -i directory
but never fuzzed? Or do you just mean that if they're somehow already
there when you resume they will be internally marked as dead but not
deleted?

FWIW I have a similar problem at the moment. I'm doing lots of testing
involving broad corpora (for complex formats like PDF) so a regular
thing I do is start a fuzz against a target with a curated corpus. My
dream behaviour would be for the new run to not import crashes, not
import "no new instrumentation", and not import timeouts.

It is better (as per your previous suggestion) to start by using
afl-cmin on the 'general' corpus to make a new corpus just for this
target, but I still sometimes have coverage dups etc.

Anyway, my 0.02, an expert setting to agressively skip _anything_ in -i
that you would normally warn or abort on would be most welcome.

Cheers,

ben

[Michal Zalewski]

> Wait.. so are saying that they will be copied in from the -i directory
> but never fuzzed? Or do you just mean that if they're somehow already
> there when you resume they will be internally marked as dead but not
> deleted?

They will be internally marked as bad, and that's it. It's implemented
in a manner very similar to -t nn+, which marks timeouts as "bad" but
doesn't abort. The main reason why it's kept as an env variable rather
than something like -t nn+ is that it disables many useful crash
diagnostics and because intermittently crashing test cases will
probably produce somewhat sketchy results anyway.

> My dream behaviour would be for the new run to not import crashes, not
> import "no new instrumentation", and not import timeouts.

afl-cmin + afl-tmin is still the way to go; afl-fuzz just isn't
designed to do any complex discrimination before importing stuff. It
gets loaded into memory first, sorted, renamed, given sequential IDs,
etc. Calibration happens only later on, and at that point, we can at
best mark stuff as "bad" (but it will still be used for splicing,
among other things).

Another alternative is to use the sync behavior and put the corpus in
a sync directory. This would skip timeouts, crashes, and "no new
coverage" entries. Perhaps that's what you want?

/mz

[Ben Nagy]

* Michal Zalewski <lca...@gmail.com> [2015-08-08 22:05:12 -0700]:

>> My dream behaviour would be for the new run to not import crashes, not
>> import "no new instrumentation", and not import timeouts.
>
>afl-cmin + afl-tmin is still the way to go; afl-fuzz just isn't
>designed to do any complex discrimination before importing stuff. It
>gets loaded into memory first, sorted, renamed, given sequential IDs,
>etc. Calibration happens only later on, and at that point, we can at
>best mark stuff as "bad" (but it will still be used for splicing,
>among other things).
>
>Another alternative is to use the sync behavior and put the corpus in
>a sync directory. This would skip timeouts, crashes, and "no new
>coverage" entries. Perhaps that's what you want?

That will import crashes (to crashes/), not skip them, but that's not really a
problem. That's a pretty nice idea.. I guess running afl-cmin is still
best, because that's the only way to bias coverage by size, right?

Reminds me I still need a dumb wrapper to bulk-tmin :/

Thanks!

ben

[Michal Zalewski]

> I guess running afl-cmin is still
> best, because that's the only way to bias coverage by size, right?

AFL does that internally anyway, but only in a probabilistic fashion -
basically, it spends more time on a smaller sub-corpus, but assumes
that you might be still trying to say something meaningful by giving
it a larger set and tries to get to every input every now and then.
Running afl-cmin removes any ambiguity (and is also more thorough).

/mz

[Ben Nagy]

* Michal Zalewski <lca...@gmail.com> [2015-08-08 22:29:36 -0700]:

>> I guess running afl-cmin is still
>> best, because that's the only way to bias coverage by size, right?
>
>AFL does that internally anyway, but only in a probabilistic fashion -
>basically, it spends more time on a smaller sub-corpus

I meant by individual file size, not corpus size. Are we on the same page?

ben

[Michal Zalewski]

>>> I guess running afl-cmin is still
>>> best, because that's the only way to bias coverage by size, right?
>> AFL does that internally anyway, but only in a probabilistic fashion -
>> basically, it spends more time on a smaller sub-corpus
>
> I meant by individual file size, not corpus size. Are we on the same page?

The internal afl-fuzz algo prefers small sizes & fast runtimes for the
"favored" corpus, too. Relevant lines:

/* Faster-executing or smaller test cases are favored. */

if (fav_factor > top_rated[i]->exec_us * top_rated[i]->len) continue;

/mz

[Ben Nagy]

* Michal Zalewski <lca...@gmail.com> [2015-08-08 22:55:52 -0700]:

>> I meant by individual file size, not corpus size. Are we on the same page?
>
>The internal afl-fuzz algo prefers small sizes & fast runtimes for the
>"favored" corpus, too. Relevant lines:
>
> /* Faster-executing or smaller test cases are favored. */
>
> if (fav_factor > top_rated[i]->exec_us * top_rated[i]->len) continue;

Awesome. I'll do some experiments.

ben

[floyd]

On 09/08/15 07:14, Ben Nagy wrote:
>
> Reminds me I still need a dumb wrapper to bulk-tmin :/

I know you probably won't be happy with python and this is not a
standalone tool, but afl-crash-analyzer has it:
https://github.com/floyd-fuh/afl-crash-analyzer/blob/master/modules/InputMinimizer.py

[Leo Barnes]

That will import crashes (to crashes/), not skip them, but that's not really a
problem. That's a pretty nice idea.. I guess running afl-cmin is still
best, because that's the only way to bias coverage by size, right?

Good idea with doing resumes by sync rather than actual resume. Should be pretty trivial to modify my fuzzing jobs to do that. Just to get my options straight, these are my options from what I can see (please correct me if I'm wrong on any of the details):

1) Resume-by-sync:

Crashes are handled by being moved to crashes/. (I think?)

+ Very easy to implement/use

+ More or less as fast as resuming (I assume)

2) Running afl-cmin before resuming:

Not completely sure what afl-cmin does when it finds a crash. Is it removed?

+ Removes lots of redundant input which should make continued fuzzing faster

- Slow

3) AFL_SKIP_CRASHES

Crashes are marked as ignored input in the same way as slow inputs or inputs without instrumentation are.

+ Fast

- Removes some sanity checks? (Not completely sure what this means. Does it remove sanity checks during dry run only?)

Are there any drawbacks with resume-by-sync that I'm missing?

Thanks!

[Michal Zalewski]

> 1) Resume-by-sync:

> 2) Running afl-cmin before resuming:

> 3) AFL_SKIP_CRASHES

Normally, AFL takes all inputs as-is. It also preserves state
metadata, so a master instance that resumes a job won't needlessly
repeat deterministic fuzzing steps that take up the bulk of the first
few queue cycles. Use the normal -i logic whenever you can.

AFL_SKIP_CRASHES in conjunction with -i inhibits many of the "dry run"
error messages that would give you hints about memory limits and so
on, so don't set it unless you're running into some real problems with
concurrency-related intermittent crashes in the corpus (which should
be rare).

Sync functionality is designed to pull in only a subset of
sequentially numbered test cases from another fuzzer, choosing only
the ones that improve coverage and do not crash or time out. It's not
inherently faster or slower, but it ignores metadata and disables many
useful diagnostics that would warn you about your input corpus being
poorly chosen. It may optimize a corpus somewhat, but it doesn't do as
much as afl-cmin. So, don't use it unless you're actually syncing with
other fuzzers, or unless it scratches some other itch (with -t nn+ and
AFL_SKIP_CRASHES implemented, I imagine that it no longer offers any
real advantage).

The afl-cmin tool is useful for shrinking and optimizing a starting
corpus obtained from some other source (test suites, web crawls, ...),
so that you don't end up with more files than you really need. It's
also useful for optimizing the output corpus produced by afl-fuzz
before feeding it to other tools (e.g., testing harnesses for
interaction-requiring apps). You generally don't need to run it in
other circumstances.

/mz