Core dumps / crash data

[j.david...@gmail.com]

Are there any options for getting output/coredump/abort data on crashes only? 

I'm seeing crashes that are super infrequent, and not very reproducible while running afl-fuzz, and this data would help that. 

Thank you,

Justin

[Michal Zalewski]

> Are there any options for getting output/coredump/abort data on crashes
> only?

Not easily. Capturing core dumps adds latency. Capturing stderr
shouldn't be too hard, but isn't currently implemented (the simplest
way to do it may be to just open a log file from within the targeted
program, and then do dup2()).

You can probably get faulting address from dmesg if you're on Linux, too.

Do you mean that the crashes are intermittent (e.g., due
to thread concurrency issues), or that they are consistently
non-reproducible? Perhaps trying the test case in a loop until you get
a core dump would be a viable starting point.

/mz

[j.david...@gmail.com]

I have it set up to be able to log to a file, however figuring out which ones are for failed runs vs successful runs is a bit difficult. 

I should have thought of just looping it and waiting for it to crash on its own. This particular error was an assert being thrown though, so I don't believe it would have been showing up in dmesg. 

It was a thread concurrency issue; I thought maybe the overhead and CPU thrashing that goes along with running afl-fuzz might have been a factor in making the edge case happen, but it did happen eventually outside of it using test cases that afl-fuzz came up with that seemed to be a particular tough torture case for the faulty logic. 

I do get warnings -- "[!] WARNING: Instrumentation output varies across runs." -- I assume this is due to the threading and lack of absolute consistency in thread timings? 

While I've only used the tool a little bit, it is extremely impressive the stuff it finds. The application is a simulation of, essentially, a cnc machine, where the input is the commands to the machine, and asserts are thrown if invalid/dangerous conditions occur. The fuzzer has already come up with a number of edge cases that weren't properly handled and I haven't even let it finish a cycle yet. Great work!

Have a good night,

Justin

[Michal Zalewski]

> I do get warnings -- "[!] WARNING: Instrumentation output varies across
> runs." -- I assume this is due to the threading and lack of absolute
> consistency in thread timings?

Yup. This is generally not a problem, except that it may be worth
investigating when you do not expect it (say, in a single-threaded
application that doesn't use randomness, where it could be indicative
of some actual bugs).

/mz

[Heiko Eißfeldt]

Interesting! Could you elaborate on that?

I got this often in what is supposed to be deterministic test programs,

but did not understand the issues behind that.

So i ignored these messages, but it seems not wise to do so.

 

Instrumentation feedback should depend only on test stimulus, right?

How is this 'variable behaviour of test programs' determined by afl-fuzz?

 

Thanks, Heiko

[Michal Zalewski]

> Interesting! Could you elaborate on that?

Quoting status_screen.txt:

There are four quasi-common explanations for variable behavior of the
tested program:

- Use of uninitialized memory in conjunction with some intrinsic sources of
entropy in the tested binary. This can be indicative of a security bug.

- Multiple threads executing at once in semi-random order. Not a big deal,
but to avoid hiccups, it's best to restrict instrumented programs to a
single thread. Check compile-time options or run-time flags. For example,
for ImageMagick, you can try --without-threads --disable-openmp; for
ffmpeg, look for --disable-pthreads instead.

- Attempts to create files that were already created during previous runs, or
otherwise interact with some form of persistent state. This is harmless,
but you may want to instruct the targeted program to write to stdout or to
/dev/null to avoid surprises (and disable the creation of temporary files
and similar artifacts, if applicable).

- Hitting functionality that is actually designed to behave randomly. For
example, when fuzzing sqlite, the fuzzer will dutifully detect variable
behavior once the mutation engine generates something like:

select random();

Less likely causes may include running out of disk space, SHM handles, or other
globally limited resources.

The paths where variable behavior is detected are marked with with a matching
entry in the <out_dir>/.state/variable_behavior/ directory, so you can look
them up easily.

If you can't suppress variable behavior and don't want to see these warnings,
simply set AFL_NO_VAR_CHECK=1 in the environment before running afl-fuzz. This
will also dramatically speed up session resumption.

> Instrumentation feedback should depend only on test stimulus, right?

That's usually the expected behavior.

> How is this 'variable behaviour of test programs' determined by afl-fuzz?

The sequence in which basic blocks are executed differs from one run
to another. AFL doesn't detect or handle threads in any special way.

/mz

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2015-05-07, 07:25:

>There are four quasi-common explanations for variable behavior of the
>tested program:

[...]

> - Multiple threads executing at once in semi-random order. Not a big deal,

I'm somewhat surprised by this claim. Surely noise in the
instrumentation could interfere with afl-fuzz detecting new behaviors?

Oh, and heavily-multi-threaded code can saturate the bitmap pretty
quickly.

--
Jakub Wilk

[Michal Zalewski]

> I'm somewhat surprised by this claim. Surely noise in the instrumentation
> could interfere with afl-fuzz detecting new behaviors?

In a hypothetical worst-case scenario, yup. In practice, it usually
just contributes a couple spurious paths, but scheduler behavior and
the fact that threads are idle much of the time and tend to have
explicit synchronization points means that it's usually not causing
major problems.

/mz