afl-sid: a PoC repository of ready-to-fuzz packages (debian dev needed)

[Jacek Wielemborek]

List,

I had been playing with this idea for a while and it looks like the
proof of concept is ready for releasing!

There was a discussion on #afl-users IRC channel about whether it makes
sense to repeat fuzzing of already tested projects. While quite a few
people would argue that it makes more sense to concentrate on building
better test corporas, I find some benefit in regression testing and
inviting people to continue my fuzzing where I left it, adding new test
cases and other ideas. Given that I'm still going through the "fuzz all
the things" phase, I figured I'd try to lower the bar a bit by making
fuzzing more accessible.

So this was my idea: create a Debian repository where the user can
quickly download a package for the program he wants to fuzz, without
messing with the build system and waiting for the compilation to finish.
The package built with afl-fuzz instrumentation can then easily be
shared with others by just copying the repo URL to /etc/apt/sources.list.

In order to achieve that, I read up on how Debian packages can be
rebuilt and created "aflize" script that basically sets CC=afl-gcc
CXX=afl-g++, enables ASAN and AFL_HARDEN. Then I created a Docker
environment in which all the packages are rebuilt with those settings
and created a Debian repository out of it. A few packages didn't
actually use AFL instrumentation (for example GCC, which tries to
bootstrap itself), some didn't built at all (e.g. systemd, possibly
because of compiler errors), but for example I had some success with
dpkg and I think I even found some suspicious input. Anyway, having no
idea where to put there repo I decided to upload it to Github pages. In
order to try it out, run "sudo docker run -ti debian:sid" and instead of
original /etc/apt/sources.list contents, just put this:

deb http://d33tah.github.io/afl-sid-repo sid main

Then run apt-get update and try some package from Debian base by running
a command line apt-get install --reinstall packagename. The packages
aren't signed at the moment.

You can also try downloading my afl-sid Github repository [1] and trying
out aflize on a Debian package you come up with - let me know if you
managed to get it running or need some more help.

What do you think about this idea?

Cheers,
d33tah

[Michal Zalewski]

This is pretty cool. One thing that I get asked quite often is "I see
program X on the trophy list; is it worthwhile to still fuzz it?" -
and much of the time, I don't know the answer, because - among other
things - it's not clear how many CPU-hours went into the project, and
how good the original researcher was about selecting a good corpus,
coming up with a dictionary, exercising all possible settings /
parsing modes, or tracking down and reporting all the bugs.

In addition to prepackaged fuzzer + binary images, I wonder if it'd
make sense to put more effort into tracking this.

> --
> You received this message because you are subscribed to the Google Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jakub Wilk]

I normally delete all mails matching /[Dd]ocker/ without reading ;> but
this time I made an exception:

* Jacek Wielemborek <d33...@gmail.com>, 2015-09-02, 17:02:

>You can also try downloading my afl-sid Github repository [1]

404 Footnote Not Found

--
Jakub Wilk

[Jacek Wielemborek]

W dniu 03.09.2015 o 09:28, Jakub Wilk pisze:

Sorry:

https://github.com/d33tah/afl-sid

[Jacek Wielemborek]

W dniu 02.09.2015 o 17:41, Michal Zalewski pisze:

> This is pretty cool. One thing that I get asked quite often is "I see
> program X on the trophy list; is it worthwhile to still fuzz it?" -
> and much of the time, I don't know the answer, because - among other
> things - it's not clear how many CPU-hours went into the project, and
> how good the original researcher was about selecting a good corpus,
> coming up with a dictionary, exercising all possible settings /
> parsing modes, or tracking down and reporting all the bugs.
>
> In addition to prepackaged fuzzer + binary images, I wonder if it'd
> make sense to put more effort into tracking this.

I like it and already tried to contact Hanno once about it. Just had
another thought about it:

Just the ability to upload and view fuzzer_stats files would be a great
improvement - knowing how long the program was fuzzed and how many paths
were found can tell how much effort was put into testing. Also, there's
the command line visible, which tells a bit about coverage.

[Paris Zoumpouloglou]

There has been a lot of interest in fuzzing OSS during this year.
Fuzzing Project's attempt to categorize the fuzzing progress using a few
labels (WIP,OK etc.) was a good start but I think we need to get into
some more detail regarding how much work has been done.

If we can determine that stats/options/metrics that interest us there
can be a more formal way to report and keep track of research through
fuzzing project

[Jacek Wielemborek]

W dniu 05.09.2015 o 17:15, Paris Zoumpouloglou pisze:

Is there any example you can come up with of how this could look like?

[Hanno Böck]

Hi,

On Sat, 5 Sep 2015 13:49:07 +0200
Jacek Wielemborek <d33...@gmail.com> wrote:

> W dniu 02.09.2015 o 17:41, Michal Zalewski pisze:

> > In addition to prepackaged fuzzer + binary images, I wonder if it'd
> > make sense to put more effort into tracking this.
>
> I like it and already tried to contact Hanno once about it. Just had
> another thought about it:

Sorry for being slow to answer lately, have been busy with other
things. I hope I can be more responsitive again now.

Actually originally the main idea behind the fuzzing project was this
kind of "tracking what has been fuzzed". However lately I was wondering
what to do about that, it didn't really work out as I had planned.
For one I had some specific ideas how I would track things, but that
wasn't easy to grasp for others. On the other hand what I had hoped -
that this would be some kind of "quality sign" and people would
complain "I want my software to be green" didn't really happen as far
as I could see.

Then I got requests like the one from Jacek about publishing more info.
Right now I think I might just create a wiki where people can track
their fuzzing effort. That way everyone can just track it up to the
detail they see suitable, including stats etc.

I'm always a bit hesistant in setting up wikis. They can be great if
they work, but very often they don't. But it may very well be that we
have a community here that will be active enough to make it work.
So I'll probably set up a wiki within the next days, move the software
list there and see how it evolves.

--
Hanno Böck
http://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[Jacek Wielemborek]

W dniu 05.09.2015 o 19:01, Hanno Böck pisze:

Perhaps a semantic mediawiki [1] would make it feel more organized for
you? I used it while registering for some Chaos Communication Cogress
assembly chair and it was really nice.

[1]: https://semantic-mediawiki.org/

[Michael Rash]

The type of information I'd like to see for a given OSS project would be something like:

- fuzzer_stats

- Corresponding code coverage - the lcov report is probably best here I'm guessing.

- How the project was built and whether a patch was applied to circumvent a CRC or a crypto operation.

- Maybe also the initial AFL test cases?

- Project version and fuzzing environment

If we're trying to be more effective as a fuzzing community built around AFL, then having the above information would at a minimum help reduce duplicated effort. Personally, I think all of this data could just go into a git repository on github.

Thanks,

--Mike

 

[1]: https://semantic-mediawiki.org/

[Jacek Wielemborek]

W dniu 06.09.2015 o 05:22, Michael Rash pisze:
> and fuzzing environment

Apart from that, I think I understand and agree with everything. What do
you mean by "fuzzing environment" though?

[Sami Liedes]

On Sat, Sep 05, 2015 at 11:22:46PM -0400, Michael Rash wrote:
> The type of information I'd like to see for a given OSS project would be
> something like:
>
> - fuzzer_stats
> - Corresponding code coverage - the lcov report is probably best here I'm
> guessing.
> - How the project was built and whether a patch was applied to circumvent a
> CRC or a crypto operation.
> - Maybe also the initial AFL test cases?
> - Project version and fuzzing environment

How about which libraries were instrumented? It's a nontrivial choice
in many cases.

Sami

[dennis knorr]

Perhaps like providing a clean environment via docker or (s)chroot, like
with sbuild/pbuilder environments? Otherweise libraries which are not
used normally could interact with the fuzzing/the fuzzed program.

On the other side, laboratory-environments like these perhaps do not
catch every error like in a real-world-setting.

[Paris Zoumpouloglou]

Yes, community wikis are prone to failure. But if we begin with some standard page structures and work from there it will certainly be more useful than the list. Mike's list is good to begin with. Initial test cases would be a nice resource as well but I don't know how can we manage the amount after sometime.

In any case I believe Hanno's list and Michal's bug-o-rama trophy case is the starting point. Create a structure for projects that have already been fuzzed and see how it goes.

I am willing to help and I would also add the following info

- Fuzzer and version (let's not limit ourselves to AFL)
- Bug types. I think this is a nice metric because you can keep track of the errors dev. teams (and software) are prone to. A pre-defined list with counters would suffice.

-- 
Paris Zoumpouloglou
@pzmini0n

https://projectzero.gr

[Jacek Wielemborek]

W dniu 05.09.2015 o 19:01, Hanno Böck pisze:

> Right now I think I might just create a wiki where people can track
> their fuzzing effort. That way everyone can just track it up to the
> detail they see suitable, including stats etc.

Hello Hanno,


Any updates to this? Do you still plan to work on this project? How can
I contribute?

Cheers,
d33tah

[Jacek Wielemborek]

Hello,

I played with my afl-sid project a bit [1] and created a branch where
you can play with automatic coverage reports for the fuzzed samples [2].
Here's an asciicast where you can see it working with a one-byte
testcase on GNU bison:

https://asciinema.org/a/27037

At the moment, the new fuzz-pkg.sh script isn't really flexible, but you
can see the idea - using Debian packaging system, I automatically
grabbed the build dependencies and source code for a package, then
recompiled it with afl-gcc, fuzzed for 15 seconds, then recompiled with
coverage support. For each testcase from the queue we run lcov to see
the coverage statistics - this can let us see which paths various files
trigger. It's also possible to just grab a more detailed report and see
exactly which spots need more testing.

What do you think about it?

Cheers,
d33tah

[1] https://groups.google.com/d/msg/afl-users/efoTrF57Fqc/v54vvpVLCQAJ
[2] https://github.com/d33tah/aflize/tree/coverage

[Jacek Wielemborek]

W dniu 15.09.2015 o 15:09, Jacek Wielemborek pisze:

Hanno,

Ping again. I wonder if you are still interested in the project.

Cheers,
d33tah

[Doug Birdwell]

I like the ideas in this thread, and I finally have a little more time to work on the network fuzzing code I wrote for afl -- one item on my list is figuring out what happened with Jacek's attempts.

I've also been thinking about publishing a docker image with a development environment and the network fuzzing version of afl, and I think Jacek's work may be a good place to start, either in that or as a branch.  I find I'm using docker quite a lot for network fuzzing because I'm able to separate afl-fuzz executions on the same machine so they don't have port clashes, and docker is also highly useful because one can avoid installing all the dependencies that are necessary for each target in the host machine -- dependencies that can conflict across targets.

One (obvious) caveat: a site that posts fuzzing works in progress could become very popular for the wrong reasons if one is able to see unpatched exploits in the findings directories -- especially for network attacks.

Doug

[Jacek Wielemborek]

W dniu 09.10.2015 o 17:15, Doug Birdwell pisze:

> I like the ideas in this thread, and I finally have a little more
> time to work on the network fuzzing code I wrote for afl -- one item
> on my list is figuring out what happened with Jacek's attempts.

Do you mean testing the project or would you like any more specific
information from me?

W dniu 09.10.2015 o 17:15, Doug Birdwell pisze:

> One (obvious) caveat: a site that posts fuzzing works in progress could
> become very popular for the wrong reasons if one is able to see unpatched
> exploits in the findings directories -- especially for network attacks.
>
> Doug

To be honest I wasn't ever properly educated on the philosophy of fair
bug disclosure, so excuse a potentially stupid question: isn't it still
better than not having those bugs ever found or have them found by all
the wrong people? Also, AFL generates just crashing test cases, not
complete exploits.

[Doug Birdwell]

If it's an active project I believe it is good to give the people who maintain the code, if they are still active, an opportunity to fix the problem before it has the potential to become a crisis.

I'll email you separately about your projects; no need to take up others' time with my questions.

--
You received this message because you are subscribed to a topic in the Google Groups "afl-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/afl-users/efoTrF57Fqc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to afl-users+...@googlegroups.com.

[Jacek Wielemborek]

W dniu 05.09.2015 o 19:01, Hanno Böck pisze:

> I'm always a bit hesistant in setting up wikis. They can be great if
> they work, but very often they don't. But it may very well be that we
> have a community here that will be active enough to make it work.
> So I'll probably set up a wiki within the next days, move the software
> list there and see how it evolves.

Is there any progress on that? Over a week passed so far.