Fuzzing with the UBSan

[Erik de Castro Lopo]

Hi all,

I've now had two separate bug reports for two different projects
(FLAC and libsndfile) with someone fuzzing these projects with AFL
and the UBSan sanitizer.

For both of these projects, I have already spent a week of CPU time
(on each) fuzzing with the ASan sanitizer and I fixed every single
issue.

These UBSan issues do not cause the program to abort. The program
terminates normally with an exit code of 0. I suspect that AFL is
collecting them because they time out rather than the UBSan warnings.

Anyone with any options on UBSan warnings that do not cause the
program to abort abnormally?

Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

[Erik de Castro Lopo]

Erik de Castro Lopo wrote:

> I've now had two separate bug reports for two different projects
> (FLAC and libsndfile) with someone fuzzing these projects with AFL
> and the UBSan sanitizer.

I've looked at this some more. For the target programs of both of
the above projects, the file that generates undefined behaviour
exits with an exit code of either 0 or 1. In all cases, where the
target progam exits with 1, it prints an error message first.

[Alexander Cherepanov]

On 2015-08-07 10:54, Erik de Castro Lopo wrote:
> Anyone with any options on UBSan warnings that do not cause the
> program to abort abnormally?

I usually create a library with definitions for all ubsan function which
cause them to die:

(
echo '#include <stdlib.h>'
echo '#include <assert.h>'
strings /usr/lib/x86_64-linux-gnu/libubsan.so.0.0.0 | \
grep __ubsan_handle_ | \
perl -lne 'print "void $_() { assert(!\"$_\"); }"'
) | gcc -xc -std=c99 -Wall -fpic -shared -o ubsan-die.so -

And then preload it with LD_PRELOAD.

--
Alexander Cherepanov

[teor]

Under clang, the following arguments cause a trap or abort respectively on undefined behaviour.

-fsanitize-undefined-trap-on-error
-fno-sanitize-recover=all

GCC may have similar arguments.

>> These UBSan issues do not cause the program to abort. The program
>> terminates normally with an exit code of 0. I suspect that AFL is
>> collecting them because they time out rather than the UBSan warnings.
>>

>> Anyone with any options on UBSan warnings that do not cause the
>> program to abort abnormally?

Undefined behaviour gives you no guarantee about the behaviour of the program, particularly with different compiler flags, compiler versions, compilers, or platforms.

While you can often get away with undefined behaviour, it's often a clever compiler optimisation, or seemingly unrelated code change, away from failure.

http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html

Tim

Tim Wilson-Brown (teor)

teor2345 at gmail dot com
pgp ABFED1AC
https://gist.github.com/teor2345/d033b8ce0a99adbc89c5

teor at blah dot im
OTR D5BE4EC2 255D7585 F3874930 DB130265 7C9EBBC7

[Alexander Cherepanov]

On 2015-08-07 13:12, teor wrote:
>
>> On 7 Aug 2015, at 19:43 , Alexander Cherepanov <ch3...@openwall.com> wrote:
>>
>> On 2015-08-07 10:54, Erik de Castro Lopo wrote:
>>> Anyone with any options on UBSan warnings that do not cause the
>>> program to abort abnormally?
>>
>> I usually create a library with definitions for all ubsan function which cause them to die:
>>
>> (
>> echo '#include <stdlib.h>'
>> echo '#include <assert.h>'
>> strings /usr/lib/x86_64-linux-gnu/libubsan.so.0.0.0 | \
>> grep __ubsan_handle_ | \
>> perl -lne 'print "void $_() { assert(!\"$_\"); }"'
>> ) | gcc -xc -std=c99 -Wall -fpic -shared -o ubsan-die.so -
>>
>> And then preload it with LD_PRELOAD.
>
> Under clang, the following arguments cause a trap or abort respectively on undefined behaviour.
>
> -fsanitize-undefined-trap-on-error
> -fno-sanitize-recover=all
>
> GCC may have similar arguments.

Right, thanks for reminding. I've done it that way when there was no
such options in my version of gcc, I have to check what have changed
since then.

--
Alexander Cherepanov

[Michal Zalewski]

>> -fsanitize-undefined-trap-on-error
>> -fno-sanitize-recover=all

Neat, I'll mention that in the docs. The basic reason why UBSAN wasn't
originally supported is just that it didn't have a habit of reporting
failures in a way that would make sense to the fuzzer (MSAN also had
this problem, but it at least supported specifying a custom exit
code). These new options are useful, although it's going to be a while
before they are universally supported.

/mz

[Alexander Cherepanov]

BTW is it useful to fuzz with afl-fuzz and UBSAN at all? Instead of
generating a corpus and running it through UBSAN.

--
Alexander Cherepanov

[Michal Zalewski]

> BTW is it useful to fuzz with afl-fuzz and UBSAN at all? Instead of
> generating a corpus and running it through UBSAN.

...maybe? It's hard to speak in absolutes. Theoretically, there can be
some advantages (same for ASAN). But whether they offset the
performance impact, hard to tell. I generally recommend the "fuzz
first, sanitize corpus later" approach.

/mz

[Konstantin Serebryany]

Quite the opposite to what I usually suggest :) 

I have little experience with afl, but with other fuzzers I see lots of cases where 

we already have a good corpus and it's hard to find any new coverage, 

but mutating elements of this corpus leads to new asan-ish (msan-ish, ubsan-ish) bugs

even w/o discovering new coverage. 

 

/mz

--
You received this message because you are subscribed to the Google Groups "afl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Cherepanov]

Not only performance impact. IIUC with ASAN, afl-as reduces
instrumentation ratio theefold so you get much worse coverage for main
code, right? OTOH paths touching asan-specific branches could be
interesting too?

--
Alexander Cherepanov

[Hanno Böck]

On Fri, 7 Aug 2015 14:12:45 -0700
Konstantin Serebryany <konstantin....@gmail.com> wrote:

> On Fri, Aug 7, 2015 at 2:06 PM, Michal Zalewski <lca...@gmail.com>
> wrote:
>
> > > BTW is it useful to fuzz with afl-fuzz and UBSAN at all? Instead
> > > of generating a corpus and running it through UBSAN.
> >
> > ...maybe? It's hard to speak in absolutes. Theoretically, there can
> > be some advantages (same for ASAN). But whether they offset the
> > performance impact, hard to tell. I generally recommend the "fuzz
> > first, sanitize corpus later" approach.
>
> Quite the opposite to what I usually suggest :)

I tend to side with Kostya, at least that's what I do and suggest.
But seriously, this is something we can find out with some good
experiments. Take a couple of bugs afl found in the past that can only
be found with asan enabled, run afl on the same machine both with and
without asan. And see who hits first.
I'll probably try this once I find time for it.

--
Hanno Böck
http://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[Jakub Wilk]

* Tim Wilson-Brown <teor...@gmail.com>, 2015-08-07, 20:12:

>Under clang, the following arguments cause a trap or abort respectively on
>undefined behaviour.
>
>-fsanitize-undefined-trap-on-error

This option exists at least since clang 3.4, but seems broken in early versions:

$ clang -fsanitize=undefined -fsanitize-undefined-trap-on-error test.c
clang: error: invalid argument '-fsanitize=undefined' not allowed with '-fsanitize-undefined-trap-on-error'

But then -fsanitize-undefined-trap-on-error doesn't do anything without
-fsanitize=undefined...

It works correctly in clang >= 3.7 and in GCC >= 5.

Unfortunately, this option disables error messages. This doesn't matter for
fuzzing, but makes triaging UB crashes unnecessarily hard. :\

>-fno-sanitize-recover=all

This option works in clang >= 3.6 and in GCC >= 5.

But it makes the program exit with status 1, which is unhelpful for fuzzing. :(


The good news is that if your UBSan runtime is new enough[*], you don't need
any extra compiler options. Instead, you put this in environment:

UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1

Then you get both an error message and abort() on error.


[*] GCC >= 6.0 or LLVM >= 3.8, I believe.

--
Jakub Wilk

[Jakub Wilk]

If you're building stuff on x86 with clang+UBSan, you're likely to run into a
linker error similar to this:

foobar.c:(.text+0xf00b): undefined reference to `__mulodi4'

This bug is tracked here:
https://bugs.llvm.org/show_bug.cgi?id=17693
(scroll down for work-arounds)

--
Jakub Wilk