Blackbox binary instrumentation

[floyd]

Hi *

A small sad story about me trying to fuzz a closed source binary...

I wanted to fuzz a closed source binary on a network device (which was
x86, I was surprised by that). So I guess I had the following options
(let me know if I miss one):

a) compile AFL/qemu on network device
b) compile AFL/qemu on another box and copy to network device
c) use dyninst and copy AFL from another box (although stripped target
binary...)
d) copy the target binary to another box and run it there

I started with a) and I wasn't able to get a compile environment on the
network device (custom cli, not enough disc space, qemu complaining
about glibc version, etc. one big nightmare)

So I went for option b).

Small hint here: to compile qemu on Ubuntu 14.04 you need to change the
following line in qemu_mode/qemu-2.2.0/linux-user/syscall.c:
#include <linux/soundcard.h>
to
#include <linux/soundcard.h.oss3>

After compiling and copying AFL/qemu to the target box, dumb fuzzing
(afl-fuzz -n) works fine. When using -Q I get the "Fork server handshake
failed". The reason is probably that my other machine is not similar
enough (other libc):

$./afl-qemu-trace
./afl-qemu-trace: /lib/libc.so.6: version 'GLIBC_2.15' not found
(required by ./afl-qemu-trace)
./afl-qemu-trace: /lib/libc.so.6: version 'GLIBC_2.14' not found
(required by ./afl-qemu-trace)
./afl-qemu-trace: /lib/libc.so.6: version 'GLIBC_2.16' not found
(required by ./afl-qemu-trace)

I went back and tried to compile qemu statically, by adding --static to
the configure command for qemu in the build_qemu_support.sh script. Of
course, I got warnings such as:

warning: Using 'getaddrinfo' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking
Same for getpwuid, getpwnam_r, getpwuid_r. And of course the build fails:

/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libgcrypt.a(libgcrypt_la-misc.o):
In function `_gcry_fatal_error':
(.text+0x2df): undefined reference to `gpg_strerror'
[...]

So I got fed up and I went for option c). Which was another nightmare.
Getting dyninst installed is hard, it wouldn't compile because the new
cmake approach of the project throws one error after another. The
dyninst-dev.deb package they offer have dependency on (guess what) the
dyninst package, which you can't download. Using an old dyninst.deb
package works, but it won't install the header files, which afl-dyninst
needs, so afl-dyninst didn't compile...

So option d). Although the target binary works kind of when copied to
another box, it looks for daemons and other programs on the system that
it won't be able to connect to. So we are not fuzzing properly with this
approach. However, after compiling AFL with Qemu support I get:

$ ./afl-fuzz -i /opt/xxx-fuzzing/input/ -o /opt/xxx-fuzzing/output -Q
/opt/xxx-fuzzing/xxx -f @@
afl-fuzz 1.71b by <lca...@google.com>
[+] You have 1 CPU cores and 4 runnable tasks (utilization: 400%).
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '/opt/xxx-fuzzing/input/'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:test.txt'...
[*] Spinning up the fork server...
[+] All right - fork server is up.

[-] PROGRAM ABORT : Unable to communicate with fork server
Location : run_target(), afl-fuzz.c:2069

But for me it looks like qemu works fine:

$ ./afl-qemu-trace
usage: qemu-i386 [options] program [arguments...]
Linux CPU emulator (compiled for i386 emulation)

Options and associated environment variables:

Argument Env-variable Description
-h print this help
[...]

So, any hints/help on one of the cases, especially d)? Also if somebody
could elaborate on which version is necessary and what build environment
works for afl-dyninst would be helpful.

It would be so much fun to see the AFL results, because even python -c
"print 'A'*1000" as input results in a crash.

cheers,
floyd

[Zach Riggle]

Did you try qemu-user-static instead of qemu-user? That would solve
your libs problems.

> --
> You received this message because you are subscribed to the Google Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michal Zalewski]

You can use ./afl-showmap -o /dev/null -Q ./path/to/binary to see if
the binary will even run correctly under QEMU. QEMU user mode provides
a lightweight translation mode that works OK with typical user-space
programs, but may fail with something more exotic, so that would be a
good starting point.

/mz

[Andrew Griffiths]

On Fri, May 8, 2015 at 7:51 AM, floyd <fl...@floyd.ch> wrote:

So, any hints/help on one of the cases, especially d)? Also if somebody
could elaborate on which version is necessary and what build environment
works for afl-dyninst would be helpful.

Is the program running normally under afl-qemu-trace? 

To help debug, you might find qemu's -strace option useful for debugging and why it's failing.

Another option is to go down the route of copying the entire devices FS to your local computer, and copy static AFL binaries to that new fs, and chroot into it. That tends to work more reliably, except when the target binaries mess around with kernel module loading and other trickery.

[floyd]

On 08/05/15 16:59, 'Zach Riggle' via afl-users wrote:
> Did you try qemu-user-static instead of qemu-user? That would solve
> your libs problems.
>

What do you mean by that? I googled "qemu-user-static" and see a debian
package, but as we need to compile a patched version of qemu 2.20 for
AFL I don't know how that would help.

[floyd]

> You can use ./afl-showmap -o /dev/null -Q ./path/to/binary to see if
> the binary will even run correctly under QEMU. QEMU user mode provides
> a lightweight translation mode that works OK with typical user-space
> programs, but may fail with something more exotic, so that would be a
> good starting point.
>

That seems to work fine:

[+] Captured 2431 tuples in '/dev/null'.

[Turo Lamminen]

On Fri, 08 May 2015 16:51:17 +0200
floyd <fl...@floyd.ch> wrote:

> $ ./afl-fuzz -i /opt/xxx-fuzzing/input/ -o /opt/xxx-fuzzing/output -Q
> /opt/xxx-fuzzing/xxx -f @@

You need to manually specify memory limit. The default is too
restrictive for QEMU mode.

[floyd]

Thanks for the suggestion, but that's also not the case. Same when
starting with -m 1000 or even -m 2000.

Btw. in -Q mode afl-fuzz will increase memory limit to 200MB and it
should usually tell you in the error message if it's the memory problem

[Aleksandar Nikolich]

Which version of Dyninst did you try to compile? 

I had most success with their latest release tarball, namely 8.2.1. Not so much with the master branch from their git repo. On fedora 20, 21 and Ubuntu 14.10 though, so far.

Cheers,

Aleks 

[floyd]

On 08/05/15 17:12, 'Andrew Griffiths' via afl-users wrote:
>
> On Fri, May 8, 2015 at 7:51 AM, floyd <fl...@floyd.ch
> <mailto:fl...@floyd.ch>> wrote:
>
> So, any hints/help on one of the cases, especially d)? Also if somebody
> could elaborate on which version is necessary and what build environment
> works for afl-dyninst would be helpful.
>
>
> Is the program running normally under afl-qemu-trace?
>

Yes it does (as in running it with ./afl-qemu-trace
/path/to/target-binary), it also works in afl-showmap with -Q.

> To help debug, you might find qemu's -strace option useful for debugging
> and why it's failing.
>
> Another option is to go down the route of copying the entire devices FS
> to your local computer, and copy static AFL binaries to that new fs, and
> chroot into it. That tends to work more reliably, except when the target
> binaries mess around with kernel module loading and other trickery.
>

The chroot is a good option if nothing else works, thanks for the hint.

Hmm, maybe afl-fuzz has problems because the binary starts in
interactive mode (that's why the last line of the input file should
usually be "quit"). But that seems unlikely.

I just updated to afl-1.77b (and therefore qemu 2.3) just to be sure,
same problem there.

[Michal Zalewski]

The next step would be to try:

export AFL_NO_FORKSRV=1

...before running AFL.

/mz

[floyd]

So without the fork server it actually runs fine, although with
performance loss of no forkserver + qemu, which means less than 8 execs
per second

What's the best way to debug why the fork server fails? I can't share
the binary at the moment, I would need to do responsible disclosure first :(

btw. first time I use AFL for real pentest work, not just for breaking
stuff for fun

[Michal Zalewski]

> What's the best way to debug why the fork server fails? I can't share
> the binary at the moment, I would need to do responsible disclosure first :(

Running strace -f -o logfile.txt -s 1000 ./afl-fuzz ... and figuring
out what goes wrong :-/ Requires digging into the forkserver handshake
process a bit (although it's dead simple).

Most likely reasons:

1) The binary or one of the libraries it uses manages to create
threads before the forkserver initializes,

2) The binary, at least in some circumstances, exits in a way that
also kills the parent process - raise() could be the culprit,

/mz

[floyd]

Oh and I didn't mention yet that it's a SUID binary...

It seems to be the case that qemu gets a segmentation fault:

22847 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR,
si_addr=0x40aedb04} ---
22847 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
22847 prlimit64(0, RLIMIT_CORE, NULL, {rlim_cur=0, rlim_max=0}) = 0
22847 prlimit64(0, RLIMIT_CORE, NULL, {rlim_cur=0, rlim_max=0}) = 0
22847 prlimit64(0, RLIMIT_CORE, {rlim_cur=0, rlim_max=0}, NULL) = 0
22847 futex(0xb7426980, FUTEX_WAKE_PRIVATE, 2147483647) = 0
22847 write(2, "qemu: uncaught target signal 11 (Segmentation fault) -
core dumped\n", 67) = 67
22847 rt_sigaction(SIGSEGV, {SIG_DFL, ~[RTMIN RT_1], 0}, NULL, 8) = 0
22847 kill(22847, SIGSEGV) = 0
22847 --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_USER, si_pid=22847,
si_uid=1000} ---
22851 <... write resumed> ) = -1 EPIPE (Broken pipe)
22851 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=22851,
si_uid=1000} ---
22851 rt_sigreturn() = -1 EPIPE (Broken pipe)
[...]
22851 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=22851,
si_uid=1000} ---
22851 rt_sigreturn() = -1 EPIPE (Broken pipe)
22851 exit_group(0) = ?
22851 +++ exited with 0 +++
22848 +++ killed by SIGSEGV +++
22847 +++ killed by SIGSEGV +++
22846 <... read resumed> "", 4) = 0
22846 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=22847,
si_status=SIGSEGV, si_utime=2, si_stime=0} ---
22846 write(1, "\17\33)B\33[?25h\33[1;31m\n", 18) = 18
22846 write(1, "[-] PROGRAM ABORT : \33[1;37mUnable to communicate with
fork server\33[1;31m\n", 73) = 73
22846 write(1, " Location : \33[0mrun_target(),
afl-fuzz.c:2207\n\n", 55) = 55
22846 shmctl(42500106, IPC_64|IPC_RMID, 0) = 0
22846 write(6, "# unix_time, cycles_done, cur_path, paths_total,
pending_total, pending_favs, map_size, unique_crashes, unique_hangs,
max_depth, execs_per_sec\n", 143) = 143
22846 exit_group(1) = ?
22846 +++ exited with 1 +++


When I run it without strace, dmesg will tell me:

[462970.717959] Unsafe core_pattern used with suid_dumpable=2. Pipe
handler or fully qualified core dump path required.

[Michal Zalewski]

Hm, hard to say why QEMU dies without understanding what the process
does. One situation that could cause it is if the child process
dynamically loads a library with dlopen, or perhaps creates an
executable mapping and then jumps into it, and the parent tries to
pre-translate that block without having it mapped (a known limitation
of the QEMU approach).

/mz

[floyd]

I tried dyninst 8.2.1 again, and after some fixing and 2 hours of
compiling (yes, dyninst compiled for 2 hours with cmake), I have
afl-fuzz running with over 100 execs/second.

afl-dyninst actually didn't like it when I specified the -e address of
__libc_start_main (although it is a stripped binary), but when I removed
the -e option it instrumented just fine.

Good thing about this approach: I can copy it back to the network device
and it should work there as well

On 08/05/15 17:49, Aleksandar Nikolich wrote:
> Which version of Dyninst did you try to compile?
> I had most success with their latest release tarball, namely 8.2.1. Not
> so much with the master branch from their git repo. On fedora 20, 21 and
> Ubuntu 14.10 though, so far.
>
> Cheers,
> Aleks
>

> On Friday, May 8, 2015, floyd <fl...@floyd.ch <mailto:fl...@floyd.ch>>

> wrote:
>
> On 08/05/15 17:29, Turo Lamminen wrote:
> > On Fri, 08 May 2015 16:51:17 +0200
> > floyd <fl...@floyd.ch <javascript:;>> wrote:
> >
> >> $ ./afl-fuzz -i /opt/xxx-fuzzing/input/ -o /opt/xxx-fuzzing/output -Q
> >> /opt/xxx-fuzzing/xxx -f @@
> > You need to manually specify memory limit. The default is too
> > restrictive for QEMU mode.
> >
>
> Thanks for the suggestion, but that's also not the case. Same when
> starting with -m 1000 or even -m 2000.
>
> Btw. in -Q mode afl-fuzz will increase memory limit to 200MB and it
> should usually tell you in the error message if it's the memory problem
>
> --
> You received this message because you are subscribed to the Google
> Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to afl-users+...@googlegroups.com <javascript:;>.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to afl-users+...@googlegroups.com

> <mailto:afl-users+...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--
floyd
@floyd_ch
http://www.floyd.ch