network fuzzing against a client

[kittencrap]

Hey,

I've turned a binary client that does network/packet comms (using recvmsg) - into a binary that gets packets transparently from files by putting in a shim on getopt_long and recvmsg.

The getopt_long shim makes the target binary accept (and store) two extra parameters "--static" and "--fuzz".

The --static parameter (dir)  is used to specify a directory where static packets headers( msghdr, cmsghdr, return value, flags, etc) have been stored in binary files like '1.bin', '2.bin'...etc

The --fuzz parameter (file) is used to specify a file where the actual packet contents have been stored, all in one file (the iovec structures for each packet).

Packet contents, and packet headers are stored separately, because i have the intention to fuzz the former, and not the latter.

Leaving out afl-fuzz for the moment, if you run it as is, and no files exist - it creates files for each network packet it sees. 

If you run it again, and the files are actually present - the recvmsg hook will build msghdr/control/iov memory from the files, without actually going to the wire to get the contents, and return those to the caller (the target).

This works, and the target binary believes it is talking to the network, even though the packets have been loaded from files.

So, my question is, when running with afl-fuzz, rather than directly, it bails on me, with:

--8<--

$ LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -m 1024 -i fuzz/fuzz/ -o ./findings/ ./mybinary --static=./fuzz/static/ --fuzz=@@

afl-fuzz 1.76b by <lcamtuf@google.com>
[+] You have 1 CPU cores and 3 runnable tasks (utilization: 300%).
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning 'fuzz/fuzz/'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:wibble.cap'...
[*] Spinning up the fork server...
[-] Hmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. There are two probable explanations:
    - The current memory limit (1.00 GB) is too restrictive, causing an OOM
      fault in the dynamic linker. This can be fixed with the -m option. A
      simple way to confirm the diagnosis may be:
      ( ulimit -Sv $[1023 << 10]; /path/to/fuzzed_app )
      Tip: you can use http://jwilk.net/software/recidivm to quickly
      estimate the required amount of virtual memory for the binary.
    - Less likely, there is a horrible bug in the fuzzer. If other options
      fail, poke <lcamtuf@coredump.cx> for troubleshooting tips.
[-] PROGRAM ABORT : Fork server handshake failed
         Location : init_forkserver(), afl-fuzz.c:1919
--8<--

any thoughts?

cheers!

[Michal Zalewski]

There are three things you can try:

1) Run the target binary with the same LD_PRELOAD but via afl-showmap,
rather than afl-fuzz. This will give you better visibility into any
obvious errors generated by the binary.

2) If afl-showmap runs without errors, the next best option may be to
do strace -f -o trace.log -s 1000 and then see what happens to the
child process in trace.log.

3) If afl-showmap works, you can't figure out the root cause in #2,
and you don't care about performance, you can set AFL_NO_FORKSRV=1
before running afl-fuzz.

/mz

[Hanno Böck]

I wanted to wait till this "works a bit better", but as this idea is
now dropped here I think I can spoil this: I've been working on pretty
much the same.

LD_PRELOADing an so file that will intercept the TCP networking
functions and read packets from a file, controlling it by env vars.

This turned out to be trickier than first thought because you need to
a) make sure you get a reasonable networkishy behaviour
b) make sure at the same time functions that relate to networking and
file operations (fnctl, read, write etc.) to still beave properly when
they're not used on networking (varargs I have you).

I was unable to use it on curl or wget, but ssh worked. And right
away found a read overflow in ssh, which got fixed here:
https://anongit.mindrot.org/openssh.git/commit/?id=77199d6ec8986d470487e66f8ea8f4cf43d2e20c

(ssh devs think this is not exploitable so they fixed it in the public
repo, it may be worth checking that in detail)

Anyway: I'm a bit hesitant to release what I have, because really it is
almost-non-working-work-in-progress, but I hope I'll get it in a
working state and can release something.

--
Hanno Böck
http://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[James Kearney]

Mine is working without afl-fuzz (i.e. directly on the target binary), but not with afl-fuzz. However it seems to work with afl-showmap.

I don't use env-vars to control, i pass extra parameters that are caught by a getopt_long hook. 

That hook does however then use setenv to pass information to the recvmsg hook.

if i was targetting another binary that didn't use getopt_long, i'd need to find another way to do it like hooking getopt instead, or somewhere else to deal with the extra paramerers.

This has the added benefit that i can then use afl-fuzz @@ switch as intended.

since i can now build msghdr / cmsghdr / ioevec etc. from files - i assume i'll also be able to get this working with 'other' functions that use similar to hook those as well (generic receive for example).

I have one bug at the moment, in the instance that the 'real' recvmsg actually returns -1 (error), and captures that as one of it's example packets - then i store the same thing, and build a similar file to the memory (msghdr) that was present when recvmsg returned -1; however it seems to throw a bug where i am returning -1 from my hook, but the linux socket.c seg faults on parsing the iov_len (which in my error packet is 0). I am not actually sure how the structure is 'supposed' to be built in the instance that the call returns -1.

recvmsg is defined as "ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);". 

My hooked recvmsg returns the same ssize_t as when originally captured, the same msghdr struct (with same inner controlmsg / iovec), and the same flags. I've byte compared the original msghdr (from network) with the one i build from file, and that is exactly the same (as intended at the moment). 

So the only thing i can think is causing the bug is that there is a flag, or value, or something in the 'sockfd' that has changed that would normally indicate to the system that this particular received packet is an error (so therefore don't start trying to parse the msghdr/cmsghdr/iovec/etc).

anyway, once i've ironed out the remaining bugs, I am happy enough to share it (say in another few days) - so feel free to share yours as well.

--
You received this message because you are subscribed to a topic in the Google Groups "afl-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/afl-users/gahT13i4zG4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to afl-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cha...@ceriksen.com]

I don't really have a good answer per say. But are you not sort of looking to do something like Preeny does? Maybe you can gleam off a few tricks from it:

https://github.com/zardus/preeny

Here's a blog post demonstrating how it may be used with AFL and a network-based applicaton: https://lolware.net/2015/04/28/nginx-fuzzing.html

-Charlie

[kittencrap]

thanks Charlie, yep i've seen preeny (and that post), i wanted to do something slightly different, separate the packet header (msghdr/control header) from the packet data contents. 

Then use the packet contents as the 'fuzzable', and leave the headers (normally built by recvmsg) in place. (I also wanted my target to handle extra parameters without needing to change it's source).

I now have it working and fuzzing with afl-fuzz (without enabling AFL_NO_FORKSRV).

my test case is pretty simple, I am fuzzing dhcpcd. It sends 3 packets outgoing (requesting dhcp), and the usual response from my dhcp server has been written to files (3 files representing the 3 response packet headers, and 1 file with packet contents (for all responses)).

So, it stores 3 static files (1.file, 2.file, 3.file) for the network packet headers, and all of the packet contents are written to 'wibble.file'. 

Then i use 'wibble.file' as the fuzzable test case, a la

    $ LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -m 1024 -i fuzz/fuzz/ -o ./findings/ ./dhcpcd --static=./fuzz/static/ --fuzz=./fuzz/fuzz/@@ eth0

this (seems) to work (unless I'm just not seeing something less obvious).

I have on question, which is that when running, afl-fuzz is telling me "last new path : none yet (odd, check syntax)"

--8<--

                       american fuzzy lop 1.76b (dhcpcd)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐

│        run time : 0 days, 0 hrs, 7 min, 11 sec       │  cycles done : 89     │

│   last new path : none yet (odd, check syntax!) 

<snip>

--8<--

what does this warning mean, and might it be some indication that things are not working as expected?

thank you

[floyd]

Hi there

Maybe you want to try to fuzz a "normal" target first without using the
hooks? New paths are very important. This looks wrong:

On 19/05/15 09:55, kittencrap wrote:
> $ LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -m 1024 -i
> fuzz/fuzz/ -o ./findings/ ./dhcpcd --static=./fuzz/static/
> --fuzz=./fuzz/fuzz/@@ eth0

I would try:

LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -i fuzz/fuzz/ -o
./findings/ ./dhcpcd --static=./fuzz/static/ --fuzz=@@ eth0

Or try the -f option:

LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -i fuzz/fuzz/ -o
./findings/ -f ./fuzzfile ./dhcpcd --static=./fuzz/static/
--fuzz=./fuzzfile eth0

Your memory limit (-m) is probably too high, that's why I removed the -m
option. Increase it *slowly* if afl-fuzz complains.

>
> this (seems) to work (unless I'm just not seeing something less obvious).
>
> I have on question, which is that when running, afl-fuzz is telling me
> "last new path : none yet (odd, check syntax)"
> --8<--
> american fuzzy lop 1.76b (dhcpcd)
>
> ┌─ process timing ─────────────────────────────────────┬─ overall
> results ─────┐
> │ run time : 0 days, 0 hrs, 7 min, 11 sec │ cycles done :
> 89 │

> │ *last new path : none yet (odd, check syntax!) *

> <snip>
>
> --8<--
>
> what does this warning mean, and might it be some indication that things
> are not working as expected?

Absolutely, this is a very important criteria. It means your dhcpcd
binary is behaving exactly the same for every run (with every input
file). In other words: Your not finding new behavior, therefore your
fuzzing is not effective. The usual reason is that your setup is wrong
(see above). You need to fix this, otherwise you won't have crashes.

cheers,
floyd

>
>
> On Sunday, 17 May 2015 05:58:28 UTC+10, cha...@ceriksen.com wrote:
>
> I don't really have a good answer per say. But are you not sort of
> looking to do something like Preeny does? Maybe you can gleam off a
> few tricks from it:

> https://github.com/zardus/preeny <https://github.com/zardus/preeny>

> afl-fuzz 1.76bby<lca...@google.com>
> [+]Youhave 1CPU cores and3runnable tasks (utilization:300%).
> [*]Checkingcore_pattern...
> [*]Settingup output directories...
> [+]Outputdirectory exists but deemed OK to reuse.
> [*]Deletingold session data...
> [+]Outputdir cleanup successful.
> [*]Scanning'fuzz/fuzz/'...
> [+]Noauto-generated dictionary tokens to reuse.

> [*]Creatinghard links forall input files...
> [*]Validatingtarget binary...

> [*]Attemptingdry run with'id:000000,orig:wibble.cap'...

> [*]Spinningup the fork server...

> [-]Hmm,looks like the target binary terminated before we could

> complete a
> handshake withthe injected code.Thereare two probable
> explanations:
> -Thecurrent memory limit (1.00GB)istoo restrictive,causing
> an OOM
> fault inthe dynamiclinker.Thiscan be fixedwiththe -m option.A
> simple way to confirm the diagnosis may be:

> (ulimit -Sv$[1023<<10];/path/to/fuzzed_app )

> Tip:you can usehttp://jwilk.net/software/recidivm
> <http://jwilk.net/software/recidivm> to quickly
> estimate the required amount of virtualmemory forthe binary.
> -Lesslikely,there isa horrible bug inthe fuzzer.Ifother options

> fail,poke <lca...@coredump.cx>fortroubleshooting tips.
> [-]PROGRAM ABORT :Forkserver handshake failed
> Location:init_forkserver(),afl-fuzz.c:1919

> --8<--
> |
>
>
> any thoughts?
>
> cheers!
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "afl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to afl-users+...@googlegroups.com
> <mailto:afl-users+...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--
floyd
@floyd_ch
http://www.floyd.ch

[kittencrap]

hi floyd, thanks very much for that info, that's very useful.

I used your recommended like this:

LD_PRELOAD=./hook_recvmsg.so /root/fuzz/afl/afl-fuzz -t 2000 -i fuzz/fuzz/ -o ./findings2/ ./dhcpcd --static=./fuzz/static/ --fuzz=@@ eth0

(it gave me an error about:

    [-] The program took more than 1000 ms to process one of the initial test cases.

        This is bad news; raising the limit with the -t option is possible, but

        will probably make the fuzzing process extremely slow.

removing the -m option did not create a problem, but as you can see i've need to increas "-t".

It is now finding paths, new edges etc. (and 16 crashes) - i expect every single crash is in my "hook and get packet from file" code no doubt  :-)

i'll need to clean up the hook code (to avoid crashes), and once that's done i release it / the example of fuzzing dhcpcd.

thanks for the help / info

[James Kearney]

hey, 

would anyone have any thoughts on speed, or compiling my hook code with/without afl-gcc?

I've already had to increase the -t parameter (because it advised me to), currently running with -t 2000. 

That is with the network hooking code compiled /without/ afl-gcc (normal compile, "gcc -g -O0 -shared -fPIC hook_recvmsg.c -o hook_recvmsg.so -ldl -w -std=c99").

my afl-fuzz screen seems to be mocking me with some "zzzzs"!

thank you

--
James Kearney

--

You received this message because you are subscribed to a topic in the Google Groups "afl-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/afl-users/gahT13i4zG4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to afl-users+...@googlegroups.com.

[kittencrap]

I've reduced some of the crashes in my own network hook.

It seems the slow speed was due to the dhcpcd binary either forking to the background, or if i told it not to fork - it actually does not exit.

Running with "-T" dhcpcd test exercises less of the code, but exits out after printing the first packet, so I'm using that to just reduce some of the bugs in the hook level.

--8<--

                       american fuzzy lop 1.76b (dhcpcd)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐

│        run time : 0 days, 0 hrs, 27 min, 52 sec      │  cycles done : 0      │

│   last new path : 0 days, 0 hrs, 9 min, 26 sec       │  total paths : 35     │

│ last uniq crash : none seen yet                      │ uniq crashes : 0      │

│  last uniq hang : none seen yet                      │   uniq hangs : 0      │

├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤

│  now processing : 2 (5.71%)         │    map density : 1047 (1.60%)          │

│ paths timed out : 0 (0.00%)         │ count coverage : 1.19 bits/tuple       │

├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤

│  now trying : arith 32/8            │ favored paths : 21 (60.00%)            │

│ stage execs : 83.2k/109k (75.84%)   │  new edges on : 29 (82.86%)            │

│ total execs : 844k                  │ total crashes : 0 (0 unique)           │

│  exec speed : 565.8/sec             │   total hangs : 0 (0 unique)           │

├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤

│   bit flips : 8/24.4k, 1/24.4k, 1/24.4k             │    levels : 2          │

│  byte flips : 0/3056, 4/3054, 0/3050                │   pending : 34         │

│ arithmetics : 2/171k, 1/197k, 0/95.0k               │  pend fav : 21         │

│  known ints : 0/3434, 6/11.7k, 0/19.8k              │ own finds : 34         │

│  dictionary : 0/0, 0/0, 0/16.8k                     │  imported : n/a        │

│       havoc : 10/160k, 0/0                          │  variable : 35         │

│        trim : 0.00%/1494, 0.00%                     ├────────────────────────┘

└─────────────────────────────────────────────────────┘             [cpu:199%]

the recvmsg hook seems to work, however the above is running with -T switch, which means only one packet is sent and therefore only one is "received" (read from file).

this means i get less coverage, because i'm obviously now not fuzzing dhcp acknowledgment (packet 3) - however seems to be okay as a dry run.

however, with a hand from floyd it's at least running and testing the first incoming packet / finding paths.

thanks for the help

On Tuesday, 19 May 2015 19:08:45 UTC+10, kittencrap wrote:

hey, 

would anyone have any thoughts on speed, or compiling my hook code with/without afl-gcc?

I've already had to increase the -t parameter (because it advised me to), currently running with -t 2000. 

That is with the network hooking code compiled /without/ afl-gcc (normal compile, "gcc -g -O0 -shared -fPIC hook_recvmsg.c -o hook_recvmsg.so -ldl -w -std=c99").

my afl-fuzz screen seems to be mocking me with some "zzzzs"!

thank you

--
floyd
@floyd_ch
http://www.floyd.ch

--