Simple TCP Server fuzzing help

[Emka]

Hi all,

I wrote a very simple TCP server and want to fuzz it with AFL using approach described here: https://www.fastly.com/blog/how-fuzz-server-american-fuzzy-lop

What's wrong with the code below? What am I doing wrong? How can I correct it?

It hangs with:

SC_AFL_CMIN=1 /home/user/afl-2.01b/afl-cmin -i testcases/ -o testcases-cmin -- ./simple

corpus minimization tool for afl-fuzz by <lca...@google.com>

[*] Testing the target binary...

Also doesn't run/fuzz with:

ALF_PERSISTENT=1 /home/user/afl-2.01b/afl-fuzz -i testcases-cmin/ -o findings ./simple

afl-fuzz 2.01b by <lca...@google.com>

[+] You have 4 CPU cores and 2 runnable tasks (utilization: 50%).

[+] Try parallel jobs - see docs/parallel_fuzzing.txt.

[*] Checking core_pattern...

[*] Checking CPU scaling governor...

[*] Setting up output directories...

[+] Output directory exists but deemed OK to reuse.

[*] Deleting old session data...

[+] Output dir cleanup successful.

[*] Scanning 'testcases-cmin/'...

[+] No auto-generated dictionary tokens to reuse.

[*] Creating hard links for all input files...

[*] Validating target binary...

[*] Attempting dry run with 'id:000000,orig:in.txt'...

[*] Spinning up the fork server...

[+] All right - fork server is up.

[-] The program took more than 1000 ms to process one of the initial test cases.

    This is bad news; raising the limit with the -t option is possible, but

    will probably make the fuzzing process extremely slow.

    If this test case is just a fluke, the other option is to just avoid it

    altogether, and find one that is less of a CPU hog.

[-] PROGRAM ABORT : Test case 'id:000000,orig:in.txt' results in a hang

         Location : perform_dry_run(), afl-fuzz.c:2599

Thanks,

-----------

simple.c

-----------

/*
    C socket server example
*/
 
#include <stdio.h>
#include <string.h>    //strlen
#include <sys/socket.h>
#include <arpa/inet.h> //inet_addr
#include <unistd.h>    //write
#include <stdlib.h>
#include <signal.h>
 
int main(int argc , char *argv[])
{
    int socket_desc , client_sock , c , read_size;
    struct sockaddr_in server , client;
    char client_message[2000];
    
    //Create socket
    socket_desc = socket(AF_INET , SOCK_STREAM, 0);
    if (socket_desc == -1)
    {
        printf("Could not create socket");
    }
    puts("Socket created");
    
    //Prepare the sockaddr_in structure
    server.sin_family = AF_INET;
    server.sin_addr.s_addr = INADDR_ANY;
    server.sin_port = htons( 8888 );

    // set SO_REUSEADDR on a socket to true (1):
    int optval = 1;
    setsockopt(socket_desc, SOL_SOCKET, SO_REUSEADDR, &optval, sizeof optval);
    
    //Bind
    if( bind(socket_desc,(struct sockaddr *)&server , sizeof(server)) < 0)
    {
        //print the error message
        perror("bind failed. Error");
        return 1;
    }
    puts("bind done");
    
    //Listen
    listen(socket_desc , 3);
    
    //Accept and incoming connection
    puts("Waiting for incoming connections...");
    c = sizeof(struct sockaddr_in);
    
    //accept connection from an incoming client
    client_sock = accept(socket_desc, (struct sockaddr *)&client, (socklen_t*)&c);
    if (client_sock < 0)
    {
        perror("accept failed");
        return 1;
    }
    puts("Connection accepted");
    
    //Receive a message from client
    while( (read_size = recv(client_sock , client_message , 5000 , 0)) > 0 )
    {
#ifdef SC_AFL_PERSISTENT_SHIM

    size_t insize;
    char buf[5120];

    if (getenv("SC_AFL_STDIN") || getenv("SC_AFL_CMIN") || getenv("AFL_PERSISTENT")) {
            memset(buf, 0, 5120);
            insize = read(0, buf, 5120);

        //Send the message back to client
        write(client_sock , buf , strlen(buf));
        }
#endif

    write(client_sock , client_message , strlen(client_message));
    if(read_size>10)
    break;


#ifdef SC_AFL_PERSISTENT_SHIM  /* For AFL persistent mode fuzzing shim  */

        /* Signal AFL to fuzz input and continue execution */
        if (getenv("AFL_PERSISTENT")) {
            raise(SIGSTOP);
        } else if (getenv("SC_AFL_CMIN")) {
            exit(0);
        }

#endif
   

    }
    
    if(read_size == 0)
    {
        puts("Client disconnected");
        fflush(stdout);
    }
    else if(read_size == -1)
    {
        perror("recv failed");
    }
    close(socket_desc);
    shutdown(socket_desc,0);
    return 0;
}

[Michal Zalewski]

> What's wrong with the code below? What am I doing wrong? How can I correct
> it?

Your program just waits forever for incoming connections. AFL doesn't
know that and will not connect to the right socket to send the data.
The simplest modification is to make your daemon read its input from
stdin when given a particular command-line flag or so.

/mz

[Jacek Wielemborek]

W dniu 19.02.2016 o 18:12, Michal Zalewski pisze:

BTW, What's up with Doug's patch? [1] Is there any way it would get
merged and if not, what would have to be changed so that we could see it
in AFL's code?

[1]: https://groups.google.com/d/msg/afl-users/HxSb9Mv0dfQ/sPv91xEeDQAJ

[Michal Zalewski]

> BTW, What's up with Doug's patch? [1] Is there any way it would get
> merged and if not, what would have to be changed so that we could see it
> in AFL's code?

TBH... I worry a bit that upstreaming it as-is would not be very
compatible with the current design principle of AFL (which is to make
it reasonably difficult to shoot yourself in the foot).

Most network services use a fork-per-connect / worker pool model, so anyone who
tries to fuzz them using this code would be likely to get completely incorrect
results (in a way that is difficult for us to detect).

It would probably also need substantial work to better handle & diagnose
network latency and other issues (e.g., nothing listening on the target port,
target closing the connection before we sent all the data, target trying to
send us a response before further processing the request, us blocking while
trying to write to a non-responsive target, etc).

/mz

[Emka]

Thanks for all the answers.

Pitty that Doug's patch has these problems that Michal had mentioned.

Got it running in no time to fuzz my simple TCP server.

Michal, I and maybe others, would be interested to know more why the results could be incorrect? Can you elaborate more on this? Or point to some references? Examples?

I will try to test it also in my free time.


Thanks,
Marcin

[Emka]

Currently fuzzing "tinyproxy" with AFL 1.95b with Doug's patch.

Looks good :) Slow though .... can imagine it, since it's going through the network (localhost) and tinyproxy takes some time to start.

As I mentioned in my last post, don't understand how it can be incorrect. It looks like it is fuzzing it, I can imagine when "bad" request will be hit I get seg fault for further investigation and I can work from there.

I am very new to fuzzing and AFL, so please excuse if I write nonsense.

Thanks,
Marcin

[Doug Birdwell]

Emka,

A quick suggestion: If you have not already done so, with my networking extension use trial & error (a.k.a. binary search) to find a minimal delay value that avoids all but a small fraction of timeouts in order to maximize target execution rate (and possibly parallelize on a multi-core host using Docker - see below).  I saw target execution rates between around 100/second (for a very complex server, due to start-up processing in the server) and around 500-600/second (for simple network targets) using a Linux VM running under Mac OS X on a 3 GHz processor, and somewhat slower rates with Linux on 2.3 GHz hardware (no VM); there was little to no performance degradation running in a Docker container.  I think the upper limit for my configurations was less than 1000 executions per second and was probably due to the network stack (combined with fork server processing and the existing overhead) -- which is the reason the delay parameter on the command line is in milliseconds.  Typical delay parameter values were less than 10 milliseconds.  I hope this helps.

Emka & Michal & others,

Two things about fuzzing across a network: it's not particularly easy, and it is probably always going to be slower than using stdin.  It's not particularly easy because modifications to the target are usually required, and those typically require some reverse engineering.  Yes, among other things one needs to disable forks, and one must find a way to gracefully shut down the target after the first packet has been (completely) processed - otherwise, timeouts are generated.  For network daemons that deploy thread (or process) pools and don't implement a way to turn this off, the reverse engineering effort can be quite large.  I suspect typical AFL users are fairly advanced and anticipate these issues ;-).  It is typically slower because of start-up processing for network transactions -- and one can make changes to the target's code and use persistence (intercepting the fork much nearer to the network read) to get very large performance improvements when reading from stdin - something very difficult (or impossible) for a code using the network.  However, it's also easy to make a mistake and end up with a target that performs differently than the network-only version.

Michal, your comment: "Most network services use a fork-per-connect / worker pool model, so anyone who tries to fuzz them using this code would be likely to get completely incorrect results (in a way that is difficult for us to detect)" is a bit off the mark.  I obviously did not get incorrect results when I fuzzed ntpd and found a vulnerability, and I have fuzzed other widely used network daemons.  Others have also used the code; see, for example, Hanno's post at https://blog.fuzzing-project.org/27-Network-fuzzing-with-american-fuzzy-lop.html.  If a user does not disable the target's forks, the network version of AFL typically finds ... nothing.  More likely, if a user has not made any modifications to the typical network daemon (using a command line argument for debugging can be all that is necessary), all of the initial runs of the target that AFL performs will time out, and AFL will exit.  That's a pretty clear indication that the user needs to look more closely at the target. Give your user base some credit -- they are not average users.

I would like to see my networking code incorporated into the "official" AFL distribution.  It doesn't break what's there (or didn't as of the last time I merged it with AFL 1.95b - and yes, when I have some spare time I will update to the latest AFL version), and the networking code isn't used unless it is enabled with the appropriate switches.  Yes, it's a fairly large chunk of code to review (about 1K lines) -- but I believe it would be worth the effort.  I'm fine if you label it "experimental" - which it is, and it's not the only example of "experimental" features in AFL.  The point is to make it easy for people to access these features and generate feedback (and results).

One could add more elaborate tests to attempt to ensure that AFL/network is actually talking to the target, but my suspicion is that is likely to be difficult (although TCP often tells you when something doesn't work).   I believe it is better for a human to check this, primarily because there are so many variations in the way people write (potential) targets' networking codes, and in start-up processing and times -- and, for UDP, the sending process can't be certain that the packet arrives at the target.  A person can incorporate checks into the target's code to ensure this occurs, and if necessary to help in tuning the networking code's delay parameter.  One thing that greatly helps is computers' consistency -- once a delay value is found that works (typically easy by trial & error), it will almost always work.  I view the approach I used (a fixed delay with multiple attempts) an acceptable trade-off that works for many targets and is not overly complex (e.g., does not add significantly more than necessary to execution times).

It is important to be able to fuzz a target across a network, if only to verify the behavior is identical to fuzzing using stdin and a modified target.  For example, I corresponded with a person who was attempting to fuzz ntpd (the vulnerable version) using a modification to accept inputs across stdin, and who was not able to reproduce the flaw I found.  Perhaps this was just due to the stochastic nature of fuzzing, but I suspect not (and if you are reading this, yes, I still need to get back to you -- I'm sorry, but I forgot until I started writing this post!).

Fuzzing targets via a network is a work in progress, not because it is especially difficult or slow, but in part because it is more difficult to parallelize than fuzzing using stdin or a file system.  I provided one solution to this using Docker containers (to isolate network ports so they could be reused on the same host) and have made that code (and the containers) available.  There are other ways, and I believe it will require more experience to determine the best approaches.  In particular, I believe it will become important to be able to execute fuzzing runs on large clusters of hosts, both for normal and networked fuzzing.  At present, this can be done (with some pain) for normal fuzzing, but is more difficult for networked fuzzing.

I will continue to provide merged versions of my networking code and AFL releases as I have time, as well as Docker containers.  I am always glad to hear from people who have used the network code and can provide feedback.  The source codes (AFL with networking support and the Dockerfile to build a container) are available on my GitHub account: https://github.com/jdbirdwell .  The container can be obtained using the Docker command "docker pull birdwell/debian" and using the tag afl-1.95b-net.  I have not yet updated to more recent releases of AFL.

Doug

[Michal Zalewski]

> Give your user base some credit -- they are not average users.

For better or worse, there is a tremendous difference between shipping
something that the author or several power users very familiar with
the tool will know how to employ, and something that sort of works in
a plug-and-play mode for a person who has no experience with the tool
or the underlying concepts and *at best* skimmed through README.

In fact, features that do have esoteric failure modes or require
expert knowledge can be actually damaging to less experienced users,
because when given too many options, they can make suboptimal choices.
Generally, unless we come up with some brilliant turn-key solution, in
90%+ of all cases, the better route is to make a service read from
stdin / in "inetd mode" than to keep the network code intact, but then
try to modify the process / thread model or the main event loop. That
second part is usually the more complex step anyway.

Now, there is definitely a tension between catering to power users and
to the general public, but AFL firmly emphasizes the latter. That's
why you don't have 500 configurable settings, pluggable mutation
algorithms, etc. Its predecessor (Bunny the Fuzzer) had TCP and UDP
output from the get go, along with something like 30 other
command-line settings.

For now, my worry is that the patch requires a ton of work to make it
robust (e.g., better handle some blocking conditions, short writes,
etc), and I'm not sure I will have time to work on this any time soon;
by the looks of it, with this patch, there are many circumstances in
which an erratic network client would completely stall AFL :-( Perhaps
more importantly, with that ton of work invested, I worry that we'd be
giving a typical user (many of whom *are* newcomers to the tool or
newcomers to fuzzing in general) a mechanism that is still fairly
confusing and difficult to use.

I suspect that Preeny-style approaches (i.e., LD_PRELOAD to simulate
network I/O from a file and to alter the behavior of fork() and the
likes) may offer a more complete solution. Unfortunately, Preeny isn't
particularly AFL-friendly as-is :-(

/mz

[Emka]

@Doug

Managed to get up to 100 reqs/sec on my laptop (while fuzzing pdnsd and atftpd daemons)

@all

With that speed ... how long a full fuzz can take? What are the key statistics, numbers I have to observe to gauge it? Is it described somewhere?

Thanks,

[Michal Zalewski]

> With that speed ... how long a full fuzz can take? What are the key
> statistics, numbers I have to observe to gauge it? Is it described
> somewhere?

Depends, but see docs/status_screen.txt for some tips.

Cheers,
/mz

[zar...@gmail.com]

Hello,

I'm the author of preeny (I know, hard to believe! ;-) ), and a colleague of mine forwarded this thread to me.

I suspect that Preeny-style approaches (i.e., LD_PRELOAD to simulate
network I/O from a file and to alter the behavior of fork() and the
likes) may offer a more complete solution. Unfortunately, Preeny isn't
particularly AFL-friendly as-is :-(

This caught my eye, as I'm currently tearing apart the desock module of preeny to track down a bug that crept in recently. Do you have any suggestions for making preeny more AFL-friendly? I'm not arguing that it's not friendly; I legitimately want to make it more useful, and might as well do it while I'm working on it anyways.

- Yan

[Michal Zalewski]

> I'm the author of preeny (I know, hard to believe! ;-) ), and a colleague of mine forwarded this thread to me.

Yello :-)

Maybe "not friendly" is too strong of a word, but we had a bunch of
people running into hard-to-diagnose problems with Preeny, e.g.:

https://groups.google.com/forum/#!topic/afl-users/covb8S_bRNw

https://groups.google.com/forum/#!msg/afl-users/VRWt0Fs99rA/Qjd-b7cUBAAJ

Others have gotten better results, albeit still needed to make some changes:

https://lolware.net/2015/04/28/nginx-fuzzing.html

I've never tried to fully diagnose them, but looking at strace output
and the likes, I suspect the issues they bump into were on Preeny
side.

/mz

[Yan]

Ah yes, the poll() loop from that first issue is what I'm tackling right now. I made a decision early on in preeny's development to try to provide replacement sockets that are as realistic as possible, which forced me into using socketpairs and synchronizing them with background threads. This is a huge pain, and causes some unfortunate issues and race conditions.

After I track this down, I'm thinking of writing a desock-lite module for preeny that overrides recv(), send(), and friends. But that one will have to override almost all socket functions (setsockopt, etc), which is why I was reluctant to go that route. All that being said, I think it'll end up being a lot less of a headache to use in the end. Just need to find some time to do it...

- Yan

[Brandon Perry]

On Feb 23, 2016, at 1:54 PM, Yan <zar...@gmail.com> wrote:

Ah yes, the poll() loop from that first issue is what I'm tackling right now. I made a decision early on in preeny's development to try to provide replacement sockets that are as realistic as possible, which forced me into using socketpairs and synchronizing them with background threads. This is a huge pain, and causes some unfortunate issues and race conditions.

After I track this down, I'm thinking of writing a desock-lite module for preeny that overrides recv(), send(), and friends. But that one will have to override almost all socket functions (setsockopt, etc), which is why I was reluctant to go that route. All that being said, I think it'll end up being a lot less of a headache to use in the end. Just need to find some time to do it...

I believe I also ran into the same issue while fuzzing synergy. Would the best way to know when a potential fix is available be to simply watch github? I could try retesting a few of my projects when available (but no rush at all).

- Yan

On Tue, Feb 23, 2016 at 11:15 AM, Michal Zalewski <lca...@gmail.com> wrote:

> I'm the author of preeny (I know, hard to believe! ;-) ), and a colleague of mine forwarded this thread to me.

Yello :-)

Maybe "not friendly" is too strong of a word, but we had a bunch of
people running into hard-to-diagnose problems with Preeny, e.g.:

https://groups.google.com/forum/#!topic/afl-users/covb8S_bRNw

https://groups.google.com/forum/#!msg/afl-users/VRWt0Fs99rA/Qjd-b7cUBAAJ

Others have gotten better results, albeit still needed to make some changes:

https://lolware.net/2015/04/28/nginx-fuzzing.html

I've never tried to fully diagnose them, but looking at strace output
and the likes, I suspect the issues they bump into were on Preeny
side.

/mz

--
You received this message because you are subscribed to the Google Groups "afl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jacek Wielemborek]

W dniu 23.02.2016 o 20:54, Yan pisze:

> After I track this down, I'm thinking of writing a desock-lite module for
> preeny that overrides recv(), send(), and friends. But that one will have
> to override almost all socket functions (setsockopt, etc), which is why I
> was reluctant to go that route. All that being said, I think it'll end up
> being a lot less of a headache to use in the end. Just need to find some
> time to do it...
>
> - Yan

I probably already said it over Github, but +1 for this one. The
threaded model sometimes freezed and it's not really convenient to use -
desock-lite sounds better. I wonder how little work would it take to
make it "useful enough"? Perhaps there's some API test suite that we
could use to get the basic functionality done quickly? I was once trying
to write this on my own but it kind of overwhelmed me.

[Hanno Böck]

On Tue, 23 Feb 2016 11:54:03 -0800
Yan <zar...@gmail.com> wrote:

> After I track this down, I'm thinking of writing a desock-lite module
> for preeny that overrides recv(), send(), and friends. But that one
> will have to override almost all socket functions (setsockopt, etc),
> which is why I was reluctant to go that route. All that being said, I
> think it'll end up being a lot less of a headache to use in the end.
> Just need to find some time to do it...

Actually I have once written something with this approach. Given the
other approaches I never brought it into a release-worthy state, but as
there seems to be interest I probably should just make the code public.

It didn't work very well, but it found one bug [1]. (but actually
openssh was the only piece of software I was ever able to successfully
use that code)


[1]
https://blog.fuzzing-project.org/18-Out-of-bounds-reads-in-OpenSSH-and-some-lessons-learned-about-vulnerability-terminology.html

--
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: BBB51E42

[Yan]

Hanno and Jacek,

If you guys don't mind donating that code, could you send me a PR to preeny? Then we can integrate and expand it. MIght be quicker than doing it from scratch :-)

- Yan

--
You received this message because you are subscribed to a topic in the Google Groups "afl-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/afl-users/fpZIcdNBMNM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to afl-users+...@googlegroups.com.