Fuzzing an integer-input program (newbie question)

[zhoul...@gmail.com]

Hi all,

I am trying AFL for the first time. I test on a trivial C program that takes input from the command-line:   "./a.out n" prints the integer "n" if n>=0,  and crashes if n<0.   My goal is to use AFL to trigger the crash (i.e. automatically generate an negative input).

So far,  I have compiled the C program into a.out using afl-gcc (version 2.45b).  Then I got stuck because I do not see the exact syntax for constructing the dictionary? (Apparently a dictionary is needed before afl-fuzzying, if I understand correctly.) Can anyone help? Thanks.

Zhoulai

[Michal Zalewski]

> I am trying AFL for the first time. I test on a trivial C program that takes
> input from the command-line: "./a.out n" prints the integer "n" if n>=0,
> and crashes if n<0. My goal is to use AFL to trigger the crash (i.e.
> automatically generate an negative input).

AFL does not natively support fuzzing the parameters of a program. You
can write your program to read the integer from stdin or from a file.

Alternatively, you could use experimental/argv_fuzzing/, but I do not
recommend that unless you have a really good reason to =)

> So far, I have compiled the C program into a.out using afl-gcc (version
> 2.45b). Then I got stuck because I do not see the exact syntax for
> constructing the dictionary? (Apparently a dictionary is needed before
> afl-fuzzying, if I understand correctly.)

You don't need a dictionary.

/mz

[Zhoulai Fu]

Thanks for the quick supply.  My earlier understanding from QuickStartGuide.txt is that the dictionary is for creating valid inputs and the mutated ones. So,  would you clarify under what circumstances one needs/does not need a dictionary? Thanks.

Zhoulai 

[Michal Zalewski]

> Thanks for the quick supply. My earlier understanding from
> QuickStartGuide.txt is that the dictionary is for creating valid inputs and
> the mutated ones.

Huh? I don't think it says that... it just states:

"When fuzzing verbose syntax (SQL, HTTP, etc), create a dictionary as
described in dictionaries/README.dictionaries, too."

This is a bit terse, but basically, if you're fuzzing a program that
reads an integer from stdin, don't worry about it.

/mz

[Zhoulai Fu]

Thanks a lot for the help. Now I can run afl-fuzz on a simple C program. 

Is that possible to have the coverage results, e.g., branch or line coverage, after running afl-fuzz for a period of time?  Thanks.

[Michal Zalewski]

> Is that possible to have the coverage results, e.g., branch or line
> coverage, after running afl-fuzz for a period of time? Thanks.

Yep, with a third-party tool, afl-cov.

/mz

[Zhoulai Fu]

Hi again,

I got a strange error message when testing a simple program: 

... The current memory limit (50.0 MB) is too low for this program ...

I find the message strange because README.txt of afl-fuzz says

You can use -t and -m to override the default timeout and memory limit for the
executed process; rare examples of targets that may need these settings touched
include compilers and video decoders.

 

"

My tested C program, test.c, that triggers the issue above is quite simple:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int main(int argc, char *argv[])
{

  int val;
  scanf("%d", &val);
  if (val < 0){
    assert(1==2);
    return 1;
  }
  else{
    printf("%d\n",val);
     assert(4==5);
    return 0;
  }
}

Strangely, if I remove the "assert (4==5)" line, I can run afl-fuzz again with no memory issues.  My command line is 

cd Mytest; ../afl-gcc test.c; cd ..;
./afl-fuzz  -i Mytest -o Mytest_o -- Mytest/a.out

Any idea why that memory issue? Thanks.

[Tim Newsham]

On Tuesday, October 11, 2016 at 7:53:12 AM UTC-10, Zhoulai Fu wrote:

Hi again,

I got a strange error message when testing a simple program: 

... The current memory limit (50.0 MB) is too low for this program ...

You only read a small part of the error message, which was:

"""

[-] Oops, the program crashed with one of the test cases provided. There are

    several possible explanations:

    - The test case causes known crashes under normal working conditions. If

      so, please remove it. The fuzzer should be seeded with interesting

      inputs - but not ones that cause an outright crash.

    - The current memory limit (50.0 MB) is too low for this program, causing

      it to die due to OOM when parsing valid files. To fix this, try

      bumping it up with the -m setting in the command line. If in doubt,

      try something along the lines of:

      ( ulimit -Sv $[49 << 10]; /path/to/binary [...] <testcase )

      Tip: you can use http://jwilk.net/software/recidivm to quickly

      estimate the required amount of virtual memory for the binary. Also,

      if you are using ASAN, see /usr/local/share/doc/afl/notes_for_asan.txt.

    - Least likely, there is a horrible bug in the fuzzer. If other options

      fail, poke <lca...@coredump.cx> for troubleshooting tips.

"""

AFL is trying to run your program, and it is crashing, even for normal inputs.  Once reason this can happen is because memory limits.  But in your case it is because you have asserts that can never be true, causing the program to abort() and crash.  Fixing these will get rid of your error message.

[Zhoulai Fu]

Thanks!  My terminal was small and I missed a part of error message... Sorry for that question. 

Now I removed all  assertions that were set to trigger crashes,  afl-fuzz works well but may not give unsatisfactory code coverage.  The tested program is here ("#include ..." omitted): 

int main(int argc, char *argv[])
{
  int val;
  scanf("%d", &val);
  if (val < 0){

    return 0;

  }
  else {
    printf("%d\n",val);

    return 1;
  }
}

This program can be fully covered as long as the generated input set includes a positive and a negative numbers. However,  after running afl-fuzz for half an hour, with default settings,  I find the coverage results reported by  afl-cov is lower than what I expected (see below the results reported by afl-cov).  In particular, only one of the two branches is covered with afl-fuzz, whereas even pure random testing would  quickly cover both branches. So I am confused why afl-fuzz would not produce better results than pure random testing for this example program?  Thanks in advance!

 [+] AFL test case: id:000001,orig:test.c (1 / 3), cycle: 0
        lines......: 83.3% (5 of 6 lines)
        functions..: 100.0% (1 of 1 function)
        branches...: 50.0% (1 of 2 branches)
    [+] AFL test case: id:000002,src:000001,op:flip1,pos:0,+cov (2 / 3), cycle: 0
        lines......: 83.3% (5 of 6 lines)
        functions..: 100.0% (1 of 1 function)
        branches...: 50.0% (1 of 2 branches)
    [+] Processed 3 / 3 test cases.

    [+] Final zero coverage report: ../Mytest_o/cov/zero-cov
    [+] Final positive coverage report: ../Mytest_o/cov/pos-cov
        lines......: 83.3% (5 of 6 lines)
        functions..: 100.0% (1 of 1 function)
        branches...: 50.0% (1 of 2 branches)
    [+] Final lcov web report: ../Mytest_o/cov/web/index.html

--Zhoulai

[Michal Zalewski]

> So I am confused why afl-fuzz would not produce better
> results than pure random testing for this example program? Thanks in
> advance!

Looks like AFL generated several distinct test cases, so I'm guessing
you're hitting all the code there is in that program. Why afl-cov is
reporting 50%, I don't know - maybe it's counting some branches from
inlined #includes or something? That's probably a question for the
author of that tool.

/mz

[Zhoulai Fu]

Thanks for your reply.  However, following the lcov report I just got (attached), we see 

1. lcov does not trace  the included file for counting its coverages, and
   
2. All 3 test inputs of afl-fuzz passes "cmdval>=0" branch; the branch "cmdval<0" is never visited.
   

Thus afl-fuzz never generates negative integers in this example. Can it a bug? For your convenience, I also put the full program under test below (20 lines, 2 branches):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int main(int argc, char *argv[])
{

  int cmdval;
  scanf("%d", &cmdval);
  if (cmdval < 0){
    //assert(1==2);
    return 1;
  }
  else {
    printf("%d\n",cmdval);
    // assert(4==5);
    return 1;
  }
}

Thanks.

-Zhoulai
 
 

[Michal Zalewski]

> lcov does not trace the included file for counting its coverages, and
> All 3 test inputs of afl-fuzz passes "cmdval>=0" branch; the branch "cmdval<0" is never visited.

Have you looked at the generated test cases in the queue/
subdirectory? You have several; what's in them?

/mz

[Zhoulai Fu]

Have you looked at the generated test cases in the queue/
subdirectory? You have several; what's in them?

 

I got three files in queue/, which should correspond to the three test inputs:

1. id:000000,orig:a.out
2. id:000001,orig:test.c
3. id:000002,src:000001,op:flip1,po

What could I do to read out the generated integers? Thanks.

Zhoulai

[Brandon Perry]

for i in queue/; do cat $i | xxd; done

This will print out the hexadecimal bytes of the contents of the inputs. You can see what integers were used with that.

--
You received this message because you are subscribed to the Google Groups "afl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michal Zalewski]

> id:000000,orig:a.out
> id:000001,orig:test.c

I'm at loss as to what you're trying to accomplish here... the input
directory should contain test cases, not the source code or the
compiled binary of your target program.

> id:000002,src:000001,op:flip1,po
> What could I do to read out the generated integers? Thanks.

'cat' would be a good starting point (or Brandon's suggestion to dump hex data).

/mz

[Zhoulai Fu]

On Tuesday, October 11, 2016 at 4:31:17 PM UTC-7, Brandon Perry wrote:

for i in queue/; do cat $i | xxd; done

Thanks. Using the magical line above,  I get the three test inputs as follows:

0000000: 7f45 4c46                                .ELF
0000000: 2369 6e63                                #inc
0000000: 3369 6e63                                3inc

Then, I use https://www.mathsisfun.com/binary-decimal-hexadecimal-converter.html to convert the numbers to  32-bit signed integer:

7f45 4c46  -> 2135247942

2369 6e63 -> 594112099

3369 6e63 -> 862547555

All three are positive, which seems to confirm that afl-fuzz failed to  generate negative integers? But maybe I should have used a different configurations; I used the default one. Can someone try out my example program? Thanks.

Zhoulai
 

[Michal Zalewski]

> 0000000: 7f45 4c46 .ELF
> 0000000: 2369 6e63 #inc

These are garbage, as discussed earlier. They cause scanf() to fail,
leave cmdval uninitialized, and I'm betting that cmdval ends up being
negative and causes your program to take the negative branch. Since
this outcome would be somewhat non-deterministic, what's seen by AFL
may be not reproducible when trying later with third-party code
coverage tools.

Starting with valid inputs would be far more informative.

> 0000000: 3369 6e63 3inc

This is a valid test case that triggers the positive branch.

> https://www.mathsisfun.com/binary-decimal-hexadecimal-converter.html to
> convert the numbers to 32-bit signed integer:
> 7f45 4c46 -> 2135247942
> 2369 6e63 -> 594112099
> 3369 6e63 -> 862547555

You're doing scanf("%d"). That just expects a human-readable decimal
ASCII representation of a signed integer. In that context, this
conversion is meaningless.

/mz

[Zhoulai Fu]

> 7f45 4c46  -> 2135247942
> 2369 6e63 -> 594112099
> 3369 6e63 -> 862547555

You're doing scanf("%d"). That just expects a human-readable decimal
ASCII representation of a signed integer. In that context, this
conversion is meaningless.

There should be a misunderstanding. The three hex numbers are test inputs retrieved with Brendon's command line. I was suggested to see what integer inputs afl-fuzz generates exactly.  They have nothing to do with scanf ("%d"). 

Can someone try out my test program? It is only 20 lines. My finding is that afl-fuzz does not generate a single negative integer for 30 minutes. But I am unsure whether I should use  configurations other than the default ones.

Z.

Zhoulai

[Brandon Perry]

On Oct 11, 2016, at 7:04 PM, Zhoulai Fu <zhoul...@gmail.com> wrote:

> 7f45 4c46  -> 2135247942
> 2369 6e63 -> 594112099
> 3369 6e63 -> 862547555

You're doing scanf("%d"). That just expects a human-readable decimal
ASCII representation of a signed integer. In that context, this
conversion is meaningless.

There should be a misunderstanding. The three hex numbers are test inputs retrieved with Brendon's command line. I was suggested to see what integer inputs afl-fuzz generates exactly.  They have nothing to do with scanf ("%d"). 

Ignore me, I didn’t read your code. I just read it was trying to read integers and assumed binary.

Can someone try out my test program? It is only 20 lines. My finding is that afl-fuzz does not generate a single negative integer for 30 minutes. But I am unsure whether I should use  configurations other than the default ones.

Z.

Zhoulai

[Michal Zalewski]

> Can someone try out my test program? It is only 20 lines. My finding is that
> afl-fuzz does not generate a single negative integer for 30 minutes. But I
> am unsure whether I should use configurations other than the default ones.

As mentioned, I'm pretty sure that AFL is working correctly. Your
program is using uninitialized variables and is being given garbage
inputs, producing some non-deterministic behavior when you're trying
to measure coverage later on, but that's not a bug in AFL.

/mz

[Sean Burford]

Hi,

I suggest fixing up your undefined behaviours and then doing your science:

$ printf '\x7fELF' | ./test

0

$ printf '#inc' | ./test

0

$ printf '3inc' | ./test

3

$ ./test 0>-5

failed to interpret value

$ ./test

-1

-1

test: test.c:13: main: Assertion `val >= 0' failed.

Aborted (core dumped)

$ cat test.c

#include <assert.h>

#include <errno.h>

#include <stdio.h>

int main(int argc, char *argv[])

{

  int val = 0;

  scanf("%d", &val);

  if (errno) {

    printf("failed to interpret value\n");

  } else {

    printf("%d\n", val);

    assert (val >= 0);

  }

  return 0;

}

Thanks,

Sean

[Zhoulai Fu]

As mentioned, I'm pretty sure that AFL is working correctly. Your
program is using uninitialized variables and is being given garbage
inputs, producing some non-deterministic behavior when you're trying
to measure coverage later on, but that's not a bug in AFL.

Oh, my apologies for the misunderstanding!   Excuse me but I still have difficulties in understanding where the program is not well-defined from the AFL perspective. My program takes an integer input "cmdval" from stdin, and returns either 0 or 1 according to the input value, deterministically.  Would you propose a slightly modified version of my program so to make that one acceptable by AFL? Thanks for your patience and help. For your convenience, I put the original program below (slightly simplified). 

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int main(int argc, char *argv[])
{

  int cmdval;
  scanf("%d", &cmdval);
  if (cmdval < 0)

    return 0; 
  else
    return 1;

}

[Michal Zalewski]

> Oh, my apologies for the misunderstanding! Excuse me but I still have
> difficulties in understanding where the program is not well-defined from the
> AFL perspective.

cmdval is uninitialized. It may end up having a negative or non-negative
value depending on fairly unpredictable factors.

I suggest making two changes:

1) Changing this line:

scanf("%d", &cmdval);

...to:

if (scanf("%d", &cmdval) != 1) return 2;

This will effectively add a second branch to the program, but it
removes the non-deterministic behavior and will give you nice test
cases.

2) Seeding the fuzzer with a single, reasonable input file:

mkdir in_dir
echo 0 >in_dir/testcases
./afl-fuzz -i in_dir [...other params...]

I bet that this will quickly result in three files in the queue:

- One with a non-negative string representation of an integer
(possibly with some trailing garbage, which is ignored by your
program),

- One with a negative representation (ditto),

- One with something that does not parse as an integer.

...and if you then attempt to measure coverage in the corpus, you will
see that all branches are hit.

[IOhannes m zmölnig]

On 2016-10-12 02:04, Zhoulai Fu wrote:
> that afl-fuzz does not generate a single negative integer for 30 minutes.

30 minutes don't seem to be very much.

fgmasdr
IOhannes

[Zhoulai Fu]

Thank you so much! The strange results I got
earlier have disappeared thanks to your remark #2: I needed to use
reasonable initial test cases; the undefined behavior (#1) turns out  to be an
orthogonal issue.
 
Z.