AFL/AFL++ and Multithreaded Programs

[Connor Shugg]

Greetings, oh wise and helpful members of the AFL users group,

Thank you to those who have answered my previous question(s). I've become an avid user of both AFL and AFL++ and have come to love fuzzing. I had a question regarding multithreaded target programs.

Based on previous conversations I've found here regarding multithreading, the consensus seems to be that AFL does support the fuzzing of multithreaded targets.

From what I can tell, fuzzing a program that spawns some number of threads would probably decrease the fuzzer's "stability" score, considering pthread scheduling isn't necessarily deterministic. But my main question(s) are:

• Despite a stability decrease, is it still feasible to fuzz multithreaded targets?
  • As an end goal: is it feasible to try to have AFL discover a deadlock situation? (That would be awesome.)
• How does having multiple threads (all running instrumented code) affect AFL's shared memory region used to track target behavior?
• Is AFL's instrumentation thread-safe? Or is it not - meaning multiple threads could be writing to the same memory region simultaneously.

Hopefully these questions are clear. If anyone has any thoughts on the matter, I would greatly appreciate any insight you could provide.

Have a great weekend!

[Marc]

Hi Connor,

On 11.09.21 03:31, Connor Shugg wrote:
> Based on previous conversations I've found here regarding

> multithreading, the consensus seems to be that AFL /does /support the
> fuzzing of multithreaded targets.

I am not sure what you mean by "support" - but sure you can instrument
and fuzz them :)

> From what I can tell, fuzzing a program that spawns some number
> of threads would probably decrease the fuzzer's "stability" score,
> considering pthread scheduling isn't necessarily deterministic. But my
> main question(s) are:
>

> * Despite a stability decrease, is it still feasible to fuzz
> multithreaded targets?

it is feasible, though the lesser the stability the more pointless queue
entries. AFL++ tries to make this less an issue though by ignoring
unstable edges.

> o As an end goal: is it feasible to try to have AFL discover a
> deadlock situation? (That would be /awesome/.)

if you would trigger one, it would be detected as a hang, however due to
the nature of such issues it would not be repeatable and therefore not
be saved as a timeout test case.
For such things compile with -fsanitize=thread for a better detection of
threading issues.

> * How does having multiple threads (all running /instrumented/ code)

> affect AFL's shared memory region used to track target behavior?

well they all write to the same memory map. so if one thread is doing
things that has nothing to do with your fuzzing input then this fills up
the queue with unnecessary inputs and decreases the stability.
You should use AFL++'s AFL_LLVM_ALLOWLIST/AFL_LLVM_DENYLIST feature to
only instrument what you want to fuzz and not the rest and/or disable
the threads that you are not interested in in the code.

> * Is AFL's instrumentation thread-safe? Or is it not - meaning
> multiple threads could be writing to the /same /memory region
> simultaneously.

AFL's instrumentation is not thread-safe.
AFL++'s instrumentation is thread-safe if you compile the target with
AFL_LLVM_THREADSAFE_INST=1

Regards,
Marc

--
Marc Heuse
www.mh-sec.de

PGP: AF3D 1D4C D810 F0BB 977D 3807 C7EE D0A0 6BE9 F573

[Connor Shugg]

Hi Marc,

It's great to hear from you - thanks for taking the time to answer my questions. To clarify and keep the discussion going:

> I am not sure what you mean by "support" - but sure you can instrument and fuzz them :)

Definitely a fair point - I wasn't quite sure what "support" means in this context either. I suppose my main concern was thread-safety, which you clarified in your response.

> if you would trigger one, it would be detected as a hang, however due to the nature of such issues it would not be repeatable and therefore not be saved as a timeout test case.

This makes sense, given that AFL and AFL++ will both re-run a test case with a longer timeout to ensure it's truly a hang. A deadlock situation probably won't be recreated during that second run, thanks to thread indeterminism. -fsanitize=thread is something I hadn't considered before - thank you.

> well they all write to the same memory map. so if one thread is doing things that has nothing to do with your fuzzing input then this fills up the queue with unnecessary inputs and decreases the stability.

Thanks for clarifying this. A single memory map shared between threads definitely makes things tricky. AFL_LLVM_THREADSAFE_INST makes it thread-safe (thank you for pointing this out), but there's still the issue of a thread doing something unrelated to the fuzzer's input and 'polluting' the queue, like you said.

To be more transparent with my goals: I'm attempting to fuzz a program with multiple threads that wait for work to appear. Once some form of work does appear, a thread wakes up to handle it. I've had success with AFL++ by only utilizing one thread at a time, but I'm interested in pushing the envelope with multiple threads handling work. Given this situation, I believe the threads that aren't asleep will only be awake because they're handling the work given to them by the fuzzer (meaning, the multiple awake threads should all be running code triggered by the fuzzer's input).

That's just me thinking out loud; do you think fuzzing a multithreaded target behaving like this would be stable enough? If this is too vague or unclear, I'm happy to elaborate offline, if you're interested. Regardless, I really appreciate your answers and insight.

Thanks a lot for the help Marc!

Connor

--
You received this message because you are subscribed to the Google Groups "afl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to afl-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/afl-users/2bfa402a-f92e-29f6-9da0-fe6376f552db%40mh-sec.de.

[Marc]

Hi Connor,

On 12.09.21 23:49, Connor Shugg wrote:
> /> well they all write to the same memory map. so if one thread is doing

> things that has nothing to do with your fuzzing input then this fills up

> the queue with unnecessary inputs and decreases the stability./
> /
> /

>
> Thanks for clarifying this. A single memory map shared between
> threads definitely makes things tricky. AFL_LLVM_THREADSAFE_INST
> makes it thread-safe (thank you for pointing this out), but there's
> still the issue of a thread doing something unrelated to the
> fuzzer's input and 'polluting' the queue, like you said.
>
> To be more transparent with my goals: I'm attempting to fuzz a
> program with multiple threads that wait for work to appear. Once

> some form of work /does /appear, a thread wakes up to handle it.

> I've had success with AFL++ by only utilizing one thread at a time,
> but I'm interested in pushing the envelope with multiple threads
> handling work. Given this situation, I believe the threads that

> /aren't /asleep will only be awake because they're handling the work

> given to them by the fuzzer (meaning, the multiple awake threads
> should all be running code triggered by the fuzzer's input).
>
> That's just me thinking out loud; do you think fuzzing a
> multithreaded target behaving like this would be stable enough? If
> this is too vague or unclear, I'm happy to elaborate offline, if
> you're interested. Regardless, I really appreciate your answers and
> insight.

if the other threads are just workers waiting for more data then they
will do nothing during fuzzing and hance not have an impact on the
stability.
but you will know once you try :)

and in fuzzing you should not fuzz multiple threads from the same
fuzzing instance, unless they do different things and interact with each
other.

read about the persistent mode
(instrumentation/README.persistent_mode.md) on how to modify the source
code to have fast and efficient fuzzing.

Also note that there is a big online fuzzing community using discord for
more effective exchange: https://discord.gg/DAGdcpz

[Connor Shugg]

Marc,

Excellent, that logic makes sense to me, but you're right: the next step is to try it!

I believe I've run out of questions ;) - thanks so much for the tips and knowledge. I'll certainly be joining that Discord channel so I can keep learning.

Have a great day.

Connor