Running afl-cmin in /tmp/xxx

[Leo Barnes]

Hi!

Is there any specific reason why afl-cmin errors out when input/output/executable is in /tmp?

I'm guessing it uses /tmp to store stuff? Or is it due to concerns that /tmp may get deleted if disk space goes low? I'm guessing it's just a sanity check to guard against uncommon edge cases, but thought I'd check anyway.

Thanks!

//Leo

[Michal Zalewski]

> Is there any specific reason why afl-cmin errors out when input/output/executable is in /tmp?

It's a shell script, and writing to world-writeable directories from
within shell scripts is pretty insecure (at least without making
things a lot more complicated and potentially slower than they are).

Soo, there is a simple check for /tmp to discourage you from doing
that (and / or to discourage "vulnerability reports" for the scripts).

If it's a single-user system, feel free to comment out the relevant
four lines from afl-cmin.

/mz

[Jiri Jaburek]

 
Are you sure the extra logic (trying to outsmart the user) is necessary?

It doesn't exactly matter if it's a shell script or not, /tmp is usually
world-writable, but also has S_ISVTX ("sticky bit") set, so that no other
user can modify the files if test corpus damage is a concern.

In my case, I have a simple afl-tmin/afl-cmin script that uses mktemp() to
get a tempdir to minimize the corpus in before uploading it elsewhere. This
is done on a disposable OS instance (running container image) because running
anything with fuzz-testing generated results is a bad idea on anything
non-disposable.

As such, I've run into this issue where a tool which was supposed to do one
"simple" job failed to do so, because it tried to be smart about out-of-scope
things. As an analogy, when I do 'rm -rf *', I don't want it telling me it's
a bad idea, I want it to remove files.
A similar thing happened with afl-fuzz trying to be pedantic about core_pattern
(a fatal error which probably should have been just a warning).

Maybe use a warning or -f/--force flag or environment variable for advanced
users, so they don't have to maintain their own versions of afl?

Sorry for being a bit more "direct" and thanks for the tool,
Jiri

[Michal Zalewski]

> It doesn't exactly matter if it's a shell script or not, /tmp is usually
> world-writable, but also has S_ISVTX ("sticky bit") set, so that no other
> user can modify the files if test corpus damage is a concern.

Among other things, an evil unprivileged user on the system can
pre-create symlinks (or hardlinks) in /tmp to cause you to clobber
sensitive files.

While it's not a significant concern in most fuzzing setups, I didn't
want to spend time securing these scripts, and I did not want to have
someone post an advisory about how afl-fuzz handles this unsafely.
So... =)

> A similar thing happened with afl-fuzz trying to be pedantic about core_pattern
> (a fatal error which probably should have been just a warning).

It really should not. If your core_pattern is not set correctly, AFL
will not be able to detect crashes, and the whole point of fuzzing
will be lost.

/mz

[Jiri Jaburek]

On Monday, July 17, 2017 at 5:16:53 PM UTC+2, Michal Zalewski wrote:

> It doesn't exactly matter if it's a shell script or not, /tmp is usually
> world-writable, but also has S_ISVTX ("sticky bit") set, so that no other
> user can modify the files if test corpus damage is a concern.

Among other things, an evil unprivileged user on the system can
pre-create symlinks (or hardlinks) in /tmp to cause you to clobber
sensitive files.

That's still not something afl toolset should concern itself about, it's
a completely unrelated (system administration) issue with no impact on the
afl functionality.

While it's not a significant concern in most fuzzing setups, I didn't
want to spend time securing these scripts, and I did not want to have
someone post an advisory about how afl-fuzz handles this unsafely.
So... =)

I can understand that, even though it seems like a very rare corner case.

> A similar thing happened with afl-fuzz trying to be pedantic about core_pattern
> (a fatal error which probably should have been just a warning).

It really should not. If your core_pattern is not set correctly, AFL
will not be able to detect crashes, and the whole point of fuzzing
will be lost.

That's not what afl-fuzz.c says on the issue - crashes can still be detected
as no core_pattern can prevent you from getting the right WTERMSIG() from
waitpid, just that it will be delayed and thus be potentially counted as
a timeout if the user didn't set -t high enough.

/mz

 
Anyway, thanks for the answer.
Jiri

[Jakub Wilk]

I think these /tmp checks in afl-cmin are suboptimal for multiple reasons:

- They check too much. For example, "afl-cmin -i /tmp/foo/in -o /tmp/foo/bar
..." will barf, even when /tmp/foo is accessible only to me. (I had to
implement a gross work-around to make python-afl test suite pass in a
subdirectory of /tmp...)

- They check too little. People might want to (ab)use /dev/shm as a substitute
for /tmp, because the former is always tmpfs, unlike the latter. But afl-cmin
hates only /tmp and /var/tmp.

- It's not obvious what they are good for. People will assume they aren't
needed, implement work-arounds... and then it might turn out the checks were
necessary after all.


I had a look at the code, and afl-cmin could indeed misbehave in some
semi-reasonable[*] setups involving /tmp (or a similar dir):

1) "afl-cmin -o /tmp/foo/ ..." if /tmp/foo/ doesn't exist (or exists, but
doesn't belong to us):

find "$OUT_DIR" -name 'id[:_]*' -maxdepth 1 -exec rm -- {} \; 2>/dev/null

But "find" could follow malicious symlinks.

(I'm confused. Why does it delete any files?)

2) "afl-cmin -o /tmp/foo/ ..." even if /tmp/foo/ exists and belong to us:

rmdir "$OUT_DIR" 2>/dev/null
...
mkdir -m 700 -p "$TRACE_DIR" || exit 1

But "mkdir -p" is happy to descend into directories owned by other users.

3) "afl-cmin -f /tmp/foo" even if /tmp/foo exists and belongs to us:

rm -f "$STDIN_FILE" || exit 1
touch "$STDIN_FILE" || exit 1

Between "touch" and "rm", an attacker could replace /tmp/foo with it's own
file. There are also some "cp" calls with might have similar atomicity issues.

([*] I personally wouldn't dare to use afl-cmin this way, but maybe others are
more adventurous.)

So, can we fix these problems and remove the /tmp checks?

A comment says:
# Do a sanity check to discourage the use of /tmp, since we can't really
# handle this safely from a shell script.

Shell has "set -C", which gives you O_EXCL superpowers. What else do you need?

--
Jakub Wilk

[Michal Zalewski]

>> Among other things, an evil unprivileged user on the system can
>> pre-create symlinks (or hardlinks) in /tmp to cause you to clobber
>> sensitive files.
>
> That's still not something afl toolset should concern itself about, it's
> a completely unrelated (system administration) issue with no impact on the
> afl functionality.

Not sure I follow. AFL runs on operating systems that have semantics
that allow such attacks. That's not somebody else's problem, the
platform existed before AFL.

>> It really should not. If your core_pattern is not set correctly, AFL
>> will not be able to detect crashes, and the whole point of fuzzing
>> will be lost.
>
> That's not what afl-fuzz.c says on the issue - crashes can still be detected
> as no core_pattern can prevent you from getting the right WTERMSIG() from
> waitpid, just that it will be delayed and thus be potentially counted as
> a timeout if the user didn't set -t high enough.

There is no clear definition of "high enough"; on some OSes, this can
be several seconds (!). Setting -t to this value has an extremely
detrimental effect on many types of fuzzing jobs (to the point of
making them useless).

FWIW, you can override the check if you really want. It's just that
it's almost never a good idea - and the UX philosophy of AFL is very
different from rm -rf /, in that it really tries to help you do the
right thing.

[Jakub's mail:]

> - They check too much. For example, "afl-cmin -i /tmp/foo/in -o /tmp/foo/bar ..." will barf, even when /tmp/foo is accessible only to me. (I had to implement a gross work-around to make python-afl test suite pass in a subdirectory of /tmp...)

Yeah, they are just a prefix check to detect the most common misuse pattern.

> - They check too little. People might want to (ab)use /dev/shm as a substitute for /tmp, because the former is always tmpfs, unlike the latter. But afl-cmin hates only /tmp and /var/tmp.

Ditto.

Look, I'm not a fan of these checks. They are there basically because
I woke up one day thinking that somebody is gonna post a stupid
advisory. I can remove them if you are *really* bothered, or rework
the scripts to use set -C and then double-check that every single
utility we call (e.g., gnuplot) behaves safely (probably not). But...
why?=)

/mz

[Sami Liedes]

On Mon, Jul 17, 2017 at 11:39:25AM -0700, Michal Zalewski wrote:
> >> Among other things, an evil unprivileged user on the system can
> >> pre-create symlinks (or hardlinks) in /tmp to cause you to clobber
> >> sensitive files.
> >
> > That's still not something afl toolset should concern itself about, it's
> > a completely unrelated (system administration) issue with no impact on the
> > afl functionality.
>
> Not sure I follow. AFL runs on operating systems that have semantics
> that allow such attacks. That's not somebody else's problem, the
> platform existed before AFL.

In my opinion it violates the "do one thing well" Unix philosophy. At
some point, tools should trust that the user knows what he is doing,
not pointlessly refusing because of fairly far-fetched threat
scenarios. I believe the number of users who would legitimately want
to run AFL under /tmp is at least a magnitude or two greater than
those who would run it under /tmp in a multiuser setting and think it
would be safe. The rm -rf / case is different because almost nobody
generally wants to do that, and because that command could result from
sloppy scripting like rm -rf /$somepath.

> Look, I'm not a fan of these checks. They are there basically because
> I woke up one day thinking that somebody is gonna post a stupid
> advisory. I can remove them if you are *really* bothered, or rework
> the scripts to use set -C and then double-check that every single
> utility we call (e.g., gnuplot) behaves safely (probably not). But...
> why?=)

My use case has generally been that I have a tmpfs mounted on /tmp on
a single-user system, I wouldn't want to be bothered to mount a tmpfs
on the place where I want to run AFL, and contrary to some claims I
have found that in many cases running on tmpfs is ~10x faster than on
an ext4. Thus, I always remove the checks, but it's a minor annoyance.

Sami

[Michal Zalewski]

>> Not sure I follow. AFL runs on operating systems that have semantics
>> that allow such attacks. That's not somebody else's problem, the
>> platform existed before AFL.
>
> In my opinion it violates the "do one thing well" Unix philosophy.

I don't wanna sound rude, but I'm OK with that :-) See
docs/historical_notes.txt for some notes of why I took a somewhat
different route.

Ultimately, if you find this design frustrating, and are not happy
with the ability to disable most errors / warnings through env
options, your alternative is to suggest a better design roughly in
line with the design goals for AFL; fork it off; or use libfuzzer or
honggfuzz, both of which have achieved feature parity with AFL over
the years (and have some unique tricks up their sleeves).

> My use case has generally been that I have a tmpfs mounted on /tmp on
> a single-user system, I wouldn't want to be bothered to mount a tmpfs
> on the place where I want to run AFL, and contrary to some claims I
> have found that in many cases running on tmpfs is ~10x faster than on
> an ext4. Thus, I always remove the checks, but it's a minor annoyance.

Without getting into existential debates, I can add an env option to
disable the /tmp check. Would that work?

(Independently from that, if you find tmpfs to be much faster for
fuzzing, you might want to investigate this a bit more, because it may
be a symptom of some other bottleneck, such as suboptimal mount flags
or sysctl weirdness. In general, for simple fuzzing jobs, tmpfs should
not offer dramatic perf benefits, since virtually all the I/O it does
is cached.)

/mz

[Sami Liedes]

On Mon, Jul 17, 2017 at 01:46:20PM -0700, Michal Zalewski wrote:
> Without getting into existential debates, I can add an env option to
> disable the /tmp check. Would that work?

Of course, that's not too much of a hassle to use. Thanks :)

> (Independently from that, if you find tmpfs to be much faster for
> fuzzing, you might want to investigate this a bit more, because it may
> be a symptom of some other bottleneck, such as suboptimal mount flags
> or sysctl weirdness. In general, for simple fuzzing jobs, tmpfs should
> not offer dramatic perf benefits, since virtually all the I/O it does
> is cached.)

It's not only about what AFL does, but also about what the target
binary does. Even text editors tend to nowadays do some kind of
fdatasync(), which already makes a huge difference. I seem to remember
I've also seen this when fuzzing mutt's mbox parsing code and e2fsck.
Both I think also write to the target file.

Having said that, I did live under the impression that the vastly more
complex ext4 code would also cause more CPU usage even when everything
is cached, and I think that used to be true at some point, but now
with a recent kernel it seems I cannot reproduce AFL being more than a
few percent slower on ext4 on a fast target. Or maybe it required the
queue directory to grow huge, I don't know.

Sami

[Jakub Wilk]

* Sami Liedes <sami....@iki.fi>, 2017-07-18, 02:27:

>Even text editors tend to nowadays do some kind of fdatasync(), which already
>makes a huge difference.

Sounds like a job for eatmydata:
https://launchpad.net/libeatmydata

>I seem to remember I've also seen this when fuzzing mutt's mbox parsing code
>and e2fsck.

BTW, when fuzzing e2fsck, make sure to disable its signal handlers:
https://github.com/jwilk/fuzzing/blob/master/patches/e2fsprogs/no-sigcatcher.diff

--
Jakub Wilk

[Michal Zalewski]

> Sounds like a job for eatmydata:
> https://launchpad.net/libeatmydata

Neat. Should also be a version for sleep(), etc (looking at you, vim) =)

Btw, AFL_ALLOW_TMP is now a thing.

/mz

[Jakub Wilk]

* Michal Zalewski <lca...@gmail.com>, 2017-07-17, 17:36:

>>https://launchpad.net/libeatmydata
>
>Neat. Should also be a version for sleep(), etc (looking at you, vim) =)

https://github.com/ivoire/timescaler maybe?
(Never tried it myself.)

There's also https://github.com/majek/fluxcapacitor , but it uses ptrace, so
it's probably not useful in the AFL context.

--
Jakub Wilk