Re: Bugs involving maliciously crafted file system

135 views
Skip to first unread message

Dmitry Vyukov

unread,
May 26, 2018, 1:13:23 PM5/26/18
to Theodore Y. Ts'o, Eric Sandeen, Eric Biggers, Darrick J. Wong, Dave Chinner, Brian Foster, LKML, linu...@vger.kernel.org, syzkaller-bugs, Tetsuo Handa, syzkaller
On Thu, May 24, 2018 at 1:41 AM, Theodore Y. Ts'o <ty...@mit.edu> wrote:
> On Wed, May 23, 2018 at 01:01:59PM -0500, Eric Sandeen wrote:
>>
>> What I'm personally hung up on are the bugs where the "exploit" involves merely
>> mounting a crafted filesystem that in reality would never (until the heat death
>> of the universe) corrupt itself into that state on its own; it's the "malicious
>> image" case, which is quite different than exposing fundamental bugs like the
>> SB_BORN race or or the user-exploitable ext4 flaw you mentioned in your reply.
>> Those are more insidious and/or things which can be hit by real users in real life.
>
> Well, it *can* be hit in real life. If you have a system which auto
> mounts USB sticks, then an attacker might be able to weaponize that
> bug by creating a USB stick where mounted and the user opens a
> particular file, the buffer overrun causes code to be executed that
> grabs the user's credentials (e.g., ssh-agent keys, OATH creds, etc.)
> and exfiltrates them to a collection server.
>
> Fedora and Chrome OS might be two such platforms where someone could
> very easily create a weaponized exploit tool where you could insert a
> file system buffer overrun bug, and "hey presto!" it becomes a serious
> zero day vulnerability.
>
> (I recently suggested to a security researcher who was concerned that
> file system developers weren't taking these sorts of things seriously
> enough could do a service to the community by creating a demonstration
> about how these sorts of bugs can be weaponized. And I suspect it
> could be about just as easily on Chrome OS as Fedora, and that can be
> one way that an argument could be made to management that more
> resources should be applied to this problem. :-)
>
> Of course, not all bugs triggered by a maliciously crafted file system
> are equally weaponizable. An errors=panic or a NULL derefrence are
> probably not easily exploitable at all. A buffer overrun (and I fixed
> two in ext4 in the last two days while being stuck in a T13 standards
> meeting, so I do feel your pain) might be a very different story.
>
> Solutions
> ---------
>
> One of the things I've wanted to get help from the syzbot folks is if
> there was some kind of machine learning or expert system evaluation
> that could be done so malicious image bugs could be binned into
> different categories, based on how easily they can be weaponized.
> That way, when there is a resource shortage situation, humans can be
> more easily guided into detremining which bugs should be prioritized
> and given attention, and which we can defer to when we have more time.

Hi Ted,

I don't see that "some kind of machine learning or expert system
evaluation" is feasible. At least not in short/mid-term. There are
innocently-looking bugs that actually turn out to be very bad, and
there are badly looking at first glance bugs that actually not that
bad for some complex reasons. Full security assessment is a complex
task and I think stays "human expert area" for now. One can get some
coarse estimation by searching for "use-after-free" and
"out-of-bounds" on the dashboard.

Also note that even the most innocent bugs can block ability to
discover deeper and worse bugs during any runtime testing. So
ultimately all need to be fixed if we want correct, stable and secure
kernel. To significant degree it's like compiler warnings: you either
fix them all, or turn them off, there is no middle ground of having
thousands of unfixed warnings and still getting benefit from them.


> Or maybe it would be useful if there was a way where maintainers could
> be able to annotate bugs with priority and severity levels, and maybe
> make comments that can be viewed from the Syzbot dashboard UI.

This looks more realistic. +Tetsuo proposed something similar:
https://github.com/google/syzkaller/issues/608

I think to make it useful we need to settle on some small set of
well-defined tags for bugs that we can show on the dashboard.
Arbitrary detailed free-form comments can be left on the mailing list
threads that are always referenced from the dashboard.

What tags would you use today for existing bugs? One would be
"security-critical", right?



> The other thing that perhaps could be done is to set up a system where
> the USB stick is automounted in a guest VM (using libvirt in Fedora,
> and perhaps Crostini for Chrome OS), and the contents of the file
> system would then get exported from the guest OS to the host OS using
> either NFS or 9P. (9P2000.u is the solution that was used in
> gVisor[1].)
>
> [1] https://github.com/google/gvisor
>
> It could be that putting this kind of security layer in front to
> automounted USB sticks is less work than playing whack-a-mole fixing a
> lot of security bugs with maliciously crafted file systems.

I don't think that auto mounting or "requires root" is significantly
relevant in this context. If one needs to use a USB stick, or DVD or
just any filesystem that they did not create themselves, there is
pretty much no choice than to mount it, issuing sudo if necessary. If
you did not create it yourself with a trusted program, there is no way
you can be sure in the contents of the thing and there is no way you
can verify every byte of it before mounting. That's exactly the work
for software. Responsibility shifting like "you said sudo so now it's
all on you" is not useful for users. It's like web sites that give you
a hundred page license agreement before you can use it, but now you
clicked Agree so it's all on you, you read and understood every word
of it and if there would be any concern you would not click Agree,
right?
Fixing large legacy code bases is hard. But there is no other way than
persistent testing and fixing one bug at a time. We know that it's
doable because browsers did it over the past 10 years for much larger
set of input formats.



> For sure. I guess some subset of the crashes could be more carefully
> crafted to be more dangerous, but fuzzers really don't tell us that today,
> in fact the more insidious flaws that don't turn up as a crash or hang likely
> go unnoticed.

Well, we have KASAN, almost have KMSAN and will have KTSAN in future.
They can detect detect significant portion of bugs that go unnoticed
otherwise. At least this prevents "bad guys" from also using tooling
to cheaply harvest exploits. Systematic use of these tools on browsers
raised exploit costs to $1M+ for a reason.

Theodore Y. Ts'o

unread,
May 26, 2018, 4:24:46 PM5/26/18
to Dmitry Vyukov, Eric Sandeen, Eric Biggers, Darrick J. Wong, Dave Chinner, Brian Foster, LKML, linu...@vger.kernel.org, syzkaller-bugs, Tetsuo Handa, syzkaller
On Sat, May 26, 2018 at 07:12:49PM +0200, Dmitry Vyukov wrote:
>
> I don't see that "some kind of machine learning or expert system
> evaluation" is feasible. At least not in short/mid-term. There are
> innocently-looking bugs that actually turn out to be very bad, and
> there are badly looking at first glance bugs that actually not that
> bad for some complex reasons. Full security assessment is a complex
> task and I think stays "human expert area" for now. One can get some
> coarse estimation by searching for "use-after-free" and
> "out-of-bounds" on the dashboard.

If the kernel intentionally triggers a BUG_ON or a panic (as in file
systems configured with 'tune2fs -e panic') it's pretty obvious that
those errors can't be weaponized to execute code chosen by the
attacker. Would you agree with that?

The same should be true for "blocked for more than 120 seconds";
again, I claim that those sorts of errors are by definition less
serious than buffer overruns.

So there is at least some kind of automated evaluation that can be
done, even if the general case problem is really hard.

> > Or maybe it would be useful if there was a way where maintainers could
> > be able to annotate bugs with priority and severity levels, and maybe
> > make comments that can be viewed from the Syzbot dashboard UI.
>
> This looks more realistic. +Tetsuo proposed something similar:
> https://github.com/google/syzkaller/issues/608
>
> I think to make it useful we need to settle on some small set of
> well-defined tags for bugs that we can show on the dashboard.
> Arbitrary detailed free-form comments can be left on the mailing list
> threads that are always referenced from the dashboard.
>
> What tags would you use today for existing bugs? One would be
> "security-critical", right?

For me, it's not about tags. Things missing from the
https://syzkaller.appspot.com/ front page are:

* Whether or not a repro is available
* Which subsystems the bug has been tentatively assigned
* A maintainer assigned priority and severity level

I generally don't use the syzkaller.apptspot.com front page because
it's too hard to find the sorts of thing that I'm looking for ---
namely the most important syzkaller bug that I as an ext4 expert can
work on.

If someone else sends me a well-formed bug report on
bugzilla.kernel.org, with free-standing image file, and a simple .c
reproducer, I can make forward progress much more quickly. So if I'm
time bound, guess which bug I'm going to pay attention to first?
Especially when Syzkaller makes it hard for me to find the bug again
once it ages out of my inbox?

> Well, we have KASAN, almost have KMSAN and will have KTSAN in future.
> They can detect detect significant portion of bugs that go unnoticed
> otherwise. At least this prevents "bad guys" from also using tooling
> to cheaply harvest exploits. Systematic use of these tools on browsers
> raised exploit costs to $1M+ for a reason.

I'll note that browsers also use processes and capsicum to provide
security boundaries. This is why David and I have been suggesting
solutions like FUSE or running the mount in a guest VM, which can act
as an isolation layer. Even if there is a bug in the guest kernel,
the blast radius of the bug can be isolated, hopefully to the point
where it can be completely contained. It's not an either-or, but a
both-and.

But the much more important thing is that mangement has to be willing
to **fund** bug remediation. That was true for Chrome; it doesn't
seem to be as true for the Linux Kernel, for whatever reason.

People trying to fix Syzkaller and other fuzzer-found bugs on 20%
time, or on the weekends, or as a background activity during
low-bandwidth meetings, or as an unfunded mandate that doesn't show up
on anyone's quarterly objectives upon which they are graded, is just
not going to scale.

And if that's the reality, it may very well be that if you want
Syzkaller to make more a difference, anything you can do to reduce the
human toil needed to investigate a bug is going to be hugely
important.

And if bug remediation is only going to be funded to a very limited
extent, then it's important that the bugs we work on are the highest
priority ones, since the lower priority ones *will* have to let slide.

Regards,

- Ted

Dmitry Vyukov

unread,
Jun 11, 2018, 9:07:46 AM6/11/18
to Theodore Y. Ts'o, Dmitry Vyukov, Eric Sandeen, Eric Biggers, Darrick J. Wong, Dave Chinner, Brian Foster, LKML, linux-xfs, syzkaller-bugs, Tetsuo Handa, syzkaller
On Sat, May 26, 2018 at 10:24 PM, Theodore Y. Ts'o <ty...@mit.edu> wrote:
> On Sat, May 26, 2018 at 07:12:49PM +0200, Dmitry Vyukov wrote:
>>
>> I don't see that "some kind of machine learning or expert system
>> evaluation" is feasible. At least not in short/mid-term. There are
>> innocently-looking bugs that actually turn out to be very bad, and
>> there are badly looking at first glance bugs that actually not that
>> bad for some complex reasons. Full security assessment is a complex
>> task and I think stays "human expert area" for now. One can get some
>> coarse estimation by searching for "use-after-free" and
>> "out-of-bounds" on the dashboard.
>
> If the kernel intentionally triggers a BUG_ON or a panic (as in file
> systems configured with 'tune2fs -e panic') it's pretty obvious that
> those errors can't be weaponized to execute code chosen by the
> attacker. Would you agree with that?
>
> The same should be true for "blocked for more than 120 seconds";
> again, I claim that those sorts of errors are by definition less
> serious than buffer overruns.
>
> So there is at least some kind of automated evaluation that can be
> done, even if the general case problem is really hard.

These can't be weaponized to execute code, but if a BUG_ON is
triggerable over a network, or from VM guest, then it's likely more
critical than a local code execution. That's why I am saying that
automated evaluation is infeasible.

Anyway, bug type (UAF, BUG, task hung) is available in the bug title
on dashboard and on mailing lists, so you can just search/sort bugs on
the dashboard. What other interface you want on top of this?



>> > Or maybe it would be useful if there was a way where maintainers could
>> > be able to annotate bugs with priority and severity levels, and maybe
>> > make comments that can be viewed from the Syzbot dashboard UI.
>>
>> This looks more realistic. +Tetsuo proposed something similar:
>> https://github.com/google/syzkaller/issues/608
>>
>> I think to make it useful we need to settle on some small set of
>> well-defined tags for bugs that we can show on the dashboard.
>> Arbitrary detailed free-form comments can be left on the mailing list
>> threads that are always referenced from the dashboard.
>>
>> What tags would you use today for existing bugs? One would be
>> "security-critical", right?
>
> For me, it's not about tags. Things missing from the
> https://syzkaller.appspot.com/ front page are:
>
> * Whether or not a repro is available

This was always available in the Repro column.

> * Which subsystems the bug has been tentatively assigned
> * A maintainer assigned priority and severity level

Let's call this tags collectively (unless you have a better name). P0
or subsystem:ext4 can also be tags.
So you mean: (1) priority levels (P0, P1, P2), (2) severity levels
(S0, S1, S2) and subsystem, right?

On a related note, perhaps kernel community needs to finally start
using bugzilla for real, like with priorities, assignees, up-to-date
statuses, no stale bugs, etc. All of this is available in bug tracking
systems for decades...

Theodore Y. Ts'o

unread,
Jun 11, 2018, 9:33:48 AM6/11/18
to Dmitry Vyukov, Eric Sandeen, Eric Biggers, Darrick J. Wong, Dave Chinner, Brian Foster, LKML, linux-xfs, syzkaller-bugs, Tetsuo Handa, syzkaller
On Mon, Jun 11, 2018 at 03:07:24PM +0200, Dmitry Vyukov wrote:
>
> These can't be weaponized to execute code, but if a BUG_ON is
> triggerable over a network, or from VM guest, then it's likely more
> critical than a local code execution. That's why I am saying that
> automated evaluation is infeasible.

I can't imagine situations where a BUG_ON would be more critical than
local code execution. You can leverage local code execution to ah
remote privilege escalation attack; and local code execution can (with
less effort) be translated to a system crash. Hence, local code
execution is always more critical than a BUG_ON.

> Anyway, bug type (UAF, BUG, task hung) is available in the bug title
> on dashboard and on mailing lists, so you can just search/sort bugs on
> the dashboard. What other interface you want on top of this?

I also want to be able to search and filter based on subsystem, and
whether or not there is a reproducer. Sometimes you can't even figure
out the subsytem from the limited string shown on the dashboard,
because the original string didn't include the subsystem to begin
with, or the the subsytem name was truncated and not included on the
dashboard.

> On a related note, perhaps kernel community needs to finally start
> using bugzilla for real, like with priorities, assignees, up-to-date
> statuses, no stale bugs, etc. All of this is available in bug tracking
> systems for decades...

I do use bugzilla and in fact if syzbot would automatically file a
bugzilla.kernel.org report for things that are in the ext4 subsystem,
that would be really helpful.

As far as no stale bugs, etc., many companies (including Google)
aren't capable of doing that with their own internal bug tracking
systems, because management doesn't give them enough time to track and
fix all stale bugs. You seem to be assuming/demanding things of the
kernel community that are at least partially constrained by resource
availability --- and since you've used constrained resources as a
reason why Syzbot can't be extended as we've requested to reduce
developer toil and leverage our available resources, it would perhaps
be respectful if you also accepted that resource constraints also
exist in other areas, such as how much we can keep a fully groomed bug
tracking system.

Regards,

- Ted

Dmitry Vyukov

unread,
Jun 15, 2018, 5:32:35 AM6/15/18
to Theodore Y. Ts'o, Dmitry Vyukov, Eric Sandeen, Eric Biggers, Darrick J. Wong, Dave Chinner, Brian Foster, LKML, linux-xfs, syzkaller-bugs, Tetsuo Handa, syzkaller
On Mon, Jun 11, 2018 at 3:33 PM, Theodore Y. Ts'o <ty...@mit.edu> wrote:
> On Mon, Jun 11, 2018 at 03:07:24PM +0200, Dmitry Vyukov wrote:
>>
>> These can't be weaponized to execute code, but if a BUG_ON is
>> triggerable over a network, or from VM guest, then it's likely more
>> critical than a local code execution. That's why I am saying that
>> automated evaluation is infeasible.
>
> I can't imagine situations where a BUG_ON would be more critical than
> local code execution. You can leverage local code execution to ah
> remote privilege escalation attack; and local code execution can (with
> less effort) be translated to a system crash. Hence, local code
> execution is always more critical than a BUG_ON.


Well, if one could bring all of Google servers remotely, lots of
people would consider this as more critical as _anything_ local.


>> Anyway, bug type (UAF, BUG, task hung) is available in the bug title
>> on dashboard and on mailing lists, so you can just search/sort bugs on
>> the dashboard. What other interface you want on top of this?
>
> I also want to be able to search and filter based on subsystem, and
> whether or not there is a reproducer. Sometimes you can't even figure
> out the subsytem from the limited string shown on the dashboard,
> because the original string didn't include the subsystem to begin
> with, or the the subsytem name was truncated and not included on the
> dashboard.

How is this problem solved in kernel development for all other bug reports?

>> On a related note, perhaps kernel community needs to finally start
>> using bugzilla for real, like with priorities, assignees, up-to-date
>> statuses, no stale bugs, etc. All of this is available in bug tracking
>> systems for decades...
>
> I do use bugzilla and in fact if syzbot would automatically file a
> bugzilla.kernel.org report for things that are in the ext4 subsystem,
> that would be really helpful.
>
> As far as no stale bugs, etc., many companies (including Google)
> aren't capable of doing that with their own internal bug tracking
> systems, because management doesn't give them enough time to track and
> fix all stale bugs. You seem to be assuming/demanding things of the
> kernel community that are at least partially constrained by resource
> availability --- and since you've used constrained resources as a
> reason why Syzbot can't be extended as we've requested to reduce
> developer toil and leverage our available resources, it would perhaps
> be respectful if you also accepted that resource constraints also
> exist in other areas, such as how much we can keep a fully groomed bug
> tracking system.

I mentioned this only because you asked for this.
Whatever tracking system and style we go with, bug states need to
maintained and bugs need to be nursed. If we extend syzbot dashboard
with more traditional bug tracking system capabilities, but then
nobody cares to maintain order, it also won't be useful and nobody
will be able to easily select the current tasks to work on.
So that's a prerequisite for what you are asking for.

Well, you use bugzilla, but somebody else uses something else. This
fragmentation is kernel development practices does not allow to build
further automation on top. We can't do a personal solution for each
developer. For now the greatest common divisor seems to be freeform
emails on mailing lists...

A good example is "I submitted 7 kernel bugs to bugzilla, but nobody
answered me" email thread from today:
https://groups.google.com/forum/#!topic/syzkaller/OnbMQbbE4gQ
Reply all
Reply to author
Forward
0 new messages