syzbot dashboard

[Dmitry Vyukov]

Hi,

syzkaller/syzbot dashboard for upstream Linux kernel is now live at:
https://syzkaller.appspot.com/

syzbot is continuous fuzzing/reporting system which currently tests
mainline tree and well as net and bpf branches. In the past 8 months
syzbot has reported 500+ bugs in kernel, 130 are still unfixed.

syzbot is based on source code in dashboard/app, syz-ci and syz-hub.

[Dmitry Vyukov]

Hi,

syzkaller/syzbot dashboard is now live at:
https://syzkaller.appspot.com

syzbot is continuous fuzzing/reporting system based on syzkaller
fuzzer, currently it tests mainline tree and well as net-next and
bpf-next trees.

The dashboard shows info about active bugs reported by syzbot. There
are ~130 active bugs and I think ~2/3 of them are actionable (still
happen and have a reproducer or are simple enough to debug). For
active bugs you can see a bunch of additional info (when it started
happening, when it last happened, how frequently, on what
branches/commits and more crash reports, reproducers and duplicates).
For example:
https://syzkaller.appspot.com/bug?id=26cb120b31cd24d984fc16da67f50fb375c432a7
The report is now probably lost on LKML, but the crash still happens
and have a reproducer.

There are bugs in various pars of net subsystem, some in KVM, crypto,
keys, perf, tty, block, android drivers, loop device and a long tail
in other subsystems. If you are interested in health of a particular
subsystem, take a look for relevant bugs.

Some of the active bugs can be in fact already fixed. If so it makes
sense to tell syzbot about fixes to keep dashboard clean and enable
discovery of new similar bugs. syzbot uses Reported-by tags in commits
and #syz directives in emails to track these statuses. More details
are available at
https://github.com/google/syzkaller/blob/master/docs/syzbot.md

If syzkaller is not finding bugs in your subsystem, that's most likely
because it's not testing it. To test a particular subsystem syzkaller
needs some help in the form of interface descriptions (which are not
too hard to write, e.g.
https://github.com/google/syzkaller/blob/master/sys/linux/ipc.txt).
There is also source coverage report available at
https://storage.googleapis.com/syzkaller/cover/upstream.html (50MB),
so you can check if syzkaller currently covers your code or not and
how well.

And there is also a list of all fixed bugs so far:
https://syzkaller.appspot.com/?fixed=upstream

Thanks

[Linus Torvalds]

On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>
> syzkaller/syzbot dashboard is now live at:
> https://syzkaller.appspot.com

Ok, this may well be the thing that makes syzbot reports useful, if
they point to an external report instead of sending an absolutely huge
illegible email that nobody can read.

That said, I do think it needs some better summarizing still.

For example, landing on that front page and then going "Hmm, let's
look at that first report", I click on that

[upstream] BUG: corrupted list in remove_wait_queue

thing, and get to

https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6

fine. That page itself doesn't actually tell me really anything at all, though.

So I go to the first thing I see, click "log" and I get 6500 lines of
basically line noise.

Ok, so the real thing is under "report", which actually looks pretty legible.

Looking at a few other of those things, I get the same feeling. Can
you put one copy of the "report" in the main page for a bug? I assume
they are all slightly different, but there must be some commonality to
them that you group the syzcaller bugs by, and giving one of those
legible reports (with all the nice filename and line information and
basic register state) would likely be a good thing.

At that point, *if* a report has a reproducer, then sending reminders
to people with "this still happens, here's a link to the syzkaller
page for this report" might be much better received than the old huge
and very-hard-to-read emails.

The reminder might well want to have that legible and short "report"
in it too, so that people can just look at that to tell "is this
relevant for me" particularly if they perhaps already fixed it?

(I only looked at a handful of reports, but the ones I looked at all
seemed reasonable - maybe some are less so?)

Linus

[Dmitry Vyukov]

On Wed, Mar 21, 2018 at 9:32 PM, Linus Torvalds
<torv...@linux-foundation.org> wrote:
> On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>
>> syzkaller/syzbot dashboard is now live at:
>> https://syzkaller.appspot.com
>
> Ok, this may well be the thing that makes syzbot reports useful, if
> they point to an external report instead of sending an absolutely huge
> illegible email that nobody can read.

The dashboard was initially meant to be used only for internal
monitoring (communication with kernel developers over emails) and I am
too close to the project to see obvious stuff. So thanks for the
feedback!

Do you mean to replace all attachments in syzbot emails with links to
web dashboard content?
I've heard opinions both ways (including "I am not clicking on any
external links, include all content in the email"). So we need to
decide. Provided that lots of email clients seem to inline
attachments, rather than show them as separate files, I am leaning
towards providing just links. Then emails can be short and tidy. Also
will solve the problem of "I don't see original attachments in your
forwarded email".

> That said, I do think it needs some better summarizing still.
>
> For example, landing on that front page and then going "Hmm, let's
> look at that first report", I click on that
>
> [upstream] BUG: corrupted list in remove_wait_queue
>
> thing, and get to
>
> https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6
>
> fine. That page itself doesn't actually tell me really anything at all, though.
>
> So I go to the first thing I see, click "log" and I get 6500 lines of
> basically line noise.
>
> Ok, so the real thing is under "report", which actually looks pretty legible.
>
> Looking at a few other of those things, I get the same feeling. Can
> you put one copy of the "report" in the main page for a bug? I assume
> they are all slightly different, but there must be some commonality to
> them that you group the syzcaller bugs by, and giving one of those
> legible reports (with all the nice filename and line information and
> basic register state) would likely be a good thing.

This is now done and deployed, the page show a sample crash at the top:
https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6

> At that point, *if* a report has a reproducer, then sending reminders
> to people with "this still happens, here's a link to the syzkaller
> page for this report" might be much better received than the old huge
> and very-hard-to-read emails.

Now with your blessing we will do reminders :)
Not sure what should be periodicity, though. Obviously less frequent
than once per day. But probably more frequent than once per release.

For the bugs *without* reproducers, now we know precisely that lots of
them are fixable. For bugs *with* reproducers fix rate is 76%, but for
bugs *without* reproducers it's only down to 66% and more than 100
bugs were fixed without reproducers. So it can make sense to do
reminders for them later too.

> The reminder might well want to have that legible and short "report"
> in it too, so that people can just look at that to tell "is this
> relevant for me" particularly if they perhaps already fixed it?
>
> (I only looked at a handful of reports, but the ones I looked at all
> seemed reasonable - maybe some are less so?)

There is some, small fraction of unreasonable crashes. Part of them is
due to induced crashes after silent memory corruptions. But one can
reply with "#syz invalid" to a syzbot email, and such reports will go
off the dashboard.

[Linus Torvalds]

On Thu, Mar 22, 2018 at 2:45 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>
> Do you mean to replace all attachments in syzbot emails with links to
> web dashboard content?

Yes. I think the email should contain just the "report" and then a
link to the full information.

That would seem to be a good enough summary to be legible, and
sufficient for a developer to decide "am I intested, and does it look
like I'm the right target".

Of course, any web pointer in an email does mean that a lot of spam
logic will immediately sit up and take notice, and I don't know how
you'll avoid that.

And then if/when a developer says "that looks interesting", following
the one link gets him/her all the actual information.

Honestly, if somebody isn't willing to click on a link, that person
clearly is too lazy to bother with the bug in the first place.

But at the same time, the email does need to have enough information
that you can sanely make that "is it worth it to go click on a link"
or not.

In fact, it should be sufficient information that for an obvious bug,
you don't even really have to click on the link at all.

I do think that that "report" dump is a fairly good approximation of
just that: enough information that a clear bug doesn't even need any
more, but also not so much information that the email report itself
becomes illegible or annoying.

But auto-generated emails are still likely to be an annoyance for
people who get a lot of email (and we all do), so I'd want to make
sure that

(a) there's some reasonable ratelimiting. No more than once a week
for the same report, for example, and some way for a developer to say
"I'm not the right person for this report".

(b) only send those emails if there really is enough information: a
good report with a real WARNING or BUG with the whole file and line
number information, _and_ it has been reproduced by the automation
(which hopefully means that there is also a good C reproducer).

but given that, I think it sounds like a good balance between "not too
annoying" and "honestly useful".

Will there be cases when automation does the wrong thing and is
annoying? Yes. Unquestionably. Nothing is perfect. But make the "good
report" succinct and useful enough, and make the false positives rare
enough, and I think people will appreciate it.

And maybe practice then shows some other case that really annoys
people, but that's more an issue of "this may need some tuning" than
anything else.

Linus

[Eric Biggers]

On Thu, Mar 22, 2018 at 10:45:43AM +0100, 'Dmitry Vyukov' via syzkaller wrote:
> On Wed, Mar 21, 2018 at 9:32 PM, Linus Torvalds
> <torv...@linux-foundation.org> wrote:
> > On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> >>
> >> syzkaller/syzbot dashboard is now live at:
> >> https://syzkaller.appspot.com
> >
> > Ok, this may well be the thing that makes syzbot reports useful, if
> > they point to an external report instead of sending an absolutely huge
> > illegible email that nobody can read.
>
> The dashboard was initially meant to be used only for internal
> monitoring (communication with kernel developers over emails) and I am
> too close to the project to see obvious stuff. So thanks for the
> feedback!
>
> Do you mean to replace all attachments in syzbot emails with links to
> web dashboard content?
> I've heard opinions both ways (including "I am not clicking on any
> external links, include all content in the email"). So we need to
> decide. Provided that lots of email clients seem to inline
> attachments, rather than show them as separate files, I am leaning
> towards providing just links. Then emails can be short and tidy. Also
> will solve the problem of "I don't see original attachments in your
> forwarded email".
>

A compromise could be to remove the raw.log from the emails, and send the config
in 'defconfig' format rather than the full config; that trims it down from 5256
lines to 925 currently. That would get rid of most of what is bloating the size
of the emails, while still keeping the (very useful) C and syzkaller
reproducers. Dashboard would still have everything, of course.

Eric

[Linus Torvalds]

On Fri, Mar 23, 2018 at 1:16 PM, Eric Biggers <ebig...@gmail.com> wrote:
>
> A compromise could be to remove the raw.log from the emails, and send the config
> in 'defconfig' format rather than the full config

I really don't think the config is very legible, and isn't something
that people will look at to judge whether it's intrresting or not.
It's still "many hundreds of lines of line noise"

And it shouldn't really be part of the consideration whether a bug is
really worth looking at or not anyway, although *after* you've found
the bug as a developer, you may decide "ok, that configuration isn't
valid, so I should just disallow it in Kconfig". But that's still a
bugfix.

So I think the config can stay purely on the web (for when you go "so
how do I *reproduce* this?") along with the actual reproducer and the
full logs.

I'd rather make the emails be small and legible, and contain the
minimum amount of information.

Because I really do *not* need a few hundred emails from some
automated source that each have a roughly thousand-line config
attachements.

An email with 50 lines of THE ACTUAL DECODED OOPS? Sure. That's
critical information. That doesn't make me go "why does this stupid
bot spew garbage at me"?

Linus

[Dmitry Vyukov]

I've switched emails to links instead of attachments, here are few
recent examples:
https://lkml.org/lkml/2018/3/25/31
https://lkml.org/lkml/2018/3/25/256
https://lkml.org/lkml/2018/3/25/257

Filed https://github.com/google/syzkaller/issues/550 for periodic
pings and noted your requirements. In particular, we will need
something for "I'm not the right person for this report", and some
plan for gradual rollout. It will take some time to implement.
Meanwhile developers are welcome to look at the dashboard
https://syzkaller.appspot.com/

Thanks again

[Linus Torvalds]

Coming back to this now that the merge window is almost over ]

On Mon, Mar 26, 2018 at 1:46 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>
> I've switched emails to links instead of attachments, here are few
> recent examples:
> https://lkml.org/lkml/2018/3/25/31
> https://lkml.org/lkml/2018/3/25/256
> https://lkml.org/lkml/2018/3/25/257

Looks good to me. I notice that only the last one got any replies, though.

I wonder if some people auto-ignore the new reports because of having
been burned by the previous "huge illegible emails" issue.

I do see syzbot fixes in rdma, though, just not for that
cma_listen_on_all issue. So maybe that bug is nastier.

Linus

[Dmitry Vyukov]

"possible deadlock in handle_rx" is now marked as fixed with
"vhost_net: add missing lock nesting notation".
"KASAN: use-after-free Read in pppol2tp_connect (3)" is marked as
fixed with "l2tp: fix races in tunnel creation".
"KASAN: use-after-free Read in __list_add_valid (5)" is still open.

Though, there are still 182 open bugs on dashboard:
https://syzkaller.appspot.com/

80+ have reproducers and happened last time recently enough. Few
dozens without reproducers, but as history shows ~2/3 of them are
fixable. Part of the remaining ones is probably fixed, but nobody told
syzbot about the fixes. Part is probably still unfixed but happen
episodically.

[Randy Dunlap]

https://syzkaller.appspot.com/bug?id=0627d4551fdc39bf1ef5d82cd9eef587047f7718
is fixed by 9ad553abe66f ("fs/reiserfs/journal.c: add missing resierfs_warning() arg"),

which does contain the string:
Reported-by: <syzbot+6bd77b...@syzkaller.appspotmail.com>

Is one of those hex strings incorrect?

--
~Randy

[Dmitry Vyukov]

Hi Randy,

They are correct and no action is needed. It's just that the commit
did not reach any of the trees that syzbot polls, so it did not know
about them yet. You can see that now syzbot has picked the commit.