[RFC] syzbot process

[Dmitry Vyukov]

Hello,

You might have seen bug reports coming from syzbot on LKML recently.
syzbot is an automated system that continuously fuzzes main Linux
kernel branches using syzkaller fuzzer and reports all found bugs:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md
So far it has reported ~280 bugs in slightly less than 2 months:
https://groups.google.com/forum/#!forum/syzkaller-bugs

One of the ideas behind syzbot was maximum automation because we
simply could not keep track, and not even report all bugs that
syzkaller finds (the number has crossed 1000). Bugs were massively
lost, not reported, context was lost (e.g. kernel commit, config,
etc), we could not report similarly looking crashes, we could not test
proposed fixes, etc. syzbot aims at solving all of these problems.

However, the cost is that it needs to understand statuses of bugs:
most importantly, what commit fixes what bug. It also has support for
marking a bug as "invalid", e.g. happened once but most likely was
caused by a previous silent memory corruption. And support for marking
bugs as duplicates of other bugs, i.e. the same root cause and will be
fixed when the target bug is fixed. These simple rules are outlined in
the footer of each report and also explained in more detail at the
referenced link:

----------------------------------
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzk...@googlegroups.com.
Please credit me with: Reported-by: syzbot <syzk...@googlegroups.com>
syzbot will keep track of this bug report.
Once a fix for this bug is merged into any tree, reply to this email with:
#syz fix: exact-commit-title
If you want to test a patch for this bug, please reply with:
#syz test: git://repo/address.git branch
and provide the patch inline or as an attachment.
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug report.
Note: all commands must start from beginning of the line in the email body.
----------------------------------

Status tracking allows syzbot to (1) keep track of still unfixed bugs
(more than half actually gets lost in LKML archives if nobody keeps
track of them), (2) be able to ever report similarly looking crashes
as new bugs in future, (3) be able to test fixes.

The problem is that these rules are mostly not followed.

I understand that following these rules is a moderate pain and I am
reaching to you to discuss potential solutions for this problem. I've
tried hard to come up with a scheme that would eliminate sending these
replies altogether, but failed.
We can simply to agree to follow the current convention, which is not
too hard in the end and sounds like a fair deal -- we give you bugs,
you give us fixing commits.
Or, we could somehow employ bugzilla.kernel.org for systematic bug
tracking. However, I've got an impression that it's mostly unused and
not used systematically even when used (e.g. bugs fixed 5 years ago
still rest there as open).
One idea that was proposed is do it the other way around. Namely,
syzbot gives you bug id, and you need to add a tag along the lines of
"syzbot-fix: HASH" to the commit. I don't know if kernel community
finds this easier to use or not. And dups, invalid bugs and missed
tags still needs to be handled in some other way (e.g. the current
way).
Any other proposals, thoughts, ideas?

Thank you

[Andrey Ryabinin]

2017-12-21 15:52 GMT+03:00 Dmitry Vyukov <dvy...@google.com>:

> Any other proposals, thoughts, ideas?
>

a) Assume that patches send in replies to the bug report are fixes.

b) Almost the same as your "syzbot-fix: HASH" proposal, but slightly
closer to normal kernel development workflow.
Add hash/bug id into the From field of email, i.e.

instead of
From: syzbot <syzk...@googlegroups.com>

make it
From: syzbot-{hash} <syzk...@googlegroups.com>

And ask to include "Reported-by: syzbot-{hash}
<syzk...@googlegroups.com>" tag in a changelog.

a) doesn't exclude b) or "#syz: fix " emails, and vise versa

[Stephan Mueller]

Am Donnerstag, 21. Dezember 2017, 14:22:40 CET schrieb Andrey Ryabinin:

Hi Andrey,

One additional suggestion: Rerun all already generated reproducer tests on,
say, each updated kernel (e.g. newer rc or even full new version). If an issue
is detected again, send a reply to the original message to indicate that the
issue is still there. Note, I sometimes even fear that a patch that ought to
fix the reported issue may not completely fix it considering races.

The problem with the current approach (at least to me) is that on mailing
lists with some volume, such reports get easily buried.

Ciao
Stephan

[Greg Kroah-Hartman]

No, don't use bugzilla :)

But, use what bugzilla does do well, provide a unique identifier that
can be referenced in git commit messages to point people at the problem
report.

Why not generate a unique one per "problem" you find? And have a way to
look them up somehow?

Then you can put:
syzkaller-id: XXXXXX
in the commit log and you can track it easier?

That's what all other "tools" do that try to track kernel bug reports.
We do that for coverity as one such example, and lots of different bug
trackers as well.

thanks,

greg k-h

[Linus Torvalds]

On Thu, Dec 21, 2017 at 5:22 AM, Andrey Ryabinin <ryabin...@gmail.com> wrote:
>
> instead of
> From: syzbot <syzk...@googlegroups.com>
>
> make it
> From: syzbot-{hash} <syzk...@googlegroups.com>

That is probably good, but people drop the name all the time.

But with a very simple tweak, I think this would be close to perfect:
put the hash in the original email instead, and use "+" instead of
"-", because that's the standard email ("latter part doesn't matter").
Skip the proper name part entirely, since it doesn't really add
anything.

And then make it show up in the "Cc:" part or "Reported-by:", so it
would just look something like

Reported-by: syzkalle...@googlegroups.com

and you're all done. I think that would be easiest for us to track
(just one email that we're supposed to put in the commit message
_anyway_), and I hope/think that all email clients and servers will
still maintain the original full "syzkaller+xyz" part of the email, so
it doesn't get lost.

Because the "proper name" part definitely does get lost occasionally.
I'm not sure why it happens, but it definitely happens quite often.

Linus

[Eric Biggers]

On Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:

As others mentioned, allowing a bug ID to be in the fix's commit message,
perhaps in the Reported-by line which syzbot already suggests to include, would
make things a bit easier.

But I think the larger problem is that people in the community don't have any
visibility into the statuses of the bugs, so they don't have any motivation to
manage the statuses.

Are you planning to make a dashboard app publicly available for upstream kernel
bugs being tracked by syzbot? I think it would be very useful for the
community, especially for finding more details about a bug, e.g. when was it
last seen, how often was it seen, has it been seen in multiple trees. Also for
finding duplicates which may not have been sent to the correct mailing list.

syzbot also should be sending out reminders for bugs that are still open if the
crash is still occurring, and even moreso if there is a reproducer.

However, if the crash isn't still occurring, then I expect it will become
necessary to automatically invalidate the bug after some time, lest the list of
bugs grow without bound due to bugs that have already been fixed that no one has
time to debug to figure out exactly when/what the fix was, especially if there
is no reproducer. Or perhaps the bug was only in linux-next and only existed
due to a buggy patch which was dropped or modified before it reached mainline,
so there is no "fix" commit.

Eric

[Dmitry Vyukov]

Hi Linus,

This sound like a plan. I like that we piggy-back on the existing
process so _theoretically_ don't increase burden on developers. Also
may probably help improving general practice of properly crediting
people (which is now at around 50% from our experience). So new emails
now include (and that's also actual email sender):

====================
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+439688...@syzkaller.appspotmail.com
It will help syzbot understand when the bug is fixed. See footer for details.
If you forward the report, please keep this part and the footer.
====================

Thanks

[Dmitry Vyukov]

I've implemented this idea (with Linus improvements on top). Thanks
for the suggestion.

[Dmitry Vyukov]

Hi Stephan,

We've considered a similar idea, but there are 2 problems with it:
1. For some bugs syzbot doesn't have reproducers, so it simply can't
retest (though, more than half of them is still perfectly actionable,
e.g. LOCKDEP/KASAN reports contain enough information to rootcause,
some WARNINGs/BUGs/GPFs clearly point to simple issues like missed
input checks, off-by-ones, etc).
2. Lots of bugs are due to races and can't be reproduced with 100%
probability, or code can slightly change so the old reproducer doesn't
trigger the bug anymore. For racy bugs in the worst case syzbot will
close and remail the bug each day (which obviously won't be warmly
welcomed).

[Dmitry Vyukov]

That was my impression as well :)

> But, use what bugzilla does do well, provide a unique identifier that
> can be referenced in git commit messages to point people at the problem
> report.
>
> Why not generate a unique one per "problem" you find? And have a way to
> look them up somehow?
>
> Then you can put:
> syzkaller-id: XXXXXX
> in the commit log and you can track it easier?
>
> That's what all other "tools" do that try to track kernel bug reports.
> We do that for coverity as one such example, and lots of different bug
> trackers as well.

syzbot is capable of generating unique id's itself, and later we plan
to create an UI to look up bugs, etc (a-la bugzilla) because syzbot
does have all that info. So integrating with bugzilla won't directly
help to solve _this_ problem. We could upload them to bugzilla as
well, but taking into account amount of additional work, this is not
top priority right now.

Thanks

[Dmitry Vyukov]

Hi Eric,

Good question. I would very much like to open the UI, and I hope to do
it in near future, but we need to do some additional work to make it
possible. The good news is that information is already accumulating
and we can do pings, etc.

> syzbot also should be sending out reminders for bugs that are still open if the
> crash is still occurring, and even moreso if there is a reproducer.

Agree. The reasons why this hasn't happen yet are:
1. syzbot is being built up as it's running, I am overwhelmed with
hundreds of bugs and also doing lots of work which may be not directly
visible but important (e.g. improving quality of generated
reproducers, increasing percent of cases when reproducers are created,
improving bug title extraction logic, implementing patch testing by
request, now this new Reported-by-based process, etc).
2. Just sending an email for each open bug every week is simple, but I
afraid it won't be warmly welcomed. The open questions are: how
frequently syzbot should ping? should repro/no repro affect this? what
to do if it stopped happening? stopped happenning for how long? and
what if it happened just few times, so we can't really conclude if it
still happens or not (but we've seen very bad races manifesting this
way)? how should it interact with the following point?

> However, if the crash isn't still occurring, then I expect it will become
> necessary to automatically invalidate the bug after some time, lest the list of
> bugs grow without bound due to bugs that have already been fixed that no one has
> time to debug to figure out exactly when/what the fix was, especially if there
> is no reproducer. Or perhaps the bug was only in linux-next and only existed
> due to a buggy patch which was dropped or modified before it reached mainline,
> so there is no "fix" commit.

Good point. I think we will need to do this in some form in future.
Again open questions:
- what is the precise formula behind "isn't still occurring"?
- should we only close "no repro" bugs?
- should we re-test bugs with repro? (re-testing is not 100% precise,
so we will lose some real subtle bugs this way)

Thanks

[Ozgur]

28.12.2017, 13:41, "Dmitry Vyukov" <dvy...@google.com>:

Hello Dmitry,

I think not useful to be a GUI, for example it can be console based ui we can conenct and get information and fixed patches.
So syzbot is perfectly, I founded a patc last time :)

https://09738734946362323617.googlegroups.com/attach/3c6ef7059f77c/patch.txt?part=0.2&view=1&vt=ANaJVrFm49WFVkkKiomlnsrdfnv4P-0znjiC4agFB72ibq9_6iqg1rmZtw9-DxS5VvoOoKx8Ikl88sYEQQ45X0vjrwFkKDRaZELV-oU9DVmmrRAMSfStn24

And, I have a my suggestions:

Please keep to short url addresses and I think syzbot use to .txt file attached.
.txt is not good.

Ozgur

[Dmitry Vyukov]

Hi Ozgur,

We will do web UI first as it's something that's already partially
there and syzbot itself is not a console process, it's a cloud
service. It's also handy because there are lots of contextual
information and in a web UI one can just just click links to navigate
or download a blob. Later we could do an API for console clients, etc
if there is an interest in developing these types of UIs. But
generally UI is not the main business of syzbot, it's only a side
thing that helps it achieve the main goal, so it's doesn't have a team
of people assigned to it. But you are welcome to contribute, it's all
open-source:
https://github.com/google/syzkaller/tree/master/dashboard/app

> So syzbot is perfectly, I founded a patc last time :)
>
> https://09738734946362323617.googlegroups.com/attach/3c6ef7059f77c/patch.txt?part=0.2&view=1&vt=ANaJVrFm49WFVkkKiomlnsrdfnv4P-0znjiC4agFB72ibq9_6iqg1rmZtw9-DxS5VvoOoKx8Ikl88sYEQQ45X0vjrwFkKDRaZELV-oU9DVmmrRAMSfStn24
>
> And, I have a my suggestions:
>

> Please keep to short url addresses.

Well, that's an URL generated by google groups, we don't have control
over it. You also received the patch as an attachment in the syzbot
email.

> and I think syzbot use to .txt file attached.
> .txt is not good.

Why are not .txt attachments good? What do you propose to use?

Thanks

[Ozgur]

28.12.2017, 14:45, "Dmitry Vyukov" <dvy...@google.com>:

Hello,

> We will do web UI first as it's something that's already partially
> there and syzbot itself is not a console process, it's a cloud
> service. It's also handy because there are lots of contextual
> information and in a web UI one can just just click links to navigate
> or download a blob. Later we could do an API for console clients, etc
> if there is an interest in developing these types of UIs. But
> generally UI is not the main business of syzbot, it's only a side
> thing that helps it achieve the main goal, so it's doesn't have a team
> of people assigned to it. But you are welcome to contribute, it's all
> open-source:
> https://github.com/google/syzkaller/tree/master/dashboard/app

I understand.

>>  So syzbot is perfectly, I founded a patc last time :)
>>
>>  https://09738734946362323617.googlegroups.com/attach/3c6ef7059f77c/patch.txt?part=0.2&view=1&vt=ANaJVrFm49WFVkkKiomlnsrdfnv4P-0znjiC4agFB72ibq9_6iqg1rmZtw9-DxS5VvoOoKx8Ikl88sYEQQ45X0vjrwFkKDRaZELV-oU9DVmmrRAMSfStn24
>>
>>  And, I have a my suggestions:
>>
>>  Please keep to short url addresses.
>
> Well, that's an URL generated by google groups, we don't have control
> over it. You also received the patch as an attachment in the syzbot
> email.

I know, understand. sure.

>>  and I think syzbot use to .txt file attached.
>>  .txt is not good.
>
> Why are not .txt attachments good? What do you propose to use?

I think I'm misunderstood that is good to have text output in a file but not useful if the file extension is ".txt"
Not comfortable use it for mutt / vim and diff.

I think needs to be an new extension, would be like this ".log" or ".syz" :)

[Dmitry Vyukov]

On Thu, Dec 21, 2017 at 6:09 PM, Eric W. Biederman
<ebie...@xmission.com> wrote:
> The thing is syzbot sucks. It tells us things are wrong but not how to
> reproduce the problem. Apparently syzbot will test fixes, but that
> doesn't help when more information is needed to track down the problem.
>
> The long of the short of it is that I don't care about about bug reports
> that no one can reproduce and no human cares about. syzbot doesn't care
> in the sense of helping to fix things, that things are broken. syzbot
> just cries like a baby "It's broken! It's broken!"
>
> Further syzbot is written in a language (go) that switches which kernel
> thread things run in at arbitrary times. That is absolutely not
> productive to understanding what is happening when things break. I have
> heard too many complaints from container run-times that they can't make
> what should be a couple of line change but is completely non-trivial
> because someone choose go for their implementation language. Whatever
> benefits go has it is not a programming lanauge I would choose for fine
> and reproducible control of kernel interfaces.
>
> This in addition to syzbot needing the latest and greatest version of go
> which is not packaged in a handy form by my distro.
>
> Which in my experenience makes syzbot a whining crybaby that won't do
> anything to help and fights you when you try and get close.


Hi Eric,

Re reproducers: that's not completely true. syzbot aims at providing
reproducers for reported bugs, and you can see 140 bug reports with
reproducers here:
https://groups.google.com/forum/#!searchin/syzkaller-bugs/%22reproducer$20is$20attached%22%7Csort:date
Unfortunately, localizing kernel bugs is hard and is not possible in
all cases. The root cause of this is actually in the kernel itself,
not in syzbot. Things would be much simpler if we would work on a
single-threaded, deterministic user-space library. Then we would get
preceise reproducers in 100% of cases. But kernel is a concurrent,
parallel, non-deterimnistic system that constantly accumulates state.
We do try to incrementally improve percent of cases where syzbot
manages to create reproducers in general and C reproducers in
particular. But that will never be 100% due to the nature of the
tested system.

Also, you seem to dealt with a single hard case. From what I see over
lots of hundreds of reported bugs, in ~2/3 of cases it's actually
possible to localize the bug looking at the crash report only (I see
that developers frequently don't even run the reproducer when it's
present). For example, LOCKDEP/KASAN reports frequently contain enough
context information to rootcause, lots of WARNING/BUG/GPFs are due to
simple, shallow bugs like missed input check or off-by-one, etc. So I
think it would be a mistake to not report bugs without reproducers.
Even if there is no reproducer and it's a hard bug with no obvious
cause, it happened and it would be wrong to hide this information from
the world and pretend that nothing happened. But I understand that the
bar for fixing bugs without reproducers is generally higher.

I've looked at the case you dealt with ("proc_flush_task oops").
syzbot has provided a syzkaller reproducer for it, and I was in fact
able to reproduce the crash running the reproducer. What happened
there is that reproducing the crash took ~15-20 mins, syzbot got a
lucky coin once when trying syzkaller program, but then when it tested
the corresponding C program it did not trigger the crash within
allotted time. In such cases syzbot decided to not mis-inform you that
the C program triggers the crash. Each report it provides is actual
kernel output obtained on a freshly booted machine running exact
reproducer it provides on exact kernel commit and config.

Re Go (implementation language): this is not true. The part of
syzkaller that actually executes syscalls is written in C++ from day
one. You can see the code here:
https://github.com/google/syzkaller/tree/master/executor
It does explicit, manual thread scheduling; compiled as static binary
to avoid any variance due to dynamic loading; does not use C++ runtime
support nor malloc to avoid unexpected mmap calls. This is mostly for
the reasons you outlined.

Re version of Go: yes, that's unfortunate. But there is no way we can
change this with limited human resources and without subscribing to
constant flow of maintanance work. Distros should provide more
up-to-date packages. For example, version of gcc that my distro
provides (4.8.4, about 5-years old) can't compile for arm at all. It
just can't produce binaries, compiler and assembler don't agree on
units of alignment directives. I don't see how we can realistically
work around cumulative amount of bugs in software over the last 5
years. Fortunately, obtaining a fresh Go toolchain boils down to
unpacking an archive from https://golang.org/dl/.

Please don't draw too broad conclusions from one/few negative cases
that you hit. syzkaller has found 1000+ real bugs in kernels. We are
doing our best. Problem domain is hard.

Thank you

[Dmitry Vyukov]

On Thu, Dec 28, 2017 at 1:26 PM, Ozgur <oz...@goosey.org> wrote:

>>> and I think syzbot use to .txt file attached.
>>> .txt is not good.
>>
>> Why are not .txt attachments good? What do you propose to use?
>
> I think I'm misunderstood that is good to have text output in a file but not useful if the file extension is ".txt"
> Not comfortable use it for mutt / vim and diff.
>
> I think needs to be an new extension, would be like this ".log" or ".syz" :)

There is an fortunate limitation in the mailing system we currently
use -- it infers Content-Type from file extension. So if we do .syz,
it will do application/octet-stream.

[Ozgur]

28.12.2017, 15:30, "Dmitry Vyukov" <dvy...@google.com>:

Hello Dmitry,

I understand and thanks for detailed info, like it!

Ozgur

[Dmitry Vyukov]

Double-checking the docs:
https://cloud.google.com/appengine/docs/standard/go/mail/mail-with-headers-attachments

We can do .diff extension for patches. .diff is a standard extension,
looks reasonable for patches and will be a one-line change.
Does .diff sound better than .txt for patches?