Debian kernel testing on syzbot

[Dmitry Vyukov]

Hello,

Our team works on syzkaller/syzbot kernel fuzzer:
https://github.com/google/syzkaller
https://github.com/google/syzkaller/blob/master/docs/syzbot.md

Currently we test the upstream kernel and recent LTS releases:
https://syzkaller.appspot.com/upstream
https://syzkaller.appspot.com/linux-6.1
https://syzkaller.appspot.com/linux-5.15
and report bugs to upstream developers:
https://groups.google.com/g/syzkaller-bugs

Due to Debian's relevance as one of the most widely used Linux
distributions, we plan to test the Debian kernel as well.

We were thinking about testing the "testing" release only initially.
Or do you have other suggestions here?

Do you want bugs to be reported privately first (to some closed
mailing list) with some embargo? Or do we make them public (visible on
syzbot dashboard) right away as we do for upstream/LTS?

[Dmitry Vyukov]

+Ben, you were pointed out as the person to provide "the official" response :)

To clarify: we are not asking nor imply that anybody will actually act
in any way on the reported bugs. I mean anybody is welcome to, but
don't have to.
We can also just create a public web dashboard (+new opt-in mailing
list), if that's what we agree on here.

And if there is an active interest in acting on the reports, we can
also test the unstable release (that's the better place to fix,
right).

[Dmitry Vyukov]

Ok, we will probably proceed with testing with no notifications for now.

[Bo YU]

Hi

Sorry, I'm sorry that I suddenly ran into this topic.
Personally, I think the sid(unstable) kernel is more appropriate for syzkaller.
Unstable by itself means unstable, if we have any issue we can fix it quickly.
Testing needs to be migrated from sid, which may cast a long time. In addition,
it is closer to upstream.

BTW, as a Debian RISC-V porter, I am wondering the syzkaller if is arch-indep
or not. You know Debian supports many architectures. If sure, I hope syzkaller
can support Debian/riscv64(yeah, it is in sid now, but it will be
entered into testing
soon).

Thanks,

BR,
Bo

[Dmitry Vyukov]

Hi Bo,

Thanks for your reply.
If there will be active interest in looking at the report and fixing
them, then we can set up unstable testing as well.

> BTW, as a Debian RISC-V porter, I am wondering the syzkaller if is arch-indep
> or not. You know Debian supports many architectures. If sure, I hope syzkaller
> can support Debian/riscv64(yeah, it is in sid now, but it will be
> entered into testing
> soon).

Per se syzkaller is completely arch-independent.
It may lack descriptions for some arch-specific kernel interfaces
(like arch_prctl's, arch-specific KVM ioctl's, etc). But there are
usually few of them.

However, on our testing infra we can test riscv only using qemu tcg
emulation, which is extremely slow. So the riscv instance we have just
mostly rediscovers and traps on arch-independent bugs that all other
faster instances find much easier.
Here is the list of bugs currently happening on the riscv instance:
https://syzkaller.appspot.com/upstream?manager=ci-qemu2-riscv64
And here is list of bugs that happens _only_ on riscv instance and not
on any other one:
https://syzkaller.appspot.com/upstream?only_manager=ci-qemu2-riscv64

[Bo YU]

Hi!

Thanks. But I am not a member of kernel team(just a kernel newbie) and
please follow your initial decision. Personally, I'm probably more concerned
about the bugs in the RISC-V kernel, so this may add to the burden of
the kernel team.

>
> > BTW, as a Debian RISC-V porter, I am wondering the syzkaller if is arch-indep
> > or not. You know Debian supports many architectures. If sure, I hope syzkaller
> > can support Debian/riscv64(yeah, it is in sid now, but it will be
> > entered into testing
> > soon).
>
> Per se syzkaller is completely arch-independent.
> It may lack descriptions for some arch-specific kernel interfaces
> (like arch_prctl's, arch-specific KVM ioctl's, etc). But there are
> usually few of them.
>
> However, on our testing infra we can test riscv only using qemu tcg
> emulation, which is extremely slow. So the riscv instance we have just
> mostly rediscovers and traps on arch-independent bugs that all other
> faster instances find much easier.
> Here is the list of bugs currently happening on the riscv instance:
> https://syzkaller.appspot.com/upstream?manager=ci-qemu2-riscv64
> And here is list of bugs that happens _only_ on riscv instance and not
> on any other one:
> https://syzkaller.appspot.com/upstream?only_manager=ci-qemu2-riscv64

Thank you very much for this valuable information which is what I was
looking for, thanks! I hope I can do something for it.

BR,
Bo

[Ben Hutchings]

On Wed, 2023-06-28 at 10:26 +0200, Dmitry Vyukov wrote:
> On Thu, 22 Jun 2023 at 16:46, Dmitry Vyukov <dvy...@google.com> wrote:
> >
> > Hello,
> >
> > Our team works on syzkaller/syzbot kernel fuzzer:
> > https://github.com/google/syzkaller
> > https://github.com/google/syzkaller/blob/master/docs/syzbot.md
> >
> > Currently we test the upstream kernel and recent LTS releases:
> > https://syzkaller.appspot.com/upstream
> > https://syzkaller.appspot.com/linux-6.1
> > https://syzkaller.appspot.com/linux-5.15
> > and report bugs to upstream developers:
> > https://groups.google.com/g/syzkaller-bugs
> >
> > Due to Debian's relevance as one of the most widely used Linux
> > distributions, we plan to test the Debian kernel as well.
> >
> > We were thinking about testing the "testing" release only initially.
> > Or do you have other suggestions here?

If you want to find issues affecting the next release, then that's the
right choice. But if you want to find issues that still need fixes
uploaded, then "unstable" is the right choice. Any fixes in testing
need to go via unstable.

> > Do you want bugs to be reported privately first (to some closed
> > mailing list) with some embargo? Or do we make them public (visible on
> > syzbot dashboard) right away as we do for upstream/LTS?
>
> +Ben, you were pointed out as the person to provide "the official" response :)

I'm just one person on the kernel team, and not the most active at the
moment. Salvatore Bonaccorso is doing most of the security updates.

> To clarify: we are not asking nor imply that anybody will actually act
> in any way on the reported bugs. I mean anybody is welcome to, but
> don't have to.
> We can also just create a public web dashboard (+new opt-in mailing
> list), if that's what we agree on here.
>
> And if there is an active interest in acting on the reports, we can
> also test the unstable release (that's the better place to fix,
> right).

If syzbot is able to distinguish bugs that are reproducible on Debian
patched kernels but not in the corresponding stable releases, I think
that would be very useful to us. My guess is that this would be a
manageable rate of bugs and we could receive those privately. What do
you think, Salvatore?

If this isn't possible, then it's unlikely we will have the time to
look at the issues. You can create a public web dashboard but I don't
know if that's going to help anyone.

Ben.

--
Ben Hutchings
Theory and practice are closer in theory than in practice - John Levine

[Salvatore Bonaccorso]

Hi,

Yes, I do agree with the above. If syzbot can do that separation then
it would be useful, and think we can agree to get those reports
privately to either a deidcated set up distribution list or directly
to the Debian maintainers. In first stance I guess that would be Ben
and me, and possibly the others listed as Uploaders for the src:linux
package.

OTOH, I fully agree, if syzbot canot distinguish issues reproducible
only with Debian patches applied I think we won't really have the free
time to investigate those reports. As we are doing this in our free
time.

Thank you for approaching us on this!

Regards,
Salvatore

[Dmitry Vyukov]

Hi Ben, Salvatore,

Yes, syzbot can detect "downstream" bugs with reasonable precision,
e.g. for Android 5.15 tree:
https://syzkaller.appspot.com/android-5-15?label=origin%3Adownstream
and here you can see "Bug presence" analysis by testing on
corresponding LTS and upstream trees:
https://syzkaller.appspot.com/bug?extid=ea487d1ec1a25689e4d2

However, I am not sure if we can readily set up reporting of only such
bugs privately.