Why is this Syzbot issue still marked as open, and not automatically closed?

[Theodore Tso]

https://syzkaller.appspot.com/bug?id=4a547542bb29dc957c096f0c95ef9154e93d68d3

[Aleksandr Nogikh]

The fixing commit has not reached two syzbot instances yet. You can
see it on the top of the page: Patched on: [...], missing on:
[ci-qemu2-riscv64 ci-upstream-bpf-next-kasan-gce]. Once there are no
such instances, it will close automatically.

With ci-qemu2-riscv64 the problem is that it's broken now (doesn't
boot), so syzbot doesn't consider new commits from it now. It will do
it once the instance is up again.
With ci-upstream-bpf-next-kasan-gce it's more strange - the fixing
commit is indeed not present on the branch, just checked manually.

On Mon, May 16, 2022 at 10:00 AM 'Theodore Tso' via syzkaller
<syzk...@googlegroups.com> wrote:
>
> https://syzkaller.appspot.com/bug?id=4a547542bb29dc957c096f0c95ef9154e93d68d3
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/f6c5fd81-c472-432a-a5f0-eff06a75f583n%40googlegroups.com.

[Theodore Tso]

Thanks for the clarification.

The bpf-next branch is like most subsystem branches.   It's based on 5.18-rc3, and is accumulating patches until the merge window opens (when Linus releases 5.18 probably in a week or so).    At that point, they will send a pull request to Linus, and then at some point, they will update the bpf-next branch so that it is based on 5.19-rc[2-4].    That's roughly a month from now.    So any fixes that might have landed in Linus's upstream tree post 5.18-rc3 won't be closed with respect to Syzkaller's dashboard until bpf-next gets updated to a 5.19-rcX.

This also implies that if a fix lands in 5.18-rc4, it will stay open in Syzkaller for almost 3 months.    This means that "bug lifetime" statistics and graphs are going to be a bit misleading, and scarier than things really are in reality.    It also means that it's super-annoying for an upstream developer who is trying to figure out which syzkaller bugs are still worth working on.

Is there a way to filter reports so it only includes issues that are not fixed on Linux upstream (good) or Linux-next (best)?

Thanks,

-- Ted

[Aleksandr Nogikh]

On Mon, May 16, 2022 at 5:29 PM Theodore Tso <ty...@google.com> wrote:
>
> Thanks for the clarification.
>
> The bpf-next branch is like most subsystem branches. It's based on 5.18-rc3, and is accumulating patches until the merge window opens (when Linus releases 5.18 probably in a week or so). At that point, they will send a pull request to Linus, and then at some point, they will update the bpf-next branch so that it is based on 5.19-rc[2-4]. That's roughly a month from now. So any fixes that might have landed in Linus's upstream tree post 5.18-rc3 won't be closed with respect to Syzkaller's dashboard until bpf-next gets updated to a 5.19-rcX.
>
> This also implies that if a fix lands in 5.18-rc4, it will stay open in Syzkaller for almost 3 months. This means that "bug lifetime" statistics and graphs are going to be a bit misleading, and scarier than things really are in reality. It also means that it's super-annoying for an upstream developer who is trying to figure out which syzkaller bugs are still worth working on.
>
> Is there a way to filter reports so it only includes issues that are not fixed on Linux upstream (good) or Linux-next (best)?

At least on the dashboard (https://syzkaller.appspot.com/upstream)
there's a separate section ("fix pending") for all bugs for which at
least one fixing commit is present (in any Linux tree we fuzz). Bugs
in all other sections are not fixed anywhere yet (to the best of
syzbot's knowledge).

Or did you not mean the dashboard's main page?

[Theodore Tso]

I did mean the dashboard's main page.   What had happened was I was trying to find ext4-related syzkaller bugs.  This is a bit non-trivial since sometimes the stack trace might reference ext4, and it might also be an ext4 bug, but the Syzbot title would not necessarily contain ext4 as a substring at all.   That's a long-standing problem which I've complained about in the past.  Ignoring that issue, the problem facing the upstream subsystem maintainer is that the "Open" section has 876 lines, most of which are not of interest to the maintainer.  So the maintainer, not being able to narrow down the search, will use the browser's "search in page" to look for a substring like "ext4".    (Of course, that won't catch everything due to the title being non-optimal for filtering, but it's better than nothing.)

But the problem with doing the searching in the browser is that it's not obvious when the search has landed you in the "Fix pending" section of the page.  In fact, until you pointed it out, I didn't even realize that there were different sections such as "Open", "Moderation" (what does that mean, by the way?), and "Fix pending".

Which is why it would be really nice if there was some kind of advanced filtering system, such as what is available for GMail, or at systems such as Google's Issue Tracker (https://issuetracker.google.com) so that upstream maintainers could easily find those open Syzkaller issues that they care about.    Failing that, maybe there could be separate pages for the various subsections on the dashboard's main page, so that browser's search-in-page could work a bit better?   Or maybe extracting the dashboard information as a CSV so that we can use grep and awk and perl if it's too hard to make a web page that can do that kind of filtering ala GMail or Issue Tracker?

Cheers,

-- Ted

[Aleksandr Nogikh]

Thank you very much for the feedback!

That's all true, the dashboard could do a much better job in bug
searching/grouping/filtering :(

On Thu, May 19, 2022 at 10:09 PM Theodore Tso <ty...@google.com> wrote:
>
> I did mean the dashboard's main page. What had happened was I was trying to find ext4-related syzkaller bugs. This is a bit non-trivial since sometimes the stack trace might reference ext4, and it might also be an ext4 bug, but the Syzbot title would not necessarily contain ext4 as a substring at all. That's a long-standing problem which I've complained about in the past. Ignoring that issue, the problem facing the upstream subsystem maintainer is that the "Open" section has 876 lines, most of which are not of interest to the maintainer. So the maintainer, not being able to narrow down the search, will use the browser's "search in page" to look for a substring like "ext4". (Of course, that won't catch everything due to the title being non-optimal for filtering, but it's better than nothing.)
>
> But the problem with doing the searching in the browser is that it's not obvious when the search has landed you in the "Fix pending" section of the page. In fact, until you pointed it out, I didn't even realize that there were different sections such as "Open", "Moderation" (what does that mean, by the way?), and "Fix pending".

Re. "Moderation" - these are now mostly KCSAN bugs waiting for manual
moderation + some potentially flaky bugs for which syzbot either waits
until they happen enough times or until they get a repro.

>
> Which is why it would be really nice if there was some kind of advanced filtering system, such as what is available for GMail, or at systems such as Google's Issue Tracker (https://issuetracker.google.com) so that upstream maintainers could easily find those open Syzkaller issues that they care about. Failing that, maybe there could be separate pages for the various subsections on the dashboard's main page, so that browser's search-in-page could work a bit better? Or maybe extracting the dashboard information as a CSV so that we can use grep and awk and perl if it's too hard to make a web page that can do that kind of filtering ala GMail or Issue Tracker?

To be honest, I don't think that the implementation of an advanced
filtering system is a question of the nearest future. But I'll try to
implement and deploy next week something that could improve the
situation for your use case.

[Aleksandr Nogikh]

FYI now all bugs with fixing commits are moved away -- they are no
longer on the main page (you can find them in the "Fixed" section, if
you need to). Hopefully it will make it all less confusing.

Re. bug filtering -- I think it might appear in one month or two
(depending on my load).
By the way, in the meanwhile it would help if you shared what kind of
search queries maintainers might need to make? As I understand, at the
bare minimum the syzbot dashboard should be able to perform full-text
search among bug reports and filter them depending on bug status, date
of appearance and the source code trees on which it was detected. Is
there anything else that you'd be interested in?

--
Best Regards,
Aleksandr

[Theodore Tso]

The bugfiltering sounds awesome. The search terms you've suggested looks good.  As far as date of appearance, it would be good to have date first appeared and date last appeared.  I'd also add number of reproductions, and kernel version as well as source code trees on which it has been reproduced.    (The date of appearance is a rough proxy for kernel version, but it's a lot more convenient to be able to say things like kernel_version < 5.10 or kernel_version > 5.14.)

If the dashboard could add an indication of how easy-to-reproduce a particular failure, that would be helpful.   e.g., if you need to run the reproducer N times, or at least M minutes in order to trigger, or whether it fails on the very first loop, that would be super-handy.   Once we had that, that would also be a good term to be able to search on.

Cheers,

-- Ted