Can we specify repro to use for "#syz test" request?

[Tetsuo Handa]

Hello.

For example https://syzkaller.appspot.com/bug?extid=881d65229ca4f9ae8c84 indicates
at least 5 different bugs. But since different reproducers are discovered for
different bugs within one report, multiple reproducers for multiple bugs are put
into one report.

Since some trees do not frequently rebase/merge trees containing patches, a bug
report takes long time before that bug report is closed and a new bug report with
the same title is opened. I want to make it possible to fix multiple bugs within
one bug report. To fix multiple bugs within one bug report, it is important to be
able to test proposed patches with intended reproducer that can examine the proposed
patches.

[Aleksandr Nogikh]

Hi Tetsuo,

On Wed, Nov 12, 2025 at 11:31 AM Tetsuo Handa
<penguin...@i-love.sakura.ne.jp> wrote:
>
> Hello.
>
> For example https://syzkaller.appspot.com/bug?extid=881d65229ca4f9ae8c84 indicates
> at least 5 different bugs. But since different reproducers are discovered for
> different bugs within one report, multiple reproducers for multiple bugs are put
> into one report.

Normally, it'd indicate a problem in our bug report parsing logic
(since we expect syzbot not to keep different bugs in one bucket), but
the "unregister_netdevice: waiting for DEV to become free" seems a bit
special. Do you see some reliable pattern that would still help us
automatically both split these reports into several buckets and avoid
creating too many unrelated buckets that we'd need to #syz dup
manually?

Instead of "unregister_netdevice: waiting for DEV to become free", we
could probably have e.g. "unregister_netdevice: waiting for vxcanNUM
to become free", "unregister_netdevice: waiting for netdevsimNUM to
become free", etc., but I fear that the underlying bugs will often
span several device names, so we'd drown in duplicates.

>
> Since some trees do not frequently rebase/merge trees containing patches, a bug
> report takes long time before that bug report is closed and a new bug report with
> the same title is opened. I want to make it possible to fix multiple bugs within
> one bug report. To fix multiple bugs within one bug report, it is important to be
> able to test proposed patches with intended reproducer that can examine the proposed
> patches.
>

Currently, there's no way to specify this to syzbot unfortunately.
I've noted your request in our related backlog issue:
https://github.com/google/syzkaller/issues/1611

In the meantime, I think it may be okay to just close the issue with
one particular fix commit and letting syzbot re-report the bug again.

--
Aleksandr

[Tetsuo Handa]

On 2025/11/12 22:05, Aleksandr Nogikh wrote:
> Hi Tetsuo,
>
> On Wed, Nov 12, 2025 at 11:31 AM Tetsuo Handa
> <penguin...@i-love.sakura.ne.jp> wrote:
>>
>> Hello.
>>
>> For example https://syzkaller.appspot.com/bug?extid=881d65229ca4f9ae8c84 indicates
>> at least 5 different bugs. But since different reproducers are discovered for
>> different bugs within one report, multiple reproducers for multiple bugs are put
>> into one report.
>
> Normally, it'd indicate a problem in our bug report parsing logic
> (since we expect syzbot not to keep different bugs in one bucket), but
> the "unregister_netdevice: waiting for DEV to become free" seems a bit
> special.

Yes, this one is a bit special.

> Do you see some reliable pattern that would still help us
> automatically both split these reports into several buckets and avoid
> creating too many unrelated buckets that we'd need to #syz dup
> manually?

No, unfortunately.

>
> Instead of "unregister_netdevice: waiting for DEV to become free", we
> could probably have e.g. "unregister_netdevice: waiting for vxcanNUM
> to become free", "unregister_netdevice: waiting for netdevsimNUM to
> become free", etc., but I fear that the underlying bugs will often
> span several device names, so we'd drown in duplicates.

I considered not aggregating into "DEV", for the number of devices
initialize_netdevices() in C reproducer creates is limited.

>
>>
>> Since some trees do not frequently rebase/merge trees containing patches, a bug
>> report takes long time before that bug report is closed and a new bug report with
>> the same title is opened. I want to make it possible to fix multiple bugs within
>> one bug report. To fix multiple bugs within one bug report, it is important to be
>> able to test proposed patches with intended reproducer that can examine the proposed
>> patches.
>>
>
> Currently, there's no way to specify this to syzbot unfortunately.
> I've noted your request in our related backlog issue:
> https://github.com/google/syzkaller/issues/1611

Thank you.

>
> In the meantime, I think it may be okay to just close the issue with
> one particular fix commit and letting syzbot re-report the bug again.
>

Since syzbot knows the git tree for a crash, syzbot should be able to check
whether the fix commit is already in the git tree where that crash happened.
Therefore, the most helpful approach for me is to update syzbot to be able to
create a new report when the fix commit is already in the git tree where the
crash was reported.

[Aleksandr Nogikh]

Syzbot indeed monitors all git trees where a particular crash has been
detected and, once the fix commit it was told about is present in all
of them, the bug is closed. If the crash keeps on happening, it will
be re-opened the moment it appears again after the previous bug is
"closed".

[Tetsuo Handa]

On 2025/11/18 18:46, Aleksandr Nogikh wrote:
>> Since syzbot knows the git tree for a crash, syzbot should be able to check
>> whether the fix commit is already in the git tree where that crash happened.
>> Therefore, the most helpful approach for me is to update syzbot to be able to
>> create a new report when the fix commit is already in the git tree where the
>> crash was reported.
>>
>
> Syzbot indeed monitors all git trees where a particular crash has been
> detected and, once the fix commit it was told about is present in all
> of them, the bug is closed. If the crash keeps on happening, it will
> be re-opened the moment it appears again after the previous bug is
> "closed".

I know. The problem for me is that it takes very long time before a bug is closed.
It is too slow for me to wait until the fix commit arrives at all trees which
syzbot is monitoring. My comment is that syzbot should be able to open a new bug if
a crash was reproduced on a git tree where the fix commit already has arrived at.

[Aleksandr Nogikh]

Ah, now I see what you mean. Thanks for the clarification.
FWIW we've had a similar (though not completely related) discussion in
https://github.com/google/syzkaller/issues/3150.

Overall, this sounds reasonable to me, though I am a bit worried about
how it will fit together with other syzbot dashboard functionality.
E.g. currently, closing a bug is an irreversible operation, so we can
safely open an (n+1) reincarnation of the bug once it happens again.
However, once one specifies `#syz fix` and until the moment the fix
reaches all trees, the process can be reversed at any time (there's a
`#syz unfix` command).

What if one specifies an incorrect/incomplete #syz fix, the bug keeps
on happening even on the patched instances, we open an (n+1) version
of the issue in parallel, but before the fix patch reaches all trees,
a `#syz unfix` command is sent. Now we have two open bugs for the same
issue.

[Tetsuo Handa]

On 2025/11/18 19:44, Aleksandr Nogikh wrote:
> Ah, now I see what you mean. Thanks for the clarification.
> FWIW we've had a similar (though not completely related) discussion in
> https://github.com/google/syzkaller/issues/3150.
>
> Overall, this sounds reasonable to me, though I am a bit worried about
> how it will fit together with other syzbot dashboard functionality.
> E.g. currently, closing a bug is an irreversible operation, so we can
> safely open an (n+1) reincarnation of the bug once it happens again.
> However, once one specifies `#syz fix` and until the moment the fix
> reaches all trees, the process can be reversed at any time (there's a
> `#syz unfix` command).
>
> What if one specifies an incorrect/incomplete #syz fix, the bug keeps
> on happening even on the patched instances, we open an (n+1) version
> of the issue in parallel, but before the fix patch reaches all trees,
> a `#syz unfix` command is sent. Now we have two open bugs for the same
> issue.

When a problem is found, a title for that problem is assigned for a report.
When the same problem is found after a report for that title was already
closed, a new report for the same problem gets the same title plus number.
Therefore, the algorithm we need is nothing but finding minimum number from
the array of reports with the same title.

WARNING in foo (5) status: open
WARNING in foo (4) status: open fixed_in: tree1
WARNING in foo (3) status: open fixed_in: tree1, tree2
WARNING in foo (2) status: closed fixed_in: all trees
WARNING in foo status: closed fixed_in: all trees

If a problem that gets "WARNING in foo" is found in tree1, that problem is
reported as "WARNING in foo (5)".
If a problem that gets "WARNING in foo" is found in tree2, that problem is
reported as "WARNING in foo (4)".
If a problem that gets "WARNING in foo" is found in tree3, that problem is
reported as "WARNING in foo (3)".

After "#syz unfix" was sent on "WARNING in foo (4)", the problem that gets
"WARNING in foo" found in tree1 will be reported as "WARNING in foo (4)"
instead of "WARNING in foo (5)".

[Aleksandr Nogikh]

It assumes that no new bug with the same manifestation can be
introduced to the tree3 before tree3 receives the fixes for the
previous bugs causing "WARNING in foo", right?

If the initial fix is incomplete and there are actually bug3, bug4,
and bug5 that cause "WARNING in foo", "WARNING in foo (3)" ends up
recording crashes caused by bug3, bug4, and bug5; "WARNING in foo (4)"
- crashes due to bug4, and bug5; "WARNING in foo (5)" - only crashes
due to bug5. Which is not necessarily bad, but it's still something to
keep in mind: these groups of crashes won't be perfectly separate.

>
> After "#syz unfix" was sent on "WARNING in foo (4)", the problem that gets
> "WARNING in foo" found in tree1 will be reported as "WARNING in foo (4)"
> instead of "WARNING in foo (5)".
>

In this case, "WARNING in foo (4)" and "WARNING in foo (5)" become duplicates:
- There are two reports to the public mailing lists, which increases
the chance of separate discussions about the same issue, and just
creates more confusion overall.
- If someone sends another #syz fix for `(4)` without closing `(5)`,
and the bug reappears, then new crashes will just go to `(5)`, and
1) we will end up with a collection of unrelated crashes again;
2) no new report is sent to the mailing lists.

It can of course be solved somehow, I'm just worried about the total
number of corner cases we must consider here.

[Tetsuo Handa]

On 2025/11/27 7:11, Aleksandr Nogikh wrote:
> It assumes that no new bug with the same manifestation can be
> introduced to the tree3 before tree3 receives the fixes for the
> previous bugs causing "WARNING in foo", right?
>
> If the initial fix is incomplete and there are actually bug3, bug4,
> and bug5 that cause "WARNING in foo", "WARNING in foo (3)" ends up
> recording crashes caused by bug3, bug4, and bug5; "WARNING in foo (4)"
> - crashes due to bug4, and bug5; "WARNING in foo (5)" - only crashes
> due to bug5. Which is not necessarily bad, but it's still something to
> keep in mind: these groups of crashes won't be perfectly separate.

If a fix which was already applied to tree1 is incomplete,
the same manifestation in tree1 is reported to "WARNING in foo (5)".

If a fix which was already applied to tree2 is incomplete,
the same manifestation in tree2 is reported to "WARNING in foo (4)".

If the same manifestation happened in a tree where a fix is not yet applied,
it is reported to "WARNING in foo (3)".

>
>>
>> After "#syz unfix" was sent on "WARNING in foo (4)", the problem that gets
>> "WARNING in foo" found in tree1 will be reported as "WARNING in foo (4)"
>> instead of "WARNING in foo (5)".
>>
>
> In this case, "WARNING in foo (4)" and "WARNING in foo (5)" become duplicates:
> - There are two reports to the public mailing lists, which increases
> the chance of separate discussions about the same issue, and just
> creates more confusion overall.
> - If someone sends another #syz fix for `(4)` without closing `(5)`,
> and the bug reappears, then new crashes will just go to `(5)`, and
> 1) we will end up with a collection of unrelated crashes again;
> 2) no new report is sent to the mailing lists.

That is not a big problem. Bug reports with no activity and no occurrence
will be automatically closed. That is, "WARNING in foo (5)" (which became
a duplicated of "WARNING in foo (4)" due to "#syz unfix" request) will be
automatically closed. (By the way, is "#syz unfix" used so frequently?)

>
> It can of course be solved somehow, I'm just worried about the total
> number of corner cases we must consider here.

What is important is that current "bug report opening rule" tends to fail to
figure out that already applied patches are incomplete. Once a developer sends
a patch for one bug report, that developer likely forgets about that bug report.
It likely takes at least 10 weeks before that developer is notified of the problem
being still there. (I think I am a rare developer who is monitoring not-yet-closed
but no-longer-open bug reports.) This is a side-effect of "don't close but don't
notify" until a fix is applied to all trees.

There might be projects where a bug report is not closed until a fix is applied
to all not-yet-EOL versions, but I guess that it won't take many weeks until a
fix is applied to all not-yet-EOL versions. (If it takes many weeks, they won't
be able to remember, and they likely use per-version child reports for tracking
purpose.) We can consider a git tree as a kind of version, and syzbot is managing
status for all versions using one report.

In addition to that, current situation that we can't specify which reproducer to
use for "#syz test" request makes it difficult to test patches for different
causes when multiple different causes are aggregated into one bug report.

If syzbot starts opening new bug reports, developers can figure out that already
applied patches are incomplete much earlier, and I am likely able to specify
which reproducer to use for "#syz test" request.

[Tetsuo Handa]

On 2025/11/27 19:26, Tetsuo Handa wrote:
> If syzbot starts opening new bug reports, developers can figure out that already
> applied patches are incomplete much earlier, and I am likely able to specify
> which reproducer to use for "#syz test" request.
>

Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf includes
a slide

Goal

Reduce bugs/release 100x: 20'000 -> 200

which I think that it is important to fix as many bugs as possible before
a buggy change reaches linux.git tree. A previous slide ("Bug Forking")
explains bugs are propagated to forked git trees, but we have a reversed
propagation where individual git trees gathers into linux.git tree.

Current "bug report opening rule" is needlessly waiting until a fix commit is
propagated to all individual git trees via linux.git tree. If a buggy commit is
only in one individual git tree, there is no need to wait until a fix commit for
that buggy commit propagates to all individual git trees via linux.git tree (I
think that is why https://github.com/google/syzkaller/issues/3150 was opened).

It is important to teach developers that the fix commit was incomplete before
a buggy commit and an incomplete fix commit propagates to all individual git trees
via linux.git tree, for the "Goal" is to reduce bugs before linux.git tree creates
a release.

When syzbot dashboard was published, most bugs were already propagated to all
individual git trees which syzbot was monitoring. But nowadays, some bugs are
getting reported in only one or two git trees, which is before buggy commits are
propagated to all individual git trees which syzbot is monitoring. Therefore,
"notify developers of incomplete fix commits" is valuable than worrying about
"#syz unfix" requests.