KMSAN: uninit-value in memcmp (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 74ee2200b89f kmsan: bump .config.example to v4.17-rc3
git tree: https://github.com/google/kmsan.git/master
console output: https://syzkaller.appspot.com/x/log.txt?x=11bcd7e7800000
kernel config: https://syzkaller.appspot.com/x/.config?x=4ca1e57bafa8ab1f
dashboard link: https://syzkaller.appspot.com/bug?extid=d3402c47f680ff24b29c
compiler: clang version 7.0.0 (trunk 329391)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=102533c7800000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=156dab5b800000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+d3402c...@syzkaller.appspotmail.com

random: sshd: uninitialized urandom read (32 bytes read)
random: sshd: uninitialized urandom read (32 bytes read)
random: sshd: uninitialized urandom read (32 bytes read)
IPVS: ftp: loaded support on port[0] = 21
==================================================================
BUG: KMSAN: uninit-value in memcmp+0x119/0x180 lib/string.c:861
CPU: 0 PID: 38 Comm: kworker/0:1 Not tainted 4.17.0-rc3+ #88
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x185/0x1d0 lib/dump_stack.c:113
kmsan_report+0x142/0x240 mm/kmsan/kmsan.c:1067
__msan_warning_32+0x6c/0xb0 mm/kmsan/kmsan_instr.c:683
memcmp+0x119/0x180 lib/string.c:861
__hw_addr_add_ex net/core/dev_addr_lists.c:61 [inline]
__dev_mc_add+0x1fc/0x900 net/core/dev_addr_lists.c:670
dev_mc_add+0x6d/0x80 net/core/dev_addr_lists.c:687
igmp6_group_added+0x2db/0xa00 net/ipv6/mcast.c:662
ipv6_dev_mc_inc+0xe9e/0x1130 net/ipv6/mcast.c:914
addrconf_join_solict net/ipv6/addrconf.c:2103 [inline]
addrconf_dad_begin net/ipv6/addrconf.c:3853 [inline]
addrconf_dad_work+0x462/0x2a20 net/ipv6/addrconf.c:3979
process_one_work+0x12c6/0x1f60 kernel/workqueue.c:2145
worker_thread+0x113c/0x24f0 kernel/workqueue.c:2279
kthread+0x539/0x720 kernel/kthread.c:239
ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:412

Local variable description: ----buf@igmp6_group_added
Variable was created at:
igmp6_group_added+0x4a/0xa00 net/ipv6/mcast.c:650
ipv6_dev_mc_inc+0xe9e/0x1130 net/ipv6/mcast.c:914
==================================================================


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

[syzbot]

> #syz dup: KMSAN: uninit-value in memcmp

Can't dup bug to a bug in different reporting (upstream->internal).Please
dup syzbot bugs only onto syzbot bugs for the same kernel/reporting.

[Vladis Dronov]

#syz fix: net: fix uninit-value in __hw_addr_add_ex()

[Dmitry Vyukov]

On Sun, Sep 23, 2018 at 6:02 PM, Vladis Dronov <vdr...@redhat.com> wrote:
> #syz fix: net: fix uninit-value in __hw_addr_add_ex()

Hi Vladis,

This can be fixed with "net: fix uninit-value in __hw_addr_add_ex()".
That commit landed in April, syzbot waited till the commit reached all
tested trees, and then closed the bug.
But the similar bug continued to happen, so syzbot created second
version of this bug (2). You can see on dashboard that the last crash
for the second version (2) happened just few days ago. So this is a
different bug.

[Vladis Dronov]

Hello, Dmirty,

Thank you for the reply. Can we please, discuss this further?

> You can see on dashboard that the last crash
> for the second version (2) happened just few days ago. So this is a
> different bug.

Well... yes and no. When I was looking at this bug (bug?id=088efeac32fd) I was looking
at the report at "2018/05/09 18:55" (https://syzkaller.appspot.com/text?tag=CrashReport&x=141b707b800000),
since it was the only report with a reproducer. This was my error.

The error and the call trace in this report are:

It is the same like in bug?id=3887c0d99aecb27d085180c5222d245d08a30806
which, after some more test, made me believe these bugs are duplicate
and are fixed by the same commit.

But let's look at another report at "2018/09/12 21:00"
(https://syzkaller.appspot.com/text?tag=CrashReport&x=14f99b71400000)
at the bug (bug?id=088efeac32fd), the one you've mentioned as

"the last crash for the second version (2) happened just few days ago".

Its error and the call trace are completely different:

>>>
BUG: KMSAN: uninit-value in memcmp+0x11d/0x180 lib/string.c:863
CPU: 0 PID: 6107 Comm: syz-executor4 Not tainted 4.19.0-rc3+ #45

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]

dump_stack+0x14b/0x190 lib/dump_stack.c:113
kmsan_report+0x183/0x2b0 mm/kmsan/kmsan.c:956
__msan_warning+0x70/0xc0 mm/kmsan/kmsan_instr.c:645
memcmp+0x11d/0x180 lib/string.c:863
dev_uc_add_excl+0x165/0x7b0 net/core/dev_addr_lists.c:464
ndo_dflt_fdb_add net/core/rtnetlink.c:3463 [inline]
rtnl_fdb_add+0x1081/0x1270 net/core/rtnetlink.c:3558
rtnetlink_rcv_msg+0xa0b/0x1530 net/core/rtnetlink.c:4715
netlink_rcv_skb+0x36e/0x5f0 net/netlink/af_netlink.c:2454
rtnetlink_rcv+0x50/0x60 net/core/rtnetlink.c:4733
netlink_unicast_kernel net/netlink/af_netlink.c:1317 [inline]
netlink_unicast+0x1638/0x1720 net/netlink/af_netlink.c:1343
netlink_sendmsg+0x1205/0x1290 net/netlink/af_netlink.c:1908
sock_sendmsg_nosec net/socket.c:621 [inline]
sock_sendmsg net/socket.c:631 [inline]
...
Uninit was created at:
...
slab_post_alloc_hook mm/slab.h:446 [inline]
slab_alloc_node mm/slub.c:2718 [inline]
__kmalloc_node_track_caller+0x9e7/0x1160 mm/slub.c:4351
__kmalloc_reserve net/core/skbuff.c:138 [inline]
__alloc_skb+0x2f5/0x9e0 net/core/skbuff.c:206
alloc_skb include/linux/skbuff.h:996 [inline]
netlink_alloc_large_skb net/netlink/af_netlink.c:1189 [inline]
netlink_sendmsg+0xb49/0x1290 net/netlink/af_netlink.c:1883
sock_sendmsg_nosec net/socket.c:621 [inline]
sock_sendmsg net/socket.c:631 [inline]
___sys_sendmsg+0xe70/0x1290 net/socket.c:2114
<<<

This is a different bug. How come these 2 different reports for 2 different
bugs have ended in the same syzkaller report (bug?id=088efeac32fd) ?

One bug is fixed by the "net: fix uninit-value in __hw_addr_add_ex()" commit,
the second one is not, but they are still in the same syzkaller report.

This was the reason of my confusion. I'm not sure how to fix this. If it is possible,
probably we need to cancel/revoke "#syz fix: net: fix uninit-value in __hw_addr_add_ex()"
for this syzkaller report (bug?id=088efeac32fd). And then "split" it into 2 or
more different reports, but I'm not sure if this is possible.

Probably, syzkaller needs to look deeper into the KMSAN reports to differentiate
KMSAN errors happening because of different reasons.

Best regards,
Vladis Dronov | Red Hat, Inc. | Product Security Engineer

[Alexander Potapenko]

On Mon, Sep 24, 2018 at 12:09 AM Vladis Dronov <vdr...@redhat.com> wrote:
>
> Hello, Dmirty,
>
> Thank you for the reply. Can we please, discuss this further?

Hi Vladis,

> > You can see on dashboard that the last crash
> > for the second version (2) happened just few days ago. So this is a
> > different bug.

FWIW I've just double-checked that the reproducer provided by
syzkaller in the original message still triggers the report from the
original message in the latest KMSAN tree (which already contains the
__hw_addr_add_ex() fix from April).

I suspect this is because syzbot used the top stack frame as the
report signature.
There's a mechanism to ignore frames like memcmp() in the reports, not
sure why didn't it work in this case (maybe it just wasn't in place at
the time the report happened).

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/1040580049.15456466.1537740558279.JavaMail.zimbra%40redhat.com.
> For more options, visit https://groups.google.com/d/optout.



--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Dmitry Vyukov]

>> > On Sun, Sep 23, 2018 at 6:02 PM, Vladis Dronov <vdr...@redhat.com> wrote:
>> > > #syz fix: net: fix uninit-value in __hw_addr_add_ex()
>> >
>> > Hi Vladis,
>> >
>> > This can be fixed with "net: fix uninit-value in __hw_addr_add_ex()".
>> > That commit landed in April, syzbot waited till the commit reached all
>> > tested trees, and then closed the bug.
>> > But the similar bug continued to happen, so syzbot created second
>> > version of this bug (2). You can see on dashboard that the last crash
>> > for the second version (2) happened just few days ago. So this is a
>> > different bug.

Precisely discriminating bugs (root causes) bases on crash text is
generally undecidable problem, even for humans. We even can have
literally equal crash texts, which are still different bugs. And we
can have significantly differently looking crash texts, which are
actually caused by the same root cause. syzbot extracts some
"identity" string for each crash and than uses that string to
discriminate crashes and sort them into bins. This identity string is
what you see in email subject and bug title on dashboard. This method
can have both false positives and false negatives, but works
reasonably well in most cases and looks like the best practical
option.

For this exact instance (memcmp) we actually improved the analysis
logic recently:
https://github.com/google/syzkaller/commit/0e29942f77715486995d996f80f82742812d75a2#diff-abe1515f011fad2659ff218f9eea9ae1
But this crash was analyzed and reported before the change. So if this
crash happens again it should be reported as "in __hw_addr_add_ex"
now.

Re __hw_addr_add_ex bug, as Alex noted the crash was detected _after_
the fixing commit went in. So it's something new and different and
can't be fixed by the older commit.

There are no general, single guideline as to what to do when several
different bugs glued together into a single bug. Fixing at least one
of them (any) in the context of the bug is good, fixing both is good
too. When/if a bug is closed, new occurrences of similar crashes (the
same identity string) will lead to creation of a new bug. So if we fix
only one and close the bug, eventually the second one will lead to a
new bug (won't be lost), now dedicated to this second crash.

Now syzbot thinks that this bug is fixed/closed:
https://syzkaller.appspot.com/bug?extid=d3402c47f680ff24b29c
There is specifically no "undo" functionality, because it's inherently
racy with creation of a new version of this bugs by new crashes. So if
of these crashes will happen again, syzbot will open new bugs (now
with better discriminated titles). We can wait for that. Or we can
submit new fixes without waiting for new syzbot bugs (adding
Reported-by to new commits referencing this bug should not do any
harm).

Hope this clarifies things a bit.

Thanks

[Vladis Dronov]

Hello, Dmirty,

Thank you for the explanation of how syzkaller/syzbot works in this and
other emails. I understand that is it a complicated task to determine
and categorize bugs based on just crash dump and messages, and syzkaller
does a great job of doing so.

> Re __hw_addr_add_ex bug, as Alex noted the crash was detected _after_
> the fixing commit went in. So it's something new and different and
> can't be fixed by the older commit.

Indeed, you're right, there is another issue with tun/tap devices which
leads to this bug. I've posted a patch (https://lkml.org/lkml/2018/9/26/416)
to fix it.

I hope I did not do much damage, reporting previous fix as a fix for this bug,
as syzkaller will probably create another "KMSAN: uninit-value in <...>"
report.

Best regards,
Vladis Dronov | Red Hat, Inc. | Product Security Engineer

----- Original Message -----
> From: "Dmitry Vyukov" <dvy...@google.com>
> To: "Alexander Potapenko" <gli...@google.com>
> Cc: "Vladis Dronov" <vdr...@redhat.com>, "syzbot" <syzbot+d3402c...@syzkaller.appspotmail.com>,
> "syzkaller-bugs" <syzkall...@googlegroups.com>, "David Miller" <da...@davemloft.net>, "Eric Dumazet"
> <edum...@google.com>, "LKML" <linux-...@vger.kernel.org>, "Networking" <net...@vger.kernel.org>, "sunlianwen"
> <sunlw...@cn.fujitsu.com>
> Sent: Monday, September 24, 2018 11:39:08 AM
> Subject: Re: KMSAN: uninit-value in memcmp (2)
>

[Dmitry Vyukov]

On Thu, Sep 27, 2018 at 1:17 PM, Vladis Dronov <vdr...@redhat.com> wrote:
> Hello, Dmirty,
>
> Thank you for the explanation of how syzkaller/syzbot works in this and
> other emails. I understand that is it a complicated task to determine
> and categorize bugs based on just crash dump and messages, and syzkaller
> does a great job of doing so.

Thanks.

>> Re __hw_addr_add_ex bug, as Alex noted the crash was detected _after_
>> the fixing commit went in. So it's something new and different and
>> can't be fixed by the older commit.
>
> Indeed, you're right, there is another issue with tun/tap devices which
> leads to this bug. I've posted a patch (https://lkml.org/lkml/2018/9/26/416)
> to fix it.
>
> I hope I did not do much damage, reporting previous fix as a fix for this bug,
> as syzkaller will probably create another "KMSAN: uninit-value in <...>"
> report.

No, it did not do any damage.
This is in fact already re-reported as "KMSAN: uninit-value in __dev_mc_add":
https://syzkaller.appspot.com/bug?id=0766d38c656abeace60621896d705743aeefed51