KASAN: use-after-free Read in netdevice_event_work_handler

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 0bddd227 Documentation: update for gcc 4.9 requirement
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1418afb7100000
kernel config: https://syzkaller.appspot.com/x/.config?x=66ad203c2bb6d8b
dashboard link: https://syzkaller.appspot.com/bug?extid=20b90969babe05609947
compiler: gcc (GCC) 10.1.0-syz 20200507
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12a8edff100000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=167d3bb7100000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+20b909...@syzkaller.appspotmail.com

==================================================================
BUG: KASAN: use-after-free in dev_put include/linux/netdevice.h:3853 [inline]
BUG: KASAN: use-after-free in netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627
Read of size 8 at addr ffff88807b13e568 by task kworker/u4:0/7

CPU: 0 PID: 7 Comm: kworker/u4:0 Not tainted 5.8.0-rc4-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: gid-cache-wq netdevice_event_work_handler
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x18f/0x20d lib/dump_stack.c:118
print_address_description.constprop.0.cold+0xae/0x436 mm/kasan/report.c:383
__kasan_report mm/kasan/report.c:513 [inline]
kasan_report.cold+0x1f/0x37 mm/kasan/report.c:530
dev_put include/linux/netdevice.h:3853 [inline]
netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627
process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
kthread+0x3b5/0x4a0 kernel/kthread.c:291
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:293

Allocated by task 13061:
save_stack+0x1b/0x40 mm/kasan/common.c:48
set_track mm/kasan/common.c:56 [inline]
__kasan_kmalloc.constprop.0+0xc2/0xd0 mm/kasan/common.c:494
kmalloc_node include/linux/slab.h:578 [inline]
kvmalloc_node+0x61/0xf0 mm/util.c:574
kvmalloc include/linux/mm.h:753 [inline]
kvzalloc include/linux/mm.h:761 [inline]
alloc_netdev_mqs+0x97/0xdc0 net/core/dev.c:9938
__ip_tunnel_create+0x201/0x580 net/ipv4/ip_tunnel.c:254
ip_tunnel_init_net+0x32b/0x980 net/ipv4/ip_tunnel.c:1072
ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

Freed by task 13061:
save_stack+0x1b/0x40 mm/kasan/common.c:48
set_track mm/kasan/common.c:56 [inline]
kasan_set_free_info mm/kasan/common.c:316 [inline]
__kasan_slab_free+0xf5/0x140 mm/kasan/common.c:455
__cache_free mm/slab.c:3426 [inline]
kfree+0x103/0x2c0 mm/slab.c:3757
kvfree+0x42/0x50 mm/util.c:603
device_release+0x71/0x200 drivers/base/core.c:1559
kobject_cleanup lib/kobject.c:693 [inline]
kobject_release lib/kobject.c:722 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x1c0/0x270 lib/kobject.c:739
put_device+0x1b/0x30 drivers/base/core.c:2779
free_netdev+0x35d/0x480 net/core/dev.c:10054
__ip_tunnel_create+0x48f/0x580 net/ipv4/ip_tunnel.c:274
ip_tunnel_init_net+0x32b/0x980 net/ipv4/ip_tunnel.c:1072
ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

The buggy address belongs to the object at ffff88807b13e000
which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 1384 bytes inside of
4096-byte region [ffff88807b13e000, ffff88807b13f000)
The buggy address belongs to the page:
page:ffffea0001ec4f80 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 head:ffffea0001ec4f80 order:1 compound_mapcount:0
flags: 0xfffe0000010200(slab|head)
raw: 00fffe0000010200 ffffea0001ecce88 ffffea0001987988 ffff8880aa002000
raw: 0000000000000000 ffff88807b13e000 0000000100000001 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
ffff88807b13e400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88807b13e480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88807b13e500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff88807b13e580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88807b13e600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer still triggered crash:
general protection fault in free_netdev

general protection fault, probably for non-canonical address 0xdffffc0000000080: 0000 [#1] PREEMPT SMP KASAN
KASAN: null-ptr-deref in range [0x0000000000000400-0x0000000000000407]
CPU: 1 PID: 9204 Comm: syz-executor.2 Not tainted 5.8.0-rc5-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

RIP: 0010:netif_free_tx_queues net/core/dev.c:9357 [inline]
RIP: 0010:free_netdev+0x41/0x480 net/core/dev.c:10028
Code: d2 be 2b 27 00 00 48 c7 c7 e0 67 fd 88 e8 87 b8 1c fb 48 8d bd 00 04 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 f4 03 00 00 48 8b bd 00 04 00 00 e8 39 31 6a fb
RSP: 0018:ffffc9000788fc10 EFLAGS: 00010202
RAX: dffffc0000000000 RBX: ffff88809e734000 RCX: ffffffff86f9069b
RDX: 0000000000000080 RSI: ffffffff89bc1100 RDI: 0000000000000400
RBP: 0000000000000000 R08: 0000000000000001 R09: ffff88808e1d40a3
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fffffff4
R13: ffff88809b3c8410 R14: ffff888095989000 R15: ffff88809e639578
FS: 00007fcfd9339700(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c816f86000 CR3: 0000000090efb000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
ip6gre_init_net+0x546/0x610 net/ipv6/ip6_gre.c:1595

ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

RIP: 0033:0x45cba9
Code: Bad RIP value.
RSP: 002b:00007fcfd9338c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000110
RAX: ffffffffffffffda RBX: 000000000050c4a0 RCX: 000000000045cba9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000040000000
RBP: 000000000078bfa0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000000000000ca9 R14: 00000000004cf0a8 R15: 00007fcfd93396d4
Modules linked in:
---[ end trace d085e50a8fa116e7 ]---
RIP: 0010:netif_free_tx_queues net/core/dev.c:9357 [inline]
RIP: 0010:free_netdev+0x41/0x480 net/core/dev.c:10028
Code: d2 be 2b 27 00 00 48 c7 c7 e0 67 fd 88 e8 87 b8 1c fb 48 8d bd 00 04 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80> 3c 02 00 0f 85 f4 03 00 00 48 8b bd 00 04 00 00 e8 39 31 6a fb
RSP: 0018:ffffc9000788fc10 EFLAGS: 00010202
RAX: dffffc0000000000 RBX: ffff88809e734000 RCX: ffffffff86f9069b
RDX: 0000000000000080 RSI: ffffffff89bc1100 RDI: 0000000000000400
RBP: 0000000000000000 R08: 0000000000000001 R09: ffff88808e1d40a3
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fffffff4
R13: ffff88809b3c8410 R14: ffff888095989000 R15: ffff88809e639578
FS: 00007fcfd9339700(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000200 CR3: 0000000090efb000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


Tested on:

commit: 0dc589da Merge tag 'iommu-fixes-v5.8-rc5' of git://git.ker..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=13ed3757100000
kernel config: https://syzkaller.appspot.com/x/.config?x=eccbd32667dbfc2

[syzbot]

syzbot has bisected this issue to:

commit d70c47c8dc6902db19555b7ff7e6eeb264d4ac06
Author: Heiner Kallweit <hkall...@gmail.com>
Date: Thu Apr 23 19:34:33 2020 +0000

net: phy: make phy_suspend a no-op if PHY is suspended already

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=16b2aad8900000
start commit: 0bddd227 Documentation: update for gcc 4.9 requirement
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=15b2aad8900000
console output: https://syzkaller.appspot.com/x/log.txt?x=11b2aad8900000

kernel config: https://syzkaller.appspot.com/x/.config?x=66ad203c2bb6d8b
dashboard link: https://syzkaller.appspot.com/bug?extid=20b90969babe05609947

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12a8edff100000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=167d3bb7100000

Reported-by: syzbot+20b909...@syzkaller.appspotmail.com
Fixes: d70c47c8dc69 ("net: phy: make phy_suspend a no-op if PHY is suspended already")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
KASAN: use-after-free Read in netdevice_event_work_handler

==================================================================
BUG: KASAN: use-after-free in dev_put include/linux/netdevice.h:3853 [inline]
BUG: KASAN: use-after-free in netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627

Read of size 8 at addr ffff88806b0bc568 by task kworker/u4:0/7

CPU: 0 PID: 7 Comm: kworker/u4:0 Not tainted 5.8.0-rc7-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011

Workqueue: gid-cache-wq netdevice_event_work_handler
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x18f/0x20d lib/dump_stack.c:118
print_address_description.constprop.0.cold+0xae/0x436 mm/kasan/report.c:383
__kasan_report mm/kasan/report.c:513 [inline]
kasan_report.cold+0x1f/0x37 mm/kasan/report.c:530
dev_put include/linux/netdevice.h:3853 [inline]
netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627
process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
kthread+0x3b5/0x4a0 kernel/kthread.c:291
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:293

Allocated by task 10379:

save_stack+0x1b/0x40 mm/kasan/common.c:48
set_track mm/kasan/common.c:56 [inline]
__kasan_kmalloc.constprop.0+0xc2/0xd0 mm/kasan/common.c:494
kmalloc_node include/linux/slab.h:578 [inline]
kvmalloc_node+0x61/0xf0 mm/util.c:574
kvmalloc include/linux/mm.h:753 [inline]
kvzalloc include/linux/mm.h:761 [inline]
alloc_netdev_mqs+0x97/0xdc0 net/core/dev.c:9938
__ip_tunnel_create+0x201/0x580 net/ipv4/ip_tunnel.c:254
ip_tunnel_init_net+0x32b/0x980 net/ipv4/ip_tunnel.c:1072

ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

Freed by task 10379:

save_stack+0x1b/0x40 mm/kasan/common.c:48
set_track mm/kasan/common.c:56 [inline]
kasan_set_free_info mm/kasan/common.c:316 [inline]
__kasan_slab_free+0xf5/0x140 mm/kasan/common.c:455
__cache_free mm/slab.c:3426 [inline]
kfree+0x103/0x2c0 mm/slab.c:3757
kvfree+0x42/0x50 mm/util.c:603

device_release+0x71/0x200 drivers/base/core.c:1579

kobject_cleanup lib/kobject.c:693 [inline]
kobject_release lib/kobject.c:722 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x1c0/0x270 lib/kobject.c:739

put_device+0x1b/0x30 drivers/base/core.c:2799

free_netdev+0x35d/0x480 net/core/dev.c:10054
__ip_tunnel_create+0x48f/0x580 net/ipv4/ip_tunnel.c:274
ip_tunnel_init_net+0x32b/0x980 net/ipv4/ip_tunnel.c:1072

ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

The buggy address belongs to the object at ffff88806b0bc000

which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 1384 bytes inside of

4096-byte region [ffff88806b0bc000, ffff88806b0bd000)

The buggy address belongs to the page:

page:ffffea0001ac2f00 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 head:ffffea0001ac2f00 order:1 compound_mapcount:0
flags: 0xfffe0000010200(slab|head)
raw: 00fffe0000010200 ffffea0001ac2e88 ffffea0001ac3208 ffff8880aa002000
raw: 0000000000000000 ffff88806b0bc000 0000000100000001 0000000000000000

page dumped because: kasan: bad access detected

Memory state around the buggy address:

ffff88806b0bc400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88806b0bc480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88806b0bc500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff88806b0bc580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88806b0bc600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


Tested on:

commit: 92ed3019 Linux 5.8-rc7
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1099e9c4900000
kernel config: https://syzkaller.appspot.com/x/.config?x=1ccb555827990a8e

[syzbot]

Hello,

syzbot tried to test the proposed patch but the build/boot failed:

failed to checkout kernel repo g...@github.com:coiby/linux.git/syzbot1: failed to run ["git" "fetch" "g...@github.com:coiby/linux.git" "syzbot1"]: exit status 128
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.



Tested on:

commit: [unknown
git tree: g...@github.com:coiby/linux.git syzbot1

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
KASAN: use-after-free Read in netdevice_event_work_handler

==================================================================
BUG: KASAN: use-after-free in dev_put include/linux/netdevice.h:3853 [inline]
BUG: KASAN: use-after-free in netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627

Read of size 8 at addr ffff888069024568 by task kworker/u4:2/39

CPU: 1 PID: 39 Comm: kworker/u4:2 Not tainted 5.8.0-rc7-syzkaller #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: gid-cache-wq netdevice_event_work_handler
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x18f/0x20d lib/dump_stack.c:118
print_address_description.constprop.0.cold+0xae/0x436 mm/kasan/report.c:383
__kasan_report mm/kasan/report.c:513 [inline]
kasan_report.cold+0x1f/0x37 mm/kasan/report.c:530
dev_put include/linux/netdevice.h:3853 [inline]
netdevice_event_work_handler+0x15b/0x1b0 drivers/infiniband/core/roce_gid_mgmt.c:627
process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
kthread+0x3b5/0x4a0 kernel/kthread.c:291
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:293

Allocated by task 12302:

save_stack+0x1b/0x40 mm/kasan/common.c:48
set_track mm/kasan/common.c:56 [inline]
__kasan_kmalloc.constprop.0+0xc2/0xd0 mm/kasan/common.c:494
kmalloc_node include/linux/slab.h:578 [inline]
kvmalloc_node+0x61/0xf0 mm/util.c:574
kvmalloc include/linux/mm.h:753 [inline]
kvzalloc include/linux/mm.h:761 [inline]
alloc_netdev_mqs+0x97/0xdc0 net/core/dev.c:9938
__ip_tunnel_create+0x201/0x580 net/ipv4/ip_tunnel.c:254
ip_tunnel_init_net+0x32b/0x980 net/ipv4/ip_tunnel.c:1072
ops_init+0xaf/0x470 net/core/net_namespace.c:151
setup_net+0x2d8/0x850 net/core/net_namespace.c:341
copy_net_ns+0x2cf/0x5e0 net/core/net_namespace.c:482
create_new_namespaces+0x3f6/0xb10 kernel/nsproxy.c:110
unshare_nsproxy_namespaces+0xbd/0x1f0 kernel/nsproxy.c:231
ksys_unshare+0x36c/0x9a0 kernel/fork.c:2983
__do_sys_unshare kernel/fork.c:3051 [inline]
__se_sys_unshare kernel/fork.c:3049 [inline]
__x64_sys_unshare+0x2d/0x40 kernel/fork.c:3049
do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:384
entry_SYSCALL_64_after_hwframe+0x44/0xa9

Freed by task 12302:

The buggy address belongs to the object at ffff888069024000

which belongs to the cache kmalloc-4k of size 4096
The buggy address is located 1384 bytes inside of

4096-byte region [ffff888069024000, ffff888069025000)

The buggy address belongs to the page:

page:ffffea0001a40900 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 head:ffffea0001a40900 order:1 compound_mapcount:0
flags: 0xfffe0000010200(slab|head)
raw: 00fffe0000010200 ffffea0001a5ee88 ffffea0001a40988 ffff8880aa002000
raw: 0000000000000000 ffff888069024000 0000000100000001 0000000000000000

page dumped because: kasan: bad access detected

Memory state around the buggy address:

ffff888069024400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888069024480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888069024500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff888069024580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888069024600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


Tested on:

commit: 859f8173 return NOTIFY_DONE for mpls_dev_notify
git tree: https://github.com/coiby/linux.git syzbot1
console output: https://syzkaller.appspot.com/x/log.txt?x=16625e70900000
kernel config: https://syzkaller.appspot.com/x/.config?x=1ccb555827990a8e

[Rustam Kovhaev]

IB roce driver receives NETDEV_UNREGISTER event, calls dev_hold() and
schedules work item to execute, and before wq gets a chance to complete
it, we return to ip_tunnel.c:274 and call free_netdev(), and then later
we get UAF when scheduled function references already freed net_device

i added verbose logging to ip_tunnel.c to see pcpu_refcnt:
+ pr_info("about to free_netdev(dev) dev->pcpu_refcnt %d", netdev_refcnt_read(dev));

and got the following:
[ 410.220127][ T2944] ip_tunnel: about to free_netdev(dev) dev->pcpu_refcnt 8

i tried to make IB roce driver flush wq and work item, but i ran into
lockdep issues
also tried to modify dev core and call netdev_wait_allrefs() but ran
into rntl deadlocks

any hints or help in fixing this would be appreciated, thank you!

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+20b909...@syzkaller.appspotmail.com

Tested on:

commit: 7dc6fd0f Merge branch 'i2c/for-current' of git://git.kerne..
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=1ccb555827990a8e

dashboard link: https://syzkaller.appspot.com/bug?extid=20b90969babe05609947
compiler: gcc (GCC) 10.1.0-syz 20200507

patch: https://syzkaller.appspot.com/x/patch.diff?x=1799f8a2900000

Note: testing is done by a robot and is best-effort only.

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+20b909...@syzkaller.appspotmail.com

Tested on:

commit: 260ddb9e return NOTIFY_DONE for addrconf_notify

git tree: https://github.com/coiby/linux.git syzbot1

kernel config: https://syzkaller.appspot.com/x/.config?x=1ccb555827990a8e
dashboard link: https://syzkaller.appspot.com/bug?extid=20b90969babe05609947
compiler: gcc (GCC) 10.1.0-syz 20200507

[Coiby Xu]

On Fri, Jul 31, 2020 at 09:11:22PM +0000, Rustam Kovhaev wrote:
>On Thu, Jul 09, 2020 at 04:54:19PM -0700, syzbot wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit: 0bddd227 Documentation: update for gcc 4.9 requirement
>> git tree: upstream
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1418afb7100000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=66ad203c2bb6d8b

>> dashboard link: https://syzkaller.appspot.com/bug?extid=20b90969babe05609947
>> compiler: gcc (GCC) 10.1.0-syz 20200507

Congratulations on your latest patch [1] which has fixed this issue! I've
also been trying to fix this bug. Your solution is the right one and simpler
than mine:) I've also realized struct device shouldn't be freed at all
since dev_hold is called but haven't figured out why,
// drivers/infiniband/core/roce_gid_mgmt.c
static int netdevice_queue_work(struct netdev_event_work_cmd *cmds,
struct net_device *ndev)
{
...
for (i = 0; i < ARRAY_SIZE(ndev_work->cmds) && ndev_work->cmds[i].cb; i++) {
dev_hold(ndev_work->cmds[i].ndev);
...
}
...
return NOTIFY_DONE;
}

But maybe we could make an improvement which is to fix what makes the
nofitied callback functions mpls_dev_notify and addrconf_notify fail in
the first place. Currently I simply let mpls_dev_notify and addrconf_notify
return NOTIFY_DONE [2] [3] so call_netdevice_notifiers and register_netdevice
would succeed.

My observation is that mpls_dev_notify and addrconf_notify are not per-netns
netdevice callback functions of the notified (the subscriber) and they could
fail.

When per-netns netdevice callback functions are successful and global ones
like mpls_dev_notify fail, register_netdevice would then fails.

// net/core/dev.c
static int call_netdevice_notifiers_info(unsigned long val,
struct netdev_notifier_info *info)
{
struct net *net = dev_net(info->dev);
int ret;

ASSERT_RTNL();

/* Run per-netns notifier block chain first, then run the global one.
* Hopefully, one day, the global one is going to be removed after
* all notifier block registrators get converted to be per-netns.
*/
ret = raw_notifier_call_chain(&net->netdev_chain, val, info);
if (ret & NOTIFY_STOP_MASK)
return ret;
return raw_notifier_call_chain(&netdev_chain, val, info);
}


This causes the created net_device struct to be destroyed. Since per-netns
netdevice callback functions have been successful, the netdevice is still
used by netdevice_event_work_handler which cause this "KASAN: use-after-free"
bug.


[1] https://syzkaller.appspot.com/text?tag=Patch&x=1799f8a2900000
[2] https://github.com/coiby/linux/commit/859f817317c8cd4a17af8879094db8697a2c7754
[3] https://github.com/coiby/linux/commit/260ddb9ec9b124ca7ad93f368eadb90b16edf2ef

[Rustam Kovhaev]

does the patch below look sane to you? or is it complete nonsense?
syzbot test came out OK, but it might not mean anything

diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
index 2860def84f4d..b31c8969c8b2 100644
--- a/drivers/infiniband/core/roce_gid_mgmt.c
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -626,6 +626,7 @@ static void netdevice_event_work_handler(struct work_struct *_work)
work->cmds[i].ndev);
dev_put(work->cmds[i].ndev);
dev_put(work->cmds[i].filter_ndev);
+ put_device(&work->cmds[i].ndev->dev);
}

kfree(work);
@@ -649,6 +650,7 @@ static int netdevice_queue_work(struct netdev_event_work_cmd *cmds,
ndev_work->cmds[i].filter_ndev = ndev;
dev_hold(ndev_work->cmds[i].ndev);
dev_hold(ndev_work->cmds[i].filter_ndev);
+ get_device(&ndev_work->cmds[i].ndev->dev);
}
INIT_WORK(&ndev_work->work, netdevice_event_work_handler);

[Hillf Danton]

Thanks for making it clear what is crippling dev_hold().

>
> i tried to make IB roce driver flush wq and work item, but i ran into
> lockdep issues
> also tried to modify dev core and call netdev_wait_allrefs() but ran
> into rntl deadlocks
>
> any hints or help in fixing this would be appreciated, thank you!

[1] It's quite likely at the first a couple of glances to misread
pcpu_refcnt with percpu-refcount.

[2] netdev's refcnt IMO isn't so hot that pcpu is needed, and it's
replaced with atomic one in the diff below.

[3] change helpers relevant with care for breaking no users of the
current free_netdev() by initializing the refcnt with 1, leaving it
still able to dismantle devices with non-zero refcnt, of 1 in normal
contexts.

[4] use dev_put() instead of free_netdev() in the error path on
creating ip_tunnel.

If you like let us know if it makes your box happy.

Hillf
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1732,7 +1732,7 @@ enum netdev_priv_flags {
* the watchdog (see dev_watchdog())
* @watchdog_timer: List of timers
*
- * @pcpu_refcnt: Number of references to this device
+ * @refcnt: Number of references to this device
* @todo_list: Delayed register/unregister
* @link_watch_list: XXX: need comments on this one
*
@@ -2004,7 +2004,7 @@ struct net_device {
/* These may be needed for future network-power-down code. */
struct timer_list watchdog_timer;

- int __percpu *pcpu_refcnt;
+ atomic_t refcnt;
struct list_head todo_list;

struct list_head link_watch_list;
@@ -3788,7 +3788,8 @@ void netdev_run_todo(void);
*/
static inline void dev_put(struct net_device *dev)
{
- this_cpu_dec(*dev->pcpu_refcnt);
+ if (atomic_dec_and_test(&dev->refcnt))
+ free_netdev(dev);
}

/**
@@ -3799,7 +3800,7 @@ static inline void dev_put(struct net_de
*/
static inline void dev_hold(struct net_device *dev)
{
- this_cpu_inc(*dev->pcpu_refcnt);
+ atomic_inc(&dev->refcnt);
}

/* Carrier loss detection, dial on demand. The functions netif_carrier_on
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -9690,10 +9690,7 @@ int init_dummy_netdev(struct net_device
/* napi_busy_loop stats accounting wants this */
dev_net_set(dev, &init_net);

- /* Note : We dont allocate pcpu_refcnt for dummy devices,
- * because users of this 'device' dont need to change
- * its refcount.
- */
+ atomic_set(&dev->refcnt, 1);

return 0;
}
@@ -9727,11 +9724,7 @@ EXPORT_SYMBOL(register_netdev);

int netdev_refcnt_read(const struct net_device *dev)
{
- int i, refcnt = 0;
-
- for_each_possible_cpu(i)
- refcnt += *per_cpu_ptr(dev->pcpu_refcnt, i);
- return refcnt;
+ return atomic_read(&dev->refcnt);
}
EXPORT_SYMBOL(netdev_refcnt_read);

@@ -10019,12 +10012,10 @@ struct net_device *alloc_netdev_mqs(int
dev = PTR_ALIGN(p, NETDEV_ALIGN);
dev->padded = (char *)dev - (char *)p;

- dev->pcpu_refcnt = alloc_percpu(int);
- if (!dev->pcpu_refcnt)
- goto free_dev;
+ atomic_set(&dev->refcnt, 1);

if (dev_addr_init(dev))
- goto free_pcpu;
+ goto free_dev;

dev_mc_init(dev);
dev_uc_init(dev);
@@ -10080,8 +10071,6 @@ free_all:
free_netdev(dev);
return NULL;

-free_pcpu:
- free_percpu(dev->pcpu_refcnt);
free_dev:
netdev_freemem(dev);
return NULL;
@@ -10113,8 +10102,6 @@ void free_netdev(struct net_device *dev)
list_for_each_entry_safe(p, n, &dev->napi_list, dev_list)
netif_napi_del(p);

- free_percpu(dev->pcpu_refcnt);
- dev->pcpu_refcnt = NULL;
free_percpu(dev->xdp_bulkq);
dev->xdp_bulkq = NULL;

--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -273,7 +273,7 @@ static struct net_device *__ip_tunnel_cr
return dev;

failed_free:
- free_netdev(dev);
+ dev_put(dev);
failed:
return ERR_PTR(err);
}

[Jason Gunthorpe]

On Fri, Jul 31, 2020 at 02:11:22PM -0700, Rustam Kovhaev wrote:

> IB roce driver receives NETDEV_UNREGISTER event, calls dev_hold() and
> schedules work item to execute, and before wq gets a chance to complete
> it, we return to ip_tunnel.c:274 and call free_netdev(), and then later
> we get UAF when scheduled function references already freed net_device
>
> i added verbose logging to ip_tunnel.c to see pcpu_refcnt:
> + pr_info("about to free_netdev(dev) dev->pcpu_refcnt %d", netdev_refcnt_read(dev));
>
> and got the following:
> [ 410.220127][ T2944] ip_tunnel: about to free_netdev(dev) dev->pcpu_refcnt 8

I think there is a missing call to netdev_wait_allrefs() in the
rollback_registered_many().

The normal success flow has this wait after delivering
NETDEV_UNREGISTER, the error unwind for register_netdevice should as
well.

If the netdevice can progress to free while a dev_hold is active I
think it means dev_hold is functionally useless.

Jason

[Rustam Kovhaev]

On Sun, Aug 02, 2020 at 07:22:26PM -0300, Jason Gunthorpe wrote:
> On Fri, Jul 31, 2020 at 02:11:22PM -0700, Rustam Kovhaev wrote:
>
> > IB roce driver receives NETDEV_UNREGISTER event, calls dev_hold() and
> > schedules work item to execute, and before wq gets a chance to complete
> > it, we return to ip_tunnel.c:274 and call free_netdev(), and then later
> > we get UAF when scheduled function references already freed net_device
> >
> > i added verbose logging to ip_tunnel.c to see pcpu_refcnt:
> > + pr_info("about to free_netdev(dev) dev->pcpu_refcnt %d", netdev_refcnt_read(dev));
> >
> > and got the following:
> > [ 410.220127][ T2944] ip_tunnel: about to free_netdev(dev) dev->pcpu_refcnt 8
>
> I think there is a missing call to netdev_wait_allrefs() in the
> rollback_registered_many().

calling it there leads to rtnl deadlock, i think we should call
net_set_todo(), so that later when we call rtnl_unlock() it will
execute netdev_run_todo() and there it will proceed to calling
netdev_wait_allrefs(), but in ip tunnel i will need get
free_netdev() to be called after we unlock rtnl mutex
i'll try to send a new patch for review

> The normal success flow has this wait after delivering
> NETDEV_UNREGISTER, the error unwind for register_netdevice should as
> well.
>
> If the netdevice can progress to free while a dev_hold is active I
> think it means dev_hold is functionally useless.

good point

[Jason Gunthorpe]

On Tue, Aug 04, 2020 at 01:00:13PM -0700, Rustam Kovhaev wrote:
> On Sun, Aug 02, 2020 at 07:22:26PM -0300, Jason Gunthorpe wrote:
> > On Fri, Jul 31, 2020 at 02:11:22PM -0700, Rustam Kovhaev wrote:
> >
> > > IB roce driver receives NETDEV_UNREGISTER event, calls dev_hold() and
> > > schedules work item to execute, and before wq gets a chance to complete
> > > it, we return to ip_tunnel.c:274 and call free_netdev(), and then later
> > > we get UAF when scheduled function references already freed net_device
> > >
> > > i added verbose logging to ip_tunnel.c to see pcpu_refcnt:
> > > + pr_info("about to free_netdev(dev) dev->pcpu_refcnt %d", netdev_refcnt_read(dev));
> > >
> > > and got the following:
> > > [ 410.220127][ T2944] ip_tunnel: about to free_netdev(dev) dev->pcpu_refcnt 8
> >
> > I think there is a missing call to netdev_wait_allrefs() in the
> > rollback_registered_many().
> calling it there leads to rtnl deadlock, i think we should call
> net_set_todo(), so that later when we call rtnl_unlock() it will
> execute netdev_run_todo() and there it will proceed to calling
> netdev_wait_allrefs(), but in ip tunnel i will need get
> free_netdev() to be called after we unlock rtnl mutex
> i'll try to send a new patch for review

Oh the whole register is called under rtnl? Yikes..

This is probably a systemic problem with register_netdevice error
unwind, not just ip tunnel

The other way to handle it would be to organize things so that
register cannot fail once it starts calling notifiers?

Jason

[syzbot]

Auto-closing this bug as obsolete.
No recent activity, existing reproducers are no longer triggering the issue.