net: heap out-of-bounds in fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone

[Dmitry Vyukov]

Hello,

I am getting heap out-of-bounds reports in
fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone while running
syzkaller fuzzer on 86292b33d4b79ee03e2f43ea0381ef85f077c760. They all
follow the same pattern: an object of size 216 is allocated from
ip_dst_cache slab, and then accessed at offset 272/276 withing
fib6_walk. Looks like type confusion. Unfortunately this is not
reproducible.

==================================================================
BUG: KASAN: slab-out-of-bounds in rt6_dump_route+0x293/0x2f0
net/ipv6/route.c:3547 at addr ffff88004b864514
Read of size 4 by task syz-executor7/25042
CPU: 0 PID: 25042 Comm: syz-executor7 Not tainted 4.10.0+ #234
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330
rt6_dump_route+0x293/0x2f0 net/ipv6/route.c:3547
fib6_dump_node+0x101/0x1a0 net/ipv6/ip6_fib.c:315
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576
fib6_walk+0x1cf/0x300 net/ipv6/ip6_fib.c:1621
fib6_dump_table net/ipv6/ip6_fib.c:374 [inline]
inet6_dump_fib+0x832/0xea0 net/ipv6/ip6_fib.c:447
rtnl_dump_all+0x8a/0x2a0 net/core/rtnetlink.c:2776
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
__netlink_dump_start+0x4e5/0x760 net/netlink/af_netlink.c:2217
netlink_dump_start include/linux/netlink.h:165 [inline]
rtnetlink_rcv_msg+0x4a3/0x860 net/core/rtnetlink.c:4094
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
rtnetlink_rcv+0x2a/0x40 net/core/rtnetlink.c:4110
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x514/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xa9f/0xe50 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
new_sync_write fs/read_write.c:499 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:512
vfs_write+0x187/0x530 fs/read_write.c:560
SYSC_write fs/read_write.c:607 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:599
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007fe10102bb58 EFLAGS: 00000292 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00000000004458d9
RDX: 000000000000001f RSI: 0000000020691000 RDI: 0000000000000006
RBP: 00000000006e2fc0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000292 R12: 0000000000708000
R13: 00000000209e1ff7 R14: 0000000000000001 R15: fffffffffffffffd
Object at ffff88004b864400, in cache ip_dst_cache size: 216
Allocated:
PID = 21976
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
__mkroute_output net/ipv4/route.c:2163 [inline]
__ip_route_output_key_hash+0xce3/0x2c70 net/ipv4/route.c:2373
__ip_route_output_key include/net/route.h:122 [inline]
ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2459
ip_route_output_key include/net/route.h:132 [inline]
sctp_v4_get_dst+0x5d2/0x1570 net/sctp/protocol.c:454
sctp_transport_route+0xa8/0x420 net/sctp/transport.c:292
sctp_assoc_add_peer+0x5a5/0x1470 net/sctp/associola.c:653
sctp_sendmsg+0x1800/0x3970 net/sctp/socket.c:1870
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
___sys_sendmsg+0x4a3/0x9f0 net/socket.c:1985
__sys_sendmmsg+0x25c/0x750 net/socket.c:2075
SYSC_sendmmsg net/socket.c:2106 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2101
entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:
PID = 15058
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_free include/net/dst.h:428 [inline]
rt_fibinfo_free_cpus net/ipv4/fib_semantics.c:198 [inline]
free_fib_info_rcu+0x399/0x590 net/ipv4/fib_semantics.c:213
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:
ffff88004b864400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88004b864480: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
>ffff88004b864500: fc fc fc fc fc fc fc fc 00 00 00 00 00 00 00 00
^
ffff88004b864580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88004b864600: 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

==================================================================
BUG: KASAN: slab-out-of-bounds in fib6_age+0x3fd/0x480
net/ipv6/ip6_fib.c:1769 at addr ffff880088d1bb54
Read of size 4 by task swapper/1/0
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.10.0+ #260
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330
fib6_age+0x3fd/0x480 net/ipv6/ip6_fib.c:1769
fib6_clean_node+0x356/0x550 net/ipv6/ip6_fib.c:1647
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576
fib6_walk+0x1cf/0x300 net/ipv6/ip6_fib.c:1621
fib6_clean_tree+0x266/0x3a0 net/ipv6/ip6_fib.c:1693
__fib6_clean_all+0x1e1/0x360 net/ipv6/ip6_fib.c:1709
fib6_clean_all net/ipv6/ip6_fib.c:1720 [inline]
fib6_run_gc+0x185/0x3d0 net/ipv6/ip6_fib.c:1817
fib6_gc_timer_cb+0x1c/0x20 net/ipv6/ip6_fib.c:1832
call_timer_fn+0x241/0x820 kernel/time/timer.c:1266
expire_timers kernel/time/timer.c:1305 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1599
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1612
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405
exiting_irq arch/x86/include/asm/apic.h:658 [inline]
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:487
RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:53
RSP: 0018:ffff88004dd8fc10 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: 1ffff10009bb1f85 RCX: 0000000000000000
RDX: 1ffffffff0a18ebc RSI: 0000000000000001 RDI: ffffffff850c75e0
RBP: ffff88004dd8fc10 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff10009bb1fa9
R13: ffff88004dd8fcc8 R14: ffffffff85697338 R15: ffff88004dd8fe68
</IRQ>
arch_safe_halt arch/x86/include/asm/paravirt.h:98 [inline]
default_idle+0xbf/0x440 arch/x86/kernel/process.c:271
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:262
default_idle_call+0x36/0x90 kernel/sched/idle.c:96
cpuidle_idle_call kernel/sched/idle.c:154 [inline]
do_idle+0x373/0x520 kernel/sched/idle.c:243
cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:345
start_secondary+0x36c/0x460 arch/x86/kernel/smpboot.c:272
start_cpu+0x14/0x14 arch/x86/kernel/head_64.S:306
Object at ffff880088d1ba40, in cache ip_dst_cache size: 216
Allocated:
PID = 30165
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
__mkroute_output net/ipv4/route.c:2165 [inline]
__ip_route_output_key_hash+0xce3/0x2c70 net/ipv4/route.c:2375
__ip_route_output_key include/net/route.h:122 [inline]
ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2461
ip_route_output_key include/net/route.h:132 [inline]
sctp_v4_get_dst+0x5d2/0x1570 net/sctp/protocol.c:458
sctp_transport_route+0xa8/0x420 net/sctp/transport.c:292
sctp_assoc_add_peer+0x5a5/0x1470 net/sctp/associola.c:653
sctp_sendmsg+0x1800/0x3970 net/sctp/socket.c:1870
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
SYSC_sendto+0x660/0x810 net/socket.c:1685
SyS_sendto+0x40/0x50 net/socket.c:1653
entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:
PID = 28880
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_destroy_rcu+0x15/0x40 net/core/dst.c:294
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:
ffff880088d1ba00: fc fc fc fc fc fc fc fc 00 00 00 00 00 00 00 00
ffff880088d1ba80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff880088d1bb00: 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff880088d1bb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff880088d1bc00: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
==================================================================

==================================================================
BUG: KASAN: slab-out-of-bounds in rt6_fill_node.isra.61+0x1434/0x1780
net/ipv6/route.c:3396 at addr ffff88004b5c0790
Read of size 4 by task syz-executor3/3502
CPU: 0 PID: 3502 Comm: syz-executor3 Not tainted 4.10.0+ #260
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330
rt6_fill_node.isra.61+0x1434/0x1780 net/ipv6/route.c:3396
rt6_dump_route+0x245/0x2f0 net/ipv6/route.c:3557
fib6_dump_node+0x101/0x1a0 net/ipv6/ip6_fib.c:315
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576
fib6_walk+0x1cf/0x300 net/ipv6/ip6_fib.c:1621
fib6_dump_table net/ipv6/ip6_fib.c:374 [inline]
inet6_dump_fib+0x832/0xea0 net/ipv6/ip6_fib.c:447
rtnl_dump_all+0x8a/0x2a0 net/core/rtnetlink.c:2776
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
netlink_recvmsg+0xb6a/0x1500 net/netlink/af_netlink.c:1886
sock_recvmsg_nosec net/socket.c:740 [inline]
sock_recvmsg+0xd7/0x110 net/socket.c:747
___sys_recvmsg+0x2b8/0x6b0 net/socket.c:2144
__sys_recvmsg+0x135/0x300 net/socket.c:2189
SYSC_recvmsg net/socket.c:2201 [inline]
SyS_recvmsg+0x2d/0x50 net/socket.c:2196
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:280
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x4458d9
RSP: 002b:00007f694bf1fb58 EFLAGS: 00000286 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 0000000000708000 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 00000000206a2fc8 RDI: 0000000000000019
RBP: 00000000000036d0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006e1790
R13: 0000000000000019 R14: 00000000206a2fc8 R15: 0000000000000000
Object at ffff88004b5c0680, in cache ip_dst_cache size: 216
Allocated:
PID = 1362
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
ip_route_input_slow+0xe67/0x21e0 net/ipv4/route.c:1936
ip_route_input_noref+0x13c/0x10b0 net/ipv4/route.c:2058
ip_rcv_finish+0x301/0x1b40 net/ipv4/ip_input.c:344
NF_HOOK include/linux/netfilter.h:257 [inline]
ip_rcv+0xd75/0x19a0 net/ipv4/ip_input.c:487
__netif_receive_skb_core+0x1ac8/0x33f0 net/core/dev.c:4179
__netif_receive_skb+0x2a/0x170 net/core/dev.c:4217
netif_receive_skb_internal+0xf0/0x400 net/core/dev.c:4245
napi_skb_finish net/core/dev.c:4602 [inline]
napi_gro_receive+0x4d4/0x670 net/core/dev.c:4636
e1000_receive_skb drivers/net/ethernet/intel/e1000/e1000_main.c:4033 [inline]
e1000_clean_rx_irq+0x5e0/0x1490
drivers/net/ethernet/intel/e1000/e1000_main.c:4489
e1000_clean+0xb94/0x2920 drivers/net/ethernet/intel/e1000/e1000_main.c:3834
napi_poll net/core/dev.c:5171 [inline]
net_rx_action+0xeb4/0x1580 net/core/dev.c:5236
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Freed:
PID = 25328
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_free include/net/dst.h:428 [inline]
dst_rcu_free+0x152/0x190 include/net/dst.h:438
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:
ffff88004b5c0680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88004b5c0700: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
>ffff88004b5c0780: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
^
ffff88004b5c0800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88004b5c0880: fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

==================================================================
BUG: KASAN: slab-out-of-bounds in fib6_prune_clone+0x4e/0x50
net/ipv6/ip6_fib.c:1725 at addr ffff880053497d14
Read of size 4 by task syz-executor1/20792
CPU: 0 PID: 20792 Comm: syz-executor1 Not tainted 4.10.0+ #260
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330
fib6_prune_clone+0x4e/0x50 net/ipv6/ip6_fib.c:1725
fib6_clean_node+0x356/0x550 net/ipv6/ip6_fib.c:1647
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576
fib6_walk+0x1cf/0x300 net/ipv6/ip6_fib.c:1621
fib6_clean_tree+0x266/0x3a0 net/ipv6/ip6_fib.c:1693
fib6_prune_clones net/ipv6/ip6_fib.c:1735 [inline]
fib6_add+0x2612/0x30a0 net/ipv6/ip6_fib.c:1068
__ip6_ins_rt+0x60/0x80 net/ipv6/route.c:948
ip6_route_add+0x1a7/0x310 net/ipv6/route.c:2127
addrconf_prefix_route+0x391/0x560 net/ipv6/addrconf.c:2247
inet6_addr_add+0x2aa/0x370 net/ipv6/addrconf.c:2799
addrconf_add_ifaddr+0x169/0x200 net/ipv6/addrconf.c:2878
inet6_ioctl+0x111/0x1e0 net/ipv6/af_inet6.c:523
sock_do_ioctl+0x65/0xb0 net/socket.c:895
sock_ioctl+0x2c2/0x440 net/socket.c:993
vfs_ioctl fs/ioctl.c:43 [inline]
do_vfs_ioctl+0x1bf/0x1790 fs/ioctl.c:683
SYSC_ioctl fs/ioctl.c:698 [inline]
SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007fce75526b58 EFLAGS: 00000286 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00000000004458d9
RDX: 0000000020000000 RSI: 0000000000008916 RDI: 0000000000000005
RBP: 00000000006df0c0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000708000
R13: 0000000020df4ff5 R14: 0000000000000007 R15: 0000000000034800
Object at ffff880053497c00, in cache ip_dst_cache size: 216
Allocated:
PID = 1306
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
__mkroute_output net/ipv4/route.c:2165 [inline]
__ip_route_output_key_hash+0xce3/0x2c70 net/ipv4/route.c:2375
__ip_route_output_key include/net/route.h:122 [inline]
ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2461
ip_route_output_ports include/net/route.h:159 [inline]
ip_queue_xmit+0x1581/0x1a20 net/ipv4/ip_output.c:459
tcp_transmit_skb+0x1ab4/0x3460 net/ipv4/tcp_output.c:1057
tcp_write_xmit+0x6e6/0x50d0 net/ipv4/tcp_output.c:2260
__tcp_push_pending_frames+0xfa/0x380 net/ipv4/tcp_output.c:2445
tcp_push+0x4e8/0x770 net/ipv4/tcp.c:683
tcp_sendmsg+0x1275/0x39a0 net/ipv4/tcp.c:1337
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
new_sync_write fs/read_write.c:499 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:512
vfs_write+0x187/0x530 fs/read_write.c:560
SYSC_write fs/read_write.c:607 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:599
entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:
PID = 0
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_free include/net/dst.h:428 [inline]
dst_rcu_free+0x152/0x190 include/net/dst.h:438
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:
ffff880053497c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff880053497c80: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
>ffff880053497d00: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
^
ffff880053497d80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff880053497e00: fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

==================================================================
BUG: KASAN: slab-out-of-bounds in rt6_fill_node.isra.61+0x1434/0x1780
net/ipv6/route.c:3396 at addr ffff88004af7a650
Read of size 4 by task syz-executor0/14836
CPU: 1 PID: 14836 Comm: syz-executor0 Not tainted 4.10.0+ #260
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
9pnet_virtio: no channels available for device ./bus
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330
rt6_fill_node.isra.61+0x1434/0x1780 net/ipv6/route.c:3396
rt6_dump_route+0x245/0x2f0 net/ipv6/route.c:3557
fib6_dump_node+0x101/0x1a0 net/ipv6/ip6_fib.c:315
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576
fib6_walk+0x1cf/0x300 net/ipv6/ip6_fib.c:1621
fib6_dump_table net/ipv6/ip6_fib.c:374 [inline]
inet6_dump_fib+0x832/0xea0 net/ipv6/ip6_fib.c:447
rtnl_dump_all+0x8a/0x2a0 net/core/rtnetlink.c:2776
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127
netlink_recvmsg+0xb6a/0x1500 net/netlink/af_netlink.c:1886
sock_recvmsg_nosec net/socket.c:740 [inline]
sock_recvmsg+0xd7/0x110 net/socket.c:747
___sys_recvmsg+0x2b8/0x6b0 net/socket.c:2144
__sys_recvmsg+0x135/0x300 net/socket.c:2189
SYSC_recvmsg net/socket.c:2201 [inline]
SyS_recvmsg+0x2d/0x50 net/socket.c:2196
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:280
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x4458d9
RSP: 002b:00007f84c4ef1b58 EFLAGS: 00000286 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 00000000007083f0 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 00000000206a2fc8 RDI: 000000000000001a
RBP: 00000000000036d0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006e1790
R13: 000000000000001a R14: 00000000206a2fc8 R15: 0000000000000000
Object at ffff88004af7a540, in cache ip_dst_cache size: 216
Allocated:
PID = 1298
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
ip_route_input_slow+0xe67/0x21e0 net/ipv4/route.c:1936
ip_route_input_noref+0x13c/0x10b0 net/ipv4/route.c:2058
ip_rcv_finish+0x301/0x1b40 net/ipv4/ip_input.c:344
NF_HOOK include/linux/netfilter.h:257 [inline]
ip_rcv+0xd75/0x19a0 net/ipv4/ip_input.c:487
__netif_receive_skb_core+0x1ac8/0x33f0 net/core/dev.c:4179
__netif_receive_skb+0x2a/0x170 net/core/dev.c:4217
netif_receive_skb_internal+0xf0/0x400 net/core/dev.c:4245
napi_skb_finish net/core/dev.c:4602 [inline]
napi_gro_receive+0x4d4/0x670 net/core/dev.c:4636
e1000_receive_skb drivers/net/ethernet/intel/e1000/e1000_main.c:4033 [inline]
e1000_clean_rx_irq+0x5e0/0x1490
drivers/net/ethernet/intel/e1000/e1000_main.c:4489
e1000_clean+0xb94/0x2920 drivers/net/ethernet/intel/e1000/e1000_main.c:3834
napi_poll net/core/dev.c:5171 [inline]
net_rx_action+0xeb4/0x1580 net/core/dev.c:5236
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Freed:
PID = 3947
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_destroy_rcu+0x15/0x40 net/core/dst.c:294
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:
ffff88004af7a500: fc fc fc fc fc fc fc fc 00 00 00 00 00 00 00 00
ffff88004af7a580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88004af7a600: 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88004af7a680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88004af7a700: fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc fc
==================================================================

[David Ahern]

On 3/3/17 6:39 AM, Dmitry Vyukov wrote:
> I am getting heap out-of-bounds reports in
> fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone while running
> syzkaller fuzzer on 86292b33d4b79ee03e2f43ea0381ef85f077c760. They all
> follow the same pattern: an object of size 216 is allocated from
> ip_dst_cache slab, and then accessed at offset 272/276 withing
> fib6_walk. Looks like type confusion. Unfortunately this is not
> reproducible.

I'll take a look this weekend or Monday at the latest.

[Dmitry Vyukov]

This is not from fib6_walk, but looks like the same problem:

==================================================================
BUG: KASAN: slab-out-of-bounds in find_rr_leaf net/ipv6/route.c:722
[inline] at addr ffff88004afe6f68
BUG: KASAN: slab-out-of-bounds in rt6_select net/ipv6/route.c:758
[inline] at addr ffff88004afe6f68
BUG: KASAN: slab-out-of-bounds in ip6_pol_route+0x19ff/0x1f30
net/ipv6/route.c:1091 at addr ffff88004afe6f68
Read of size 4 by task syz-executor0/24839
CPU: 1 PID: 24839 Comm: syz-executor0 Not tainted 4.10.0+ #248

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:15 [inline]
dump_stack+0x2ee/0x3ef lib/dump_stack.c:51
kasan_object_err+0x1c/0x70 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:204 [inline]
kasan_report_error mm/kasan/report.c:288 [inline]
kasan_report.part.2+0x198/0x440 mm/kasan/report.c:310
kasan_report mm/kasan/report.c:330 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:330

find_rr_leaf net/ipv6/route.c:722 [inline]
rt6_select net/ipv6/route.c:758 [inline]
ip6_pol_route+0x19ff/0x1f30 net/ipv6/route.c:1091
ip6_pol_route_output+0x4c/0x60 net/ipv6/route.c:1212
fib6_rule_lookup+0x52/0x150 net/ipv6/ip6_fib.c:291
ip6_route_output_flags+0x1f1/0x2b0 net/ipv6/route.c:1240
ip6_route_output include/net/ip6_route.h:79 [inline]
ip6_dst_lookup_tail+0x4fb/0x990 net/ipv6/ip6_output.c:954
ip6_dst_lookup+0x4b/0x60 net/ipv6/ip6_output.c:1056
icmpv6_route_lookup+0x107/0x750 net/ipv6/icmp.c:347
icmp6_send+0x145e/0x24d0 net/ipv6/icmp.c:536
icmpv6_send+0x12e/0x260 net/ipv6/ip6_icmp.c:42
ip6_fragment+0x57f/0x38a0 net/ipv6/ip6_output.c:865
ip6_finish_output+0x319/0x950 net/ipv6/ip6_output.c:147
NF_HOOK_COND include/linux/netfilter.h:246 [inline]
ip6_output+0x1cb/0x8c0 net/ipv6/ip6_output.c:163
dst_output include/net/dst.h:486 [inline]
ip6_local_out+0x95/0x170 net/ipv6/output_core.c:172
ip6_send_skb+0xa1/0x340 net/ipv6/ip6_output.c:1734
ip6_push_pending_frames+0xb3/0xe0 net/ipv6/ip6_output.c:1754
rawv6_push_pending_frames net/ipv6/raw.c:613 [inline]
rawv6_sendmsg+0x2e10/0x3fd0 net/ipv6/raw.c:930

inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
SYSC_sendto+0x660/0x810 net/socket.c:1685
SyS_sendto+0x40/0x50 net/socket.c:1653
entry_SYSCALL_64_fastpath+0x1f/0xc2

RIP: 0033:0x4458d9
RSP: 002b:00007f227bcfab58 EFLAGS: 00000282 ORIG_RAX: 000000000000002c

RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00000000004458d9

RDX: 0000000000001001 RSI: 0000000020725000 RDI: 0000000000000006
RBP: 00000000006e1bb0 R08: 00000000201ccff8 R09: 0000000000000018
R10: 0040000000004004 R11: 0000000000000282 R12: 0000000000708000
R13: 0000000020001ff7 R14: 0000000000000003 R15: 0000000000060040
Object at ffff88004afe6e00, in cache ip_dst_cache size: 216
Allocated:
PID = 1307

save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:605
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:544
kmem_cache_alloc+0x102/0x680 mm/slab.c:3571
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482

ip_route_input_slow+0xdf2/0x2160 net/ipv4/route.c:1935
ip_route_input_noref+0x137/0x10e0 net/ipv4/route.c:2056

ip_rcv_finish+0x301/0x1b40 net/ipv4/ip_input.c:344
NF_HOOK include/linux/netfilter.h:257 [inline]
ip_rcv+0xd75/0x19a0 net/ipv4/ip_input.c:487
__netif_receive_skb_core+0x1ac8/0x33f0 net/core/dev.c:4179
__netif_receive_skb+0x2a/0x170 net/core/dev.c:4217
netif_receive_skb_internal+0xf0/0x400 net/core/dev.c:4245
napi_skb_finish net/core/dev.c:4602 [inline]
napi_gro_receive+0x4d4/0x670 net/core/dev.c:4636
e1000_receive_skb drivers/net/ethernet/intel/e1000/e1000_main.c:4033 [inline]
e1000_clean_rx_irq+0x5e0/0x1490
drivers/net/ethernet/intel/e1000/e1000_main.c:4489
e1000_clean+0xb94/0x2920 drivers/net/ethernet/intel/e1000/e1000_main.c:3834
napi_poll net/core/dev.c:5171 [inline]
net_rx_action+0xeb4/0x1580 net/core/dev.c:5236
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Freed:

PID = 22752

save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:578
__cache_free mm/slab.c:3513 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3773
dst_destroy+0x1fd/0x330 net/core/dst.c:269
dst_destroy_rcu+0x15/0x40 net/core/dst.c:294
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.67+0xa31/0xe50 kernel/rcu/tree.c:2877
invoke_rcu_callbacks kernel/rcu/tree.c:3140 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3107 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3124
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Memory state around the buggy address:

ffff88004afe6e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88004afe6e80: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
>ffff88004afe6f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88004afe6f80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff88004afe7000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================

[Dmitry Vyukov]

On Fri, Mar 3, 2017 at 8:12 PM, David Ahern <d...@cumulusnetworks.com> wrote:

I've got some additional useful info on this. I think this is
use-after-free rather than out-of-bounds. I've collected stack where
the route was disposed with call_rcu, see the last "Disposed" stack.
The crash happens when cmpxchg in rt_cache_route replaces an existing
route. And that route seems to have some existing pointers to it
(rt->dst.rt6_next) which fib6_walk uses to get to it after its
deletion.

==================================================================
BUG: KASAN: slab-out-of-bounds in rt6_dump_route+0x293/0x2f0

net/ipv6/route.c:3551 at addr ffff88007e523694
Read of size 4 by task syz-executor3/24426
CPU: 2 PID: 24426 Comm: syz-executor3 Not tainted 4.10.0+ #293

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:

__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2fb/0x3fd lib/dump_stack.c:52
kasan_object_err+0x1c/0x90 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:208 [inline]
kasan_report_error mm/kasan/report.c:292 [inline]
kasan_report.part.2+0x1b0/0x460 mm/kasan/report.c:314
kasan_report mm/kasan/report.c:334 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:334
rt6_dump_route+0x293/0x2f0 net/ipv6/route.c:3551

fib6_dump_node+0x101/0x1a0 net/ipv6/ip6_fib.c:315
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1576

fib6_walk+0x91/0xf0 net/ipv6/ip6_fib.c:1621

fib6_dump_table net/ipv6/ip6_fib.c:374 [inline]
inet6_dump_fib+0x832/0xea0 net/ipv6/ip6_fib.c:447
rtnl_dump_all+0x8a/0x2a0 net/core/rtnetlink.c:2776
netlink_dump+0x54d/0xd40 net/netlink/af_netlink.c:2127

__netlink_dump_start+0x50e/0x790 net/netlink/af_netlink.c:2217

netlink_dump_start include/linux/netlink.h:165 [inline]
rtnetlink_rcv_msg+0x4a3/0x860 net/core/rtnetlink.c:4094
netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
rtnetlink_rcv+0x2a/0x40 net/core/rtnetlink.c:4110
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]

netlink_unicast+0x525/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xab3/0xe70 net/netlink/af_netlink.c:1803

sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846

call_write_iter include/linux/fs.h:1733 [inline]
new_sync_write fs/read_write.c:497 [inline]
__vfs_write+0x483/0x740 fs/read_write.c:510
vfs_write+0x187/0x530 fs/read_write.c:558
SYSC_write fs/read_write.c:605 [inline]
SyS_write+0xfb/0x230 fs/read_write.c:597
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007feb6f154b58 EFLAGS: 00000292 ORIG_RAX: 0000000000000001

RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00000000004458d9

RDX: 000000000000001f RSI: 00000000208a8000 RDI: 0000000000000006

RBP: 00000000006e2fc0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000292 R12: 0000000000708000

R13: 0000000000000005 R14: 0000000020078fd0 R15: 0000000000000030
Object at ffff88007e523580, in cache ip_dst_cache size: 216
Allocated:
PID = 21468
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:616
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:555
kmem_cache_alloc+0x102/0x6e0 mm/slab.c:3572

dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
__mkroute_output net/ipv4/route.c:2165 [inline]

__ip_route_output_key_hash+0xce3/0x2ca0 net/ipv4/route.c:2375
__ip_route_output_key include/net/route.h:122 [inline]
ip_route_connect include/net/route.h:289 [inline]
tcp_v4_connect+0x11f2/0x2070 net/ipv4/tcp_ipv4.c:170
__inet_stream_connect+0x2d1/0xf90 net/ipv4/af_inet.c:618
inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682
SYSC_connect+0x251/0x580 net/socket.c:1577
SyS_connect+0x24/0x30 net/socket.c:1558
entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:
PID = 20
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:589
__cache_free mm/slab.c:3514 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3774
dst_destroy+0x211/0x340 net/core/dst.c:269

dst_free include/net/dst.h:428 [inline]
dst_rcu_free+0x152/0x190 include/net/dst.h:438
__rcu_reclaim kernel/rcu/rcu.h:118 [inline]

rcu_do_batch.isra.66+0xa31/0xe50 kernel/rcu/tree.c:2880
invoke_rcu_callbacks kernel/rcu/tree.c:3143 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3110 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3127
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Disposed:
PID = 22571
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_set_rcu_track+0xcf/0xf0 mm/kasan/kasan.c:694
__call_rcu.constprop.77+0x1d6/0x15a0 kernel/rcu/tree.c:3230
call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3291
rt_free net/ipv4/route.c:592 [inline]
rt_cache_route+0xf5/0x130 net/ipv4/route.c:1365
rt_set_nexthop.constprop.57+0x408/0xfa0 net/ipv4/route.c:1453
__mkroute_output net/ipv4/route.c:2195 [inline]
__ip_route_output_key_hash+0xe50/0x2ca0 net/ipv4/route.c:2375

__ip_route_output_key include/net/route.h:122 [inline]
ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2461

ip_route_connect include/net/route.h:296 [inline]
tcp_v4_connect+0x784/0x2070 net/ipv4/tcp_ipv4.c:170
__inet_stream_connect+0x2d1/0xf90 net/ipv4/af_inet.c:618
inet_stream_connect+0x58/0xa0 net/ipv4/af_inet.c:682
SYSC_connect+0x251/0x580 net/socket.c:1577
SyS_connect+0x24/0x30 net/socket.c:1558
entry_SYSCALL_64_fastpath+0x1f/0xc2

Memory state around the buggy address:

ffff88007e523580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88007e523600: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
>ffff88007e523680: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
^
ffff88007e523700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88007e523780: fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

[Dmitry Vyukov]

On Sat, Mar 4, 2017 at 7:57 PM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Fri, Mar 3, 2017 at 8:12 PM, David Ahern <d...@cumulusnetworks.com> wrote:
>> On 3/3/17 6:39 AM, Dmitry Vyukov wrote:
>>> I am getting heap out-of-bounds reports in
>>> fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone while running
>>> syzkaller fuzzer on 86292b33d4b79ee03e2f43ea0381ef85f077c760. They all
>>> follow the same pattern: an object of size 216 is allocated from
>>> ip_dst_cache slab, and then accessed at offset 272/276 withing
>>> fib6_walk. Looks like type confusion. Unfortunately this is not
>>> reproducible.
>>
>> I'll take a look this weekend or Monday at the latest.
>
>
> I've got some additional useful info on this. I think this is
> use-after-free rather than out-of-bounds. I've collected stack where
> the route was disposed with call_rcu, see the last "Disposed" stack.
> The crash happens when cmpxchg in rt_cache_route replaces an existing
> route. And that route seems to have some existing pointers to it
> (rt->dst.rt6_next) which fib6_walk uses to get to it after its
> deletion.

This could explain the gazillion of various crashes I am seeing in ip6 routes:

KASAN: use-after-free Read in fib6_add
WARNING in fib6_del
KASAN: use-after-free Read in ip6_pol_route
general protection fault in fib6_add
general protection fault in ip6_rt_cache_alloc
general protection fault in sctp_v6_get_dst
KASAN: slab-out-of-bounds Read in rt6_dump_route
KASAN: slab-out-of-bounds Read in fib6_prune_clone
KASAN: slab-out-of-bounds Read in fib6_age
KASAN: slab-out-of-bounds Read in ip6_pol_route
KASAN: slab-out-of-bounds Read in rt6_fill_node

[Eric Dumazet]

On Sat, 2017-03-04 at 19:57 +0100, Dmitry Vyukov wrote:
> On Fri, Mar 3, 2017 at 8:12 PM, David Ahern <d...@cumulusnetworks.com> wrote:
> > On 3/3/17 6:39 AM, Dmitry Vyukov wrote:
> >> I am getting heap out-of-bounds reports in
> >> fib6_clean_node/rt6_fill_node/fib6_age/fib6_prune_clone while running
> >> syzkaller fuzzer on 86292b33d4b79ee03e2f43ea0381ef85f077c760. They all
> >> follow the same pattern: an object of size 216 is allocated from
> >> ip_dst_cache slab, and then accessed at offset 272/276 withing
> >> fib6_walk. Looks like type confusion. Unfortunately this is not
> >> reproducible.
> >
> > I'll take a look this weekend or Monday at the latest.
>
>
> I've got some additional useful info on this. I think this is
> use-after-free rather than out-of-bounds. I've collected stack where
> the route was disposed with call_rcu, see the last "Disposed" stack.
> The crash happens when cmpxchg in rt_cache_route replaces an existing
> route. And that route seems to have some existing pointers to it
> (rt->dst.rt6_next) which fib6_walk uses to get to it after its
> deletion.

rt_cache_route() deals with IPv4 routes.

We somehow mix IPv4 and IPv6 dsts in IPv6 tree.

We need to add type safety at IPV6 route insertions to catch the
offender.

[Dmitry Vyukov]

If you suggest additional checks, I will collect stacks.

[David Ahern]

I've seen something like this before -- a rt was on the gc list but
still linked in the tables because of some reference.

Dmitry: you seem to have reproduced this a few times. Can you share how
to run whatever tests you are using?

[Dmitry Vyukov]

We hit it several thousand times, but we get only several dozens of
crashes per day on ~80 VMs. So if you try to reproduce it on a single
machine it can take days for a single crash.
If you are ready to go that route, here are some instructions on
setting up syzkaller:
https://github.com/google/syzkaller
You also need kernel built with CONFIG_KASAN.
I am ready to help with resolving any issues.

Another possible route is if you give me a patch with some additional
WARNINGs. Then I can deploy it to bots and collect stacks.

[David Ahern]

On 3/6/17 11:51 AM, Dmitry Vyukov wrote:
> We hit it several thousand times, but we get only several dozens of
> crashes per day on ~80 VMs. So if you try to reproduce it on a single
> machine it can take days for a single crash.
> If you are ready to go that route, here are some instructions on
> setting up syzkaller:
> https://github.com/google/syzkaller
> You also need kernel built with CONFIG_KASAN.

ack and I have it setup on ubuntu 16.10 which has a fairly new compiler.

> I am ready to help with resolving any issues.
>
> Another possible route is if you give me a patch with some additional
> WARNINGs. Then I can deploy it to bots and collect stacks.

try the attached.

[Dmitry Vyukov]

This is on c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201. No other kernel
output from your patch (pr_err).

------------[ cut here ]------------
WARNING: CPU: 1 PID: 30179 at net/ipv6/ip6_fib.c:158
rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158
Kernel panic - not syncing: panic_on_warn set ...

CPU: 1 PID: 30179 Comm: syz-executor3 Not tainted 4.11.0-rc1+ #310

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2fb/0x3fd lib/dump_stack.c:52

panic+0x20f/0x426 kernel/panic.c:180
__warn+0x1c4/0x1e0 kernel/panic.c:541
warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158
rt6_release+0x1ee/0x290 net/ipv6/ip6_fib.c:189
fib6_add_rt2node net/ipv6/ip6_fib.c:922 [inline]
fib6_add+0x1d51/0x3290 net/ipv6/ip6_fib.c:1081
__ip6_ins_rt+0x60/0x80 net/ipv6/route.c:948
ip6_route_add+0x1a7/0x310 net/ipv6/route.c:2130
inet6_rtm_newroute+0x191/0x1b0 net/ipv6/route.c:3294
rtnetlink_rcv_msg+0x609/0x860 net/core/rtnetlink.c:4104

netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
rtnetlink_rcv+0x2a/0x40 net/core/rtnetlink.c:4110
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x525/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xab3/0xe70 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]

do_iter_readv_writev fs/read_write.c:696 [inline]
__do_readv_writev+0xbbc/0x10a0 fs/read_write.c:862
do_readv_writev+0x13f/0x200 fs/read_write.c:894
vfs_writev+0x87/0xc0 fs/read_write.c:921
do_writev+0x110/0x2c0 fs/read_write.c:954
SYSC_writev fs/read_write.c:1027 [inline]
SyS_writev+0x27/0x30 fs/read_write.c:1024
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007f31fcf33b58 EFLAGS: 00000292 ORIG_RAX: 0000000000000014

RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00000000004458d9

RDX: 0000000000000001 RSI: 00000000207cd000 RDI: 0000000000000005
RBP: 00000000006e30c0 R08: 0000000000000000 R09: 0000000000000000

R10: 0000000000000000 R11: 0000000000000292 R12: 0000000000708000

R13: 0000000020fad000 R14: 0000000000001000 R15: 0000000000000003



------------[ cut here ]------------
WARNING: CPU: 2 PID: 31175 at net/ipv6/ip6_fib.c:158
rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158
Kernel panic - not syncing: panic_on_warn set ...

CPU: 2 PID: 31175 Comm: syz-executor1 Not tainted 4.11.0-rc1+ #310

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2fb/0x3fd lib/dump_stack.c:52

panic+0x20f/0x426 kernel/panic.c:180
__warn+0x1c4/0x1e0 kernel/panic.c:541
warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158
rt6_release+0x1ee/0x290 net/ipv6/ip6_fib.c:189
fib6_add_rt2node net/ipv6/ip6_fib.c:922 [inline]
fib6_add+0x1d51/0x3290 net/ipv6/ip6_fib.c:1081
kvm_vm_ioctl_deassign_device: device hasn't been assigned before, so
cannot be deassigned
__ip6_ins_rt+0x60/0x80 net/ipv6/route.c:948
ip6_route_add+0x1a7/0x310 net/ipv6/route.c:2130
inet6_rtm_newroute+0x191/0x1b0 net/ipv6/route.c:3294
rtnetlink_rcv_msg+0x609/0x860 net/core/rtnetlink.c:4104

netlink_rcv_skb+0x2ab/0x390 net/netlink/af_netlink.c:2298
rtnetlink_rcv+0x2a/0x40 net/core/rtnetlink.c:4110
netlink_unicast_kernel net/netlink/af_netlink.c:1231 [inline]
netlink_unicast+0x525/0x730 net/netlink/af_netlink.c:1257
netlink_sendmsg+0xab3/0xe70 net/netlink/af_netlink.c:1803
sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643
sock_write_iter+0x326/0x600 net/socket.c:846
call_write_iter include/linux/fs.h:1733 [inline]

do_iter_readv_writev fs/read_write.c:696 [inline]
__do_readv_writev+0xbbc/0x10a0 fs/read_write.c:862
do_readv_writev+0x13f/0x200 fs/read_write.c:894
vfs_writev+0x87/0xc0 fs/read_write.c:921
do_writev+0x110/0x2c0 fs/read_write.c:954
SYSC_writev fs/read_write.c:1027 [inline]
SyS_writev+0x27/0x30 fs/read_write.c:1024
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x4458d9
RSP: 002b:00007f1639006b58 EFLAGS: 00000292 ORIG_RAX: 0000000000000014
RAX: ffffffffffffffda RBX: 0000000000000019 RCX: 00000000004458d9
RDX: 0000000000000001 RSI: 00000000207cd000 RDI: 0000000000000019
RBP: 00000000006e30c0 R08: 0000000000000000 R09: 0000000000000000

R10: 0000000000000000 R11: 0000000000000292 R12: 0000000000708000

R13: 0000000000000010 R14: 0000000000000003 R15: 0000000000000000

[Dmitry Vyukov]

I've commented that warning just to see I can obtain more information.
Then I also got this:

------------[ cut here ]------------
WARNING: CPU: 2 PID: 3990 at net/ipv6/ip6_fib.c:991
fib6_add+0x2e12/0x3290 net/ipv6/ip6_fib.c:991 net/ipv6/ip6_fib.c:991

Kernel panic - not syncing: panic_on_warn set ...

CPU: 2 PID: 3990 Comm: kworker/2:4 Not tainted 4.11.0-rc1+ #311

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011

Workqueue: ipv6_addrconf addrconf_dad_work

Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]

__dump_stack lib/dump_stack.c:16 [inline] lib/dump_stack.c:52
dump_stack+0x2fb/0x3fd lib/dump_stack.c:52 lib/dump_stack.c:52
panic+0x20f/0x426 kernel/panic.c:180 kernel/panic.c:180
__warn+0x1c4/0x1e0 kernel/panic.c:541 kernel/panic.c:541
warn_slowpath_null+0x2c/0x40 kernel/panic.c:584 kernel/panic.c:584
fib6_add+0x2e12/0x3290 net/ipv6/ip6_fib.c:991 net/ipv6/ip6_fib.c:991
__ip6_ins_rt+0x60/0x80 net/ipv6/route.c:948 net/ipv6/route.c:948
ip6_ins_rt+0x19b/0x220 net/ipv6/route.c:959 net/ipv6/route.c:959
__ipv6_ifa_notify+0x62e/0x7a0 net/ipv6/addrconf.c:5485 net/ipv6/addrconf.c:5485
ipv6_ifa_notify+0xdf/0x1d0 net/ipv6/addrconf.c:5518 net/ipv6/addrconf.c:5518
addrconf_dad_completed+0xe6/0x950 net/ipv6/addrconf.c:3983
net/ipv6/addrconf.c:3983
addrconf_dad_begin net/ipv6/addrconf.c:3797 [inline]
addrconf_dad_begin net/ipv6/addrconf.c:3797 [inline] net/ipv6/addrconf.c:3897
addrconf_dad_work+0x32a/0xea0 net/ipv6/addrconf.c:3897 net/ipv6/addrconf.c:3897
process_one_work+0xc06/0x1c40 kernel/workqueue.c:2096 kernel/workqueue.c:2096
worker_thread+0x223/0x19f0 kernel/workqueue.c:2230 kernel/workqueue.c:2230
kthread+0x334/0x400 kernel/kthread.c:229 kernel/kthread.c:229
ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
arch/x86/entry/entry_64.S:430



And this without any preceding warnings:

==================================================================
BUG: KASAN: slab-out-of-bounds in fib6_age+0x3fd/0x480
net/ipv6/ip6_fib.c:1787 at addr ffff88004d4fbe54
Read of size 4 by task swapper/2/0
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.11.0-rc1+ #311

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:

<IRQ>

__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x2fb/0x3fd lib/dump_stack.c:52

kasan_object_err+0x1c/0x90 mm/kasan/report.c:166
print_address_description mm/kasan/report.c:208 [inline]
kasan_report_error mm/kasan/report.c:292 [inline]
kasan_report.part.2+0x1b0/0x460 mm/kasan/report.c:314
kasan_report mm/kasan/report.c:334 [inline]
__asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:334

fib6_age+0x3fd/0x480 net/ipv6/ip6_fib.c:1787
fib6_clean_node+0x356/0x550 net/ipv6/ip6_fib.c:1665
fib6_walk_continue+0x4b3/0x620 net/ipv6/ip6_fib.c:1594
fib6_walk+0x91/0xf0 net/ipv6/ip6_fib.c:1639
fib6_clean_tree+0x266/0x3a0 net/ipv6/ip6_fib.c:1711
__fib6_clean_all+0x1e1/0x360 net/ipv6/ip6_fib.c:1727
fib6_clean_all net/ipv6/ip6_fib.c:1738 [inline]
fib6_run_gc+0x185/0x3d0 net/ipv6/ip6_fib.c:1835
fib6_gc_timer_cb+0x1c/0x20 net/ipv6/ip6_fib.c:1850
call_timer_fn+0x241/0x820 kernel/time/timer.c:1268
expire_timers kernel/time/timer.c:1307 [inline]
__run_timers+0x960/0xcf0 kernel/time/timer.c:1601
run_timer_softirq+0x21/0x80 kernel/time/timer.c:1614

__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:405

exiting_irq arch/x86/include/asm/apic.h:657 [inline]

smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:962
apic_timer_interrupt+0x93/0xa0 arch/x86/entry/entry_64.S:487
RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:53

RSP: 0018:ffff880089437c10 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: 1ffff10011286f85 RCX: 0000000000000000

RDX: 1ffffffff0a18ebc RSI: 0000000000000001 RDI: ffffffff850c75e0

RBP: ffff880089437c10 R08: ffffed00113835c2 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff10011286fa9
R13: ffff880089437cc8 R14: ffffffff856973f8 R15: ffff880089437e68

</IRQ>
arch_safe_halt arch/x86/include/asm/paravirt.h:98 [inline]

default_idle+0xbf/0x440 arch/x86/kernel/process.c:275
arch_cpu_idle+0xa/0x10 arch/x86/kernel/process.c:266
default_idle_call+0x36/0x90 kernel/sched/idle.c:97
cpuidle_idle_call kernel/sched/idle.c:155 [inline]
do_idle+0x373/0x520 kernel/sched/idle.c:244
cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:346
start_secondary+0x36c/0x460 arch/x86/kernel/smpboot.c:275
start_cpu+0x14/0x14 arch/x86/kernel/head_64.S:306
Object at ffff88004d4fbd40, in cache ip_dst_cache size: 216
Allocated:
PID = 8122

save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_kmalloc+0xaa/0xd0 mm/kasan/kasan.c:616
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:555
kmem_cache_alloc+0x102/0x6e0 mm/slab.c:3572
dst_alloc+0x11b/0x1a0 net/core/dst.c:209
rt_dst_alloc+0xf0/0x580 net/ipv4/route.c:1482
__mkroute_output net/ipv4/route.c:2165 [inline]
__ip_route_output_key_hash+0xce3/0x2ca0 net/ipv4/route.c:2375
__ip_route_output_key include/net/route.h:122 [inline]

ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2461
ip_route_output_key include/net/route.h:132 [inline]
sctp_v4_get_dst+0x5d2/0x1570 net/sctp/protocol.c:458
sctp_transport_route+0xa8/0x420 net/sctp/transport.c:292
sctp_assoc_add_peer+0x5a5/0x1470 net/sctp/associola.c:653

sctp_sendmsg+0x180d/0x3980 net/sctp/socket.c:1871
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761

sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643

SYSC_sendto+0x660/0x810 net/socket.c:1685
SyS_sendto+0x40/0x50 net/socket.c:1653
entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:

PID = 2038

save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_slab_free+0x6f/0xb0 mm/kasan/kasan.c:589
__cache_free mm/slab.c:3514 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3774

dst_destroy+0x211/0x340 net/core/dst.c:272
dst_free include/net/dst.h:429 [inline]
dst_rcu_free+0x152/0x190 include/net/dst.h:439

__rcu_reclaim kernel/rcu/rcu.h:118 [inline]
rcu_do_batch.isra.66+0xa31/0xe50 kernel/rcu/tree.c:2880
invoke_rcu_callbacks kernel/rcu/tree.c:3143 [inline]
__rcu_process_callbacks kernel/rcu/tree.c:3110 [inline]
rcu_process_callbacks+0x45b/0xc50 kernel/rcu/tree.c:3127
__do_softirq+0x31f/0xbe7 kernel/softirq.c:284
Disposed:

PID = 26270

save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x43/0xd0 mm/kasan/kasan.c:513
set_track mm/kasan/kasan.c:525 [inline]
kasan_set_rcu_track+0xcf/0xf0 mm/kasan/kasan.c:694
__call_rcu.constprop.77+0x1d6/0x15a0 kernel/rcu/tree.c:3230
call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3291
rt_free net/ipv4/route.c:592 [inline]
rt_cache_route+0xf5/0x130 net/ipv4/route.c:1365
rt_set_nexthop.constprop.57+0x408/0xfa0 net/ipv4/route.c:1453
__mkroute_output net/ipv4/route.c:2195 [inline]
__ip_route_output_key_hash+0xe50/0x2ca0 net/ipv4/route.c:2375
__ip_route_output_key include/net/route.h:122 [inline]
ip_route_output_flow+0x29/0xa0 net/ipv4/route.c:2461

ip_route_output_key include/net/route.h:132 [inline]
sctp_v4_get_dst+0x5d2/0x1570 net/sctp/protocol.c:458
sctp_transport_route+0xa8/0x420 net/sctp/transport.c:292
sctp_assoc_add_peer+0x5a5/0x1470 net/sctp/associola.c:653

sctp_process_param net/sctp/sm_make_chunk.c:2548 [inline]
sctp_process_init+0xf71/0x2320 net/sctp/sm_make_chunk.c:2354
sctp_sf_do_unexpected_init.isra.28+0x7b8/0x1470 net/sctp/sm_statefuns.c:1510
sctp_sf_do_5_2_1_siminit+0x35/0x40 net/sctp/sm_statefuns.c:1199
sctp_do_sm+0x1e5/0x6a30 net/sctp/sm_sideeffect.c:1144
sctp_assoc_bh_rcv+0x285/0x4b0 net/sctp/associola.c:1063
sctp_inq_push+0x22b/0x2e0 net/sctp/inqueue.c:95
sctp_backlog_rcv+0x177/0xb40 net/sctp/input.c:350
sk_backlog_rcv include/net/sock.h:896 [inline]
__release_sock+0x126/0x3a0 net/core/sock.c:2058
release_sock+0xa5/0x2b0 net/core/sock.c:2545
sctp_sendmsg+0x2b05/0x3980 net/sctp/socket.c:2011
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:761

sock_sendmsg_nosec net/socket.c:633 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:643

SYSC_sendto+0x660/0x810 net/socket.c:1685
SyS_sendto+0x40/0x50 net/socket.c:1653

entry_SYSCALL_64_fastpath+0x1f/0xc2
Memory state around the buggy address:

ffff88004d4fbd00: fc fc fc fc fc fc fc fc 00 00 00 00 00 00 00 00
ffff88004d4fbd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88004d4fbe00: 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff88004d4fbe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88004d4fbf00: 00 00 00 00 00 00 00 00 00 00 00 fc fc fc fc fc
==================================================================

[David Ahern]

On 3/7/17 1:43 AM, Dmitry Vyukov wrote:
> This is on c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201. No other kernel
> output from your patch (pr_err).

Is the below supposed to be from the same qemu instance at the time of
the crash? cpu1 and cpu2 are both supposedly doing a route insert?

[Dmitry Vyukov]

On Tue, Mar 7, 2017 at 6:17 PM, 'David Ahern' via syzkaller
<syzk...@googlegroups.com> wrote:
> On 3/7/17 1:43 AM, Dmitry Vyukov wrote:
>> This is on c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201. No other kernel
>> output from your patch (pr_err).
>
> Is the below supposed to be from the same qemu instance at the time of
> the crash? cpu1 and cpu2 are both supposedly doing a route insert?

No, it's all from different instances.

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Ahern]

On 3/7/17 1:43 AM, Dmitry Vyukov wrote:

> This is on c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201. No other kernel
> output from your patch (pr_err).
>
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 30179 at net/ipv6/ip6_fib.c:158
> rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158
> Kernel panic - not syncing: panic_on_warn set ...

you have panic_on_warn set ...

>
> CPU: 1 PID: 30179 Comm: syz-executor3 Not tainted 4.11.0-rc1+ #310
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> dump_stack+0x2fb/0x3fd lib/dump_stack.c:52
> panic+0x20f/0x426 kernel/panic.c:180
> __warn+0x1c4/0x1e0 kernel/panic.c:541
> warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
> rt6_rcu_free+0x61/0x70 net/ipv6/ip6_fib.c:158

and this is my WARN_ON in rt6_rcu_free which is showing an additional
change is needed

> rt6_release+0x1ee/0x290 net/ipv6/ip6_fib.c:189
> fib6_add_rt2node net/ipv6/ip6_fib.c:922 [inline]

in fib6_add_rt2node for the route replace path (whitespace damaged on
the copy-paste):

@@ -916,6 +919,7 @@ static int fib6_add_rt2node(struct fib6_node *fn,
struct rt6_info *rt,
}
nsiblings = iter->rt6i_nsiblings;
fib6_purge_rt(iter, fn, info->nl_net);
+ iter->dst.flags &= ~DST_IN_FIB;
rt6_release(iter);

if (nsiblings) {
@@ -926,6 +930,7 @@ static int fib6_add_rt2node(struct fib6_node *fn,
struct rt6_info *rt,
if (rt6_qualify_for_ecmp(iter)) {
*ins = iter->dst.rt6_next;
fib6_purge_rt(iter, fn,
info->nl_net);
+ iter->dst.flags &= ~DST_IN_FIB;
rt6_release(iter);
nsiblings--;
} else {

[David Ahern]

On 3/7/17 2:21 AM, Dmitry Vyukov wrote:
> I've commented that warning just to see I can obtain more information.
> Then I also got this:
>
> ------------[ cut here ]------------
> WARNING: CPU: 2 PID: 3990 at net/ipv6/ip6_fib.c:991
> fib6_add+0x2e12/0x3290 net/ipv6/ip6_fib.c:991 net/ipv6/ip6_fib.c:991
> Kernel panic - not syncing: panic_on_warn set ...

again panic_on_warn is triggering ...

>
> CPU: 2 PID: 3990 Comm: kworker/2:4 Not tainted 4.11.0-rc1+ #311
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Workqueue: ipv6_addrconf addrconf_dad_work
> Call Trace:
> __dump_stack lib/dump_stack.c:16 [inline]
> __dump_stack lib/dump_stack.c:16 [inline] lib/dump_stack.c:52
> dump_stack+0x2fb/0x3fd lib/dump_stack.c:52 lib/dump_stack.c:52
> panic+0x20f/0x426 kernel/panic.c:180 kernel/panic.c:180
> __warn+0x1c4/0x1e0 kernel/panic.c:541 kernel/panic.c:541
> warn_slowpath_null+0x2c/0x40 kernel/panic.c:584 kernel/panic.c:584
> fib6_add+0x2e12/0x3290 net/ipv6/ip6_fib.c:991 net/ipv6/ip6_fib.c:991

on this warning:

/* dst.next really should not be set at this point */
if (rt->dst.next && rt->dst.next->ops->family != AF_INET6) {
pr_warn("fib6_add: adding rt with bad next -- family %d dst
flags %x\n",
rt->dst.next->ops->family, rt->dst.next->flags);

WARN_ON(1);
}

You should have seen the pr_warn in the log preceding the WARN_ON dump.

> __ip6_ins_rt+0x60/0x80 net/ipv6/route.c:948 net/ipv6/route.c:948
> ip6_ins_rt+0x19b/0x220 net/ipv6/route.c:959 net/ipv6/route.c:959
> __ipv6_ifa_notify+0x62e/0x7a0 net/ipv6/addrconf.c:5485 net/ipv6/addrconf.c:5485
> ipv6_ifa_notify+0xdf/0x1d0 net/ipv6/addrconf.c:5518 net/ipv6/addrconf.c:5518
> addrconf_dad_completed+0xe6/0x950 net/ipv6/addrconf.c:3983
> net/ipv6/addrconf.c:3983
> addrconf_dad_begin net/ipv6/addrconf.c:3797 [inline]
> addrconf_dad_begin net/ipv6/addrconf.c:3797 [inline] net/ipv6/addrconf.c:3897
> addrconf_dad_work+0x32a/0xea0 net/ipv6/addrconf.c:3897 net/ipv6/addrconf.c:3897
> process_one_work+0xc06/0x1c40 kernel/workqueue.c:2096 kernel/workqueue.c:2096
> worker_thread+0x223/0x19f0 kernel/workqueue.c:2230 kernel/workqueue.c:2230
> kthread+0x334/0x400 kernel/kthread.c:229 kernel/kthread.c:229
> ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
> arch/x86/entry/entry_64.S:430
>
>
>
> And this without any preceding warnings:
>
> ==================================================================
> BUG: KASAN: slab-out-of-bounds in fib6_age+0x3fd/0x480
> net/ipv6/ip6_fib.c:1787 at addr ffff88004d4fbe54

another ipv4 route in ipv6 fib walk

[Dmitry Vyukov]

Right. They all have the same "IPv6: fib6_add: adding rt with bad next
-- family 2 dst flags 6"

[ 171.222795] IPv6: fib6_add: adding rt with bad next -- family 2 dst flags 6
[ 171.223809] ------------[ cut here ]------------
[ 171.224407] WARNING: CPU: 3 PID: 27 at net/ipv6/ip6_fib.c:991
fib6_add+0x2e12/0x3290
[ 171.225327] Kernel panic - not syncing: panic_on_warn set ...
[ 171.225327]
[ 171.226066] CPU: 3 PID: 27 Comm: kworker/3:0 Not tainted 4.11.0-rc1+ #311
[ 171.226304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 171.226304] Workqueue: ipv6_addrconf addrconf_dad_work
[ 171.226304] Call Trace:
[ 171.226304] dump_stack+0x2fb/0x3fd
[ 171.226304] ? arch_local_irq_restore+0x53/0x53
[ 171.226304] ? vprintk_emit+0x566/0x770
[ 171.226304] ? console_unlock+0xf50/0xf50
[ 171.226304] ? vprintk_emit+0x566/0x770
[ 171.226304] ? console_unlock+0xf50/0xf50
[ 171.226304] ? vprintk_emit+0x566/0x770
[ 171.226304] ? console_unlock+0xf50/0xf50
[ 171.226304] ? check_noncircular+0x20/0x20
[ 171.226304] ? trace_hardirqs_on+0xd/0x10
[ 171.226304] ? perf_trace_lock_acquire+0x141/0xa00
[ 171.226304] ? trace_hardirqs_off+0xd/0x10
[ 171.226304] ? quarantine_put+0xea/0x190
[ 171.226304] ? check_noncircular+0x20/0x20
[ 171.236060] ? vprintk_default+0x28/0x30
[ 171.236662] ? vprintk_func+0x47/0x90
[ 171.236662] ? printk+0xc8/0xf9
[ 171.236662] ? load_image_and_restore+0x134/0x134
[ 171.236662] ? pointer+0xac0/0xac0
[ 171.236662] panic+0x20f/0x426
[ 171.236662] ? copy_mm+0x1219/0x1219
[ 171.236662] ? vprintk_func+0x47/0x90
[ 171.236662] ? printk+0xc8/0xf9
[ 171.236662] ? fib6_add+0x2e12/0x3290
[ 171.236662] __warn+0x1c4/0x1e0
[ 171.236662] warn_slowpath_null+0x2c/0x40
[ 171.236662] fib6_add+0x2e12/0x3290
[ 171.236662] ? kasan_check_write+0x14/0x20
[ 171.236662] ? netlink_broadcast_filtered+0x734/0x1380
[ 171.236662] ? fib6_force_start_gc+0xf0/0xf0
[ 171.236662] ? netlink_has_listeners+0x450/0x450
[ 171.236662] ? memcpy+0x45/0x50
[ 171.236662] ? __nla_put+0x37/0x40
[ 171.236662] ? nla_put+0xf9/0x130
[ 171.236662] ? skb_put+0x149/0x1c0
[ 171.236662] ? kasan_check_write+0x14/0x20
[ 171.236662] ? do_raw_write_lock+0xbd/0x1e0
[ 171.236662] __ip6_ins_rt+0x60/0x80
[ 171.236662] ip6_ins_rt+0x19b/0x220
[ 171.236662] ? ip6_route_info_create+0x2380/0x2380
[ 171.236662] ? nlmsg_notify+0xaf/0x160
[ 171.236662] ? rtnl_notify+0xbb/0xe0
[ 171.236662] __ipv6_ifa_notify+0x62e/0x7a0
[ 171.251057] ipv6_ifa_notify+0xdf/0x1d0
[ 171.251057] ? __ipv6_ifa_notify+0x7a0/0x7a0
[ 171.251057] addrconf_dad_completed+0xe6/0x950
[ 171.251057] ? addrconf_verify_work+0x20/0x20
[ 171.251057] ? kasan_check_write+0x14/0x20
[ 171.251057] addrconf_dad_work+0x32a/0xea0
[ 171.251057] ? addrconf_ifdown+0x1ad0/0x1ad0
[ 171.251057] ? rcu_pm_notify+0xc0/0xc0
[ 171.251057] ? wq_update_unbound_numa+0x8d0/0x8d0
[ 171.251057] ? kasan_check_write+0x14/0x20
[ 171.251057] process_one_work+0xc06/0x1c40
[ 171.251057] ? process_one_work+0xb3d/0x1c40
[ 171.251057] ? pwq_dec_nr_in_flight+0x470/0x470
[ 171.251057] ? preempt_notifier_register+0x1f0/0x1f0
[ 171.259856] ? __schedule+0x893/0x22d0
[ 171.259856] ? kasan_check_write+0x14/0x20
[ 171.259856] ? worker_thread+0x47d/0x19f0
[ 171.259856] ? lock_set_class+0xc00/0xc00
[ 171.259856] ? worker_thread+0x467/0x19f0
[ 171.259856] ? lock_acquire+0x630/0x630
[ 171.259856] ? _raw_spin_unlock_irq+0x27/0x70
[ 171.259856] ? check_noncircular+0x20/0x20
[ 171.259856] ? mark_held_locks+0x100/0x100
[ 171.259856] ? trace_hardirqs_on_thunk+0x1a/0x1c
[ 171.259856] ? __schedule+0x22d0/0x22d0
[ 171.259856] ? do_raw_spin_trylock+0x1a0/0x1a0
[ 171.259856] ? do_raw_spin_lock+0xbd/0x1f0
[ 171.259856] worker_thread+0x223/0x19f0
[ 171.259856] ? process_one_work+0x1c40/0x1c40
[ 171.259856] ? lock_repin_lock+0x4a0/0x4a0
[ 171.259856] ? unwind_dump.isra.5.part.6+0x320/0x320
[ 171.259856] ? kasan_check_write+0x14/0x20
[ 171.259856] ? finish_task_switch+0x1ea/0x740
[ 171.259856] ? finish_task_switch+0x196/0x740
[ 171.259856] ? preempt_notifier_register+0x1f0/0x1f0
[ 171.259856] ? __schedule+0x893/0x22d0
[ 171.259856] ? lockdep_count_backward_deps+0x480/0x480
[ 171.259856] ? ret_from_fork+0x31/0x40
[ 171.259856] ? do_raw_spin_lock+0xbd/0x1f0
[ 171.259856] ? complete+0xbf/0x190
[ 171.259856] ? register_lock_class+0x1c30/0x1c30
[ 171.276560] ? __wake_up_common+0xb4/0x150
[ 171.276560] ? rcu_pm_notify+0xc0/0xc0
[ 171.276560] ? __schedule+0x22d0/0x22d0
[ 171.276560] ? __init_waitqueue_head+0x8a/0x120
[ 171.276560] ? __wake_up_bit+0x290/0x290
[ 171.279715] ? preempt_notifier_register+0x1f0/0x1f0
[ 171.279715] ? __kthread_parkme+0x173/0x240
[ 171.279715] kthread+0x334/0x400
[ 171.279715] ? process_one_work+0x1c40/0x1c40
[ 171.279715] ? kthread_create_on_node+0x110/0x110
[ 171.279715] ret_from_fork+0x31/0x40
[ 171.279715] Dumping ftrace buffer:
[ 171.279715] (ftrace buffer empty)
[ 171.279715] Kernel Offset: disabled
[ 171.279715] Rebooting in 86400 seconds..

[David Ahern]

On 3/7/17 11:13 AM, Dmitry Vyukov wrote:
>> on this warning:
>>
>> /* dst.next really should not be set at this point */
>> if (rt->dst.next && rt->dst.next->ops->family != AF_INET6) {
>> pr_warn("fib6_add: adding rt with bad next -- family %d dst
>> flags %x\n",
>> rt->dst.next->ops->family, rt->dst.next->flags);
>>
>> WARN_ON(1);
>> }
>>
>> You should have seen the pr_warn in the log preceding the WARN_ON dump.
>
> Right. They all have the same "IPv6: fib6_add: adding rt with bad next
> -- family 2 dst flags 6"

remove the previous changes and try the attached.

[Dmitry Vyukov]

Doing this now.
FWIW I've also applied your last patch with missing "iter->dst.flags
&= ~DST_IN_FIB;" and restored the warning in rt6_rcu_free and it did
not fire (in a limited run). I only saw the "WARNING in fib6_add" that
I already reported.

[Dmitry Vyukov]

So far I've hit only:
[ 1103.840031] BUG: KASAN: slab-out-of-bounds in fib6_age+0x3fd/0x480
at addr ffff8800799d2254
without any preceeding warnings.
But note that since the kernel is heavily stressed I can reliably get
any pr_err output if it happens right before BUG/WARNING. Anything
that happens minutes before will be lots because there are tons of
output.

[Dmitry Vyukov]

So far 6 "KASAN: slab-out-of-bounds Read in fib6_age" but no other warnings.

[Dmitry Vyukov]

I've got a bunch of the crashes that I was getting previously, but no
new warnings.

[Dmitry Vyukov]

A friendly ping. This still happens all the time for us.

I also see the following warning, not sure if it's related or not:

on 0dc82fa59b9d82469799c354d3307d48e13d5d5e:

#if RT6_DEBUG >= 2
if (rt->dst.obsolete > 0) {
WARN_ON(fn);
return -ENOENT;
}
#endif

------------[ cut here ]------------
WARNING: CPU: 1 PID: 23535 at net/ipv6/ip6_fib.c:1472
fib6_del+0x923/0x14d0 net/ipv6/ip6_fib.c:1472
CPU: 1 PID: 23535 Comm: syz-executor3 Not tainted 4.11.0-rc3+ #517
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011

Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]

dump_stack+0x2ee/0x3ef lib/dump_stack.c:52
panic+0x1fb/0x412 kernel/panic.c:180
__warn+0x1c4/0x1e0 kernel/panic.c:541
warn_slowpath_null+0x2c/0x40 kernel/panic.c:584
fib6_del+0x923/0x14d0 net/ipv6/ip6_fib.c:1472
__ip6_del_rt+0x100/0x160 net/ipv6/route.c:2153
ip6_del_rt+0x140/0x1b0 net/ipv6/route.c:2166
__ipv6_ifa_notify+0x269/0x780 net/ipv6/addrconf.c:5506
ipv6_ifa_notify+0xdf/0x1d0 net/ipv6/addrconf.c:5518
ipv6_del_addr+0x62b/0xa80 net/ipv6/addrconf.c:1175
inet6_addr_del+0x348/0x5b0 net/ipv6/addrconf.c:2853
addrconf_del_ifaddr+0x154/0x1e0 net/ipv6/addrconf.c:2898
inet6_ioctl+0x86/0x1e0 net/ipv6/af_inet6.c:525
sock_do_ioctl+0x65/0xb0 net/socket.c:906
sock_ioctl+0x2c2/0x440 net/socket.c:1004
vfs_ioctl fs/ioctl.c:45 [inline]
do_vfs_ioctl+0x1bf/0x1790 fs/ioctl.c:685
SYSC_ioctl fs/ioctl.c:700 [inline]
SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x44fb79
RSP: 002b:00007f4b299bfb58 EFLAGS: 00000212 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000008936 RCX: 000000000044fb79
RDX: 0000000020000000 RSI: 0000000000008936 RDI: 000000000000001a
RBP: 000000000000001a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000212 R12: 0000000000708000
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000

[David Ahern]

On 3/27/17 6:42 AM, Dmitry Vyukov wrote:
> A friendly ping. This still happens all the time for us.

Haven't looked at this in a couple of weeks. I have syzkaller installed
on a machine locally and never was able to reproduce this ipv6 problem.
I am using a jessie rootfs; from the syzkaller files I take it you are
using wheezy. Should not matter but as I recall there are differences in
sysctl setttings. Regardless, can you send me the output of 'sysctl
net.ipv6'?

It is spring break week here, and I am taking a couple of days off. With
netdev next week, I realistically won't have time to come back to this
for 2-3 weeks.

[Dmitry Vyukov]

On Mon, Mar 27, 2017 at 3:57 PM, David Ahern <d...@cumulusnetworks.com> wrote:
> On 3/27/17 6:42 AM, Dmitry Vyukov wrote:
>> A friendly ping. This still happens all the time for us.
>
> Haven't looked at this in a couple of weeks. I have syzkaller installed
> on a machine locally and never was able to reproduce this ipv6 problem.
> I am using a jessie rootfs; from the syzkaller files I take it you are
> using wheezy. Should not matter but as I recall there are differences in
> sysctl setttings. Regardless, can you send me the output of 'sysctl
> net.ipv6'?

Hi David,

So you have syzkaller running locally. Great!
Yes, we are using wheezy. I've attached output of sysctl net.ipv6.
We are also using "sandbox": "namespace" parameter in config, which
enables USER_NS-based sandboxing. It can be relevant as it results in
lots of network namespaces being created and destroyed. Also TUN
config can have effect as it make syzkaller create/destroy private
interfaces. Also make sure to enable CONFIG_KASAN as it detects most
of the failure modes, and CONFIG_KCOV which allows syzkaller to use
coverage guidance. I've attached my config.
Also try to bump count and procs parameters in syzkaller config.
"procs" is number of parallel test processes per VM, we usually use 8.
"count" is number of VMs to create, reasonable number depends on
amount of RAM you have. Both should increase fuzzing speed and
increase probability of hitting the crash.
We currently hit 20-40 crashes per day with 40 test VMs.

> It is spring break week here, and I am taking a couple of days off. With
> netdev next week, I realistically won't have time to come back to this
> for 2-3 weeks.

No problem. Just wanted to make sure that it's not completely
forgotten. Thanks for looking into this.

[Andrey Konovalov]

Hi!

I've finally managed to reproduce one of the crashes on commit
4f7d029b9bf009fbee76bb10c0c4351a1870d2f3 (4.11-rc7).

I'm not sure if this bug has the same root cause as the first one
reported in this thread, but it definitely has to do with ipv6
routing.

C reproducer, syzkaller program and my .config are attached.

Thanks!

kasan: CONFIG_KASAN_INLINE enabled
kasan: GPF could be caused by NULL-ptr deref or user memory access
general protection fault: 0000 [#1] SMP KASAN
Modules linked in:
CPU: 1 PID: 4035 Comm: a.out Not tainted 4.11.0-rc7+ #250

Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011

task: ffff880069809600 task.stack: ffff880062dc8000
RIP: 0010:ip6_rt_cache_alloc+0xa6/0x560 net/ipv6/route.c:975
RSP: 0018:ffff880062dced30 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: ffff8800670561c0 RCX: 0000000000000006
RDX: 0000000000000003 RSI: ffff880062dcfb28 RDI: 0000000000000018
RBP: ffff880062dced68 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880062dcfb28 R14: dffffc0000000000 R15: 0000000000000000
FS: 00007feebe37e7c0(0000) GS:ffff88006cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000205a0fe4 CR3: 000000006b5c9000 CR4: 00000000000006e0
Call Trace:
ip6_pol_route+0x1512/0x1f20 net/ipv6/route.c:1128
ip6_pol_route_output+0x4c/0x60 net/ipv6/route.c:1212
fib6_rule_action+0x261/0x8a0 net/ipv6/fib6_rules.c:100
fib_rules_lookup+0x3be/0xbc0 net/core/fib_rules.c:265
fib6_rule_lookup+0x175/0x360 net/ipv6/fib6_rules.c:44
ip6_route_output_flags+0x260/0x2f0 net/ipv6/route.c:1240
ip6_route_output ./include/net/ip6_route.h:79
ip6_dst_lookup_tail+0xd5e/0x18b0 net/ipv6/ip6_output.c:959
ip6_dst_lookup_flow+0xb1/0x260 net/ipv6/ip6_output.c:1082
rawv6_sendmsg+0x11b2/0x42e0 net/ipv6/raw.c:903
inet_sendmsg+0x164/0x5b0 net/ipv4/af_inet.c:762
sock_sendmsg_nosec net/socket.c:633
sock_sendmsg+0xca/0x110 net/socket.c:643
SYSC_sendto+0x660/0x810 net/socket.c:1696
SyS_sendto+0x40/0x50 net/socket.c:1664
entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
RIP: 0033:0x7feebda90b79
RSP: 002b:000000000072fee8 EFLAGS: 00000206 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00007ffe1f920180 RCX: 00007feebda90b79
RDX: 0000000000000000 RSI: 0000000020fd0fd0 RDI: 0000000000000004
RBP: 0000000000400f30 R08: 00000000205a0fe4 R09: 000000000000001c
R10: 0000000000000800 R11: 0000000000000206 R12: 0000000000000000
R13: 00007ffe1f920180 R14: 0000000000000000 R15: 0000000000000000
Code: 03 80 3c 02 00 0f 85 37 04 00 00 4d 8b 64 24 40 e8 90 dd 82 fd
49 8d 7c 24 18 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80>
3c 02 00 0f 85 61 04 00 00 49 8b 74 24 18 48 b8 00 00 00 00
RIP: ip6_rt_cache_alloc+0xa6/0x560 RSP: ffff880062dced30
---[ end trace 9f58077ffa8cf9c0 ]---

[David Ahern]

On 4/18/17 2:43 PM, Andrey Konovalov wrote:
> I've finally managed to reproduce one of the crashes on commit
> 4f7d029b9bf009fbee76bb10c0c4351a1870d2f3 (4.11-rc7).
>
> I'm not sure if this bug has the same root cause as the first one
> reported in this thread, but it definitely has to do with ipv6
> routing.
>
> C reproducer, syzkaller program and my .config are attached.
>
> Thanks!
>
> kasan: CONFIG_KASAN_INLINE enabled
> kasan: GPF could be caused by NULL-ptr deref or user memory access
> general protection fault: 0000 [#1] SMP KASAN
> Modules linked in:
> CPU: 1 PID: 4035 Comm: a.out Not tainted 4.11.0-rc7+ #250
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> task: ffff880069809600 task.stack: ffff880062dc8000
> RIP: 0010:ip6_rt_cache_alloc+0xa6/0x560 net/ipv6/route.c:975

From a quick glance seems to be a different bug than Dmitry's.

[Andrey Konovalov]

On Wed, Apr 19, 2017 at 1:20 AM, David Ahern <d...@cumulusnetworks.com> wrote:
> On 4/18/17 2:43 PM, Andrey Konovalov wrote:
>> I've finally managed to reproduce one of the crashes on commit
>> 4f7d029b9bf009fbee76bb10c0c4351a1870d2f3 (4.11-rc7).
>>
>> I'm not sure if this bug has the same root cause as the first one
>> reported in this thread, but it definitely has to do with ipv6
>> routing.
>>
>> C reproducer, syzkaller program and my .config are attached.

Just FYI, the reproducer uses interface number 9 inside a user
namespace, which is apparently ip6gre0.

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT qlen 1000
link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN
mode DEFAULT qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: ip_vti0@NONE: <NOARP> mtu 1332 qdisc noop state DOWN mode DEFAULT qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
6: ip6_vti0@NONE: <NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
link/tunnel6 :: brd ::
7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
8: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT qlen 1000
link/tunnel6 :: brd ::
9: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN mode DEFAULT qlen 1000
link/[823] 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00

>>
>> Thanks!
>>
>> kasan: CONFIG_KASAN_INLINE enabled
>> kasan: GPF could be caused by NULL-ptr deref or user memory access
>> general protection fault: 0000 [#1] SMP KASAN
>> Modules linked in:
>> CPU: 1 PID: 4035 Comm: a.out Not tainted 4.11.0-rc7+ #250
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> task: ffff880069809600 task.stack: ffff880062dc8000
>> RIP: 0010:ip6_rt_cache_alloc+0xa6/0x560 net/ipv6/route.c:975
>
> From a quick glance seems to be a different bug than Dmitry's.

It might be.

>

[David Ahern]

On 4/18/17 2:43 PM, Andrey Konovalov wrote:

> Hi!
>
> I've finally managed to reproduce one of the crashes on commit
> 4f7d029b9bf009fbee76bb10c0c4351a1870d2f3 (4.11-rc7).
>
> I'm not sure if this bug has the same root cause as the first one
> reported in this thread, but it definitely has to do with ipv6
> routing.
>
> C reproducer, syzkaller program and my .config are attached.

built a kernel with that config. booted the vm. ran the program. nada.

strace is showing:

clone(child_stack=0x72ffb0,
flags=CLONE_NEWUTS|CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNET) = -1 EINVAL
(Invalid argument)

[Andrey Konovalov]

That's weird. I usually see this when I have CONFIG_USER_NS disabled.

Anyway, I just finished simplifying the reproducer. Give this one a try.

[David Ahern]

On 4/19/17 10:12 AM, Andrey Konovalov wrote:
> That's weird. I usually see this when I have CONFIG_USER_NS disabled.

I bungled the movement of .config between servers. reproduced. will
investigate.