INFO: rcu detected stall in br_handle_frame

32 views
Skip to first unread message

syzbot

unread,
Sep 6, 2019, 10:33:07 AM9/6/19
to syzkaller...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 0fed55c2 Linux 4.19.70
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=16421efa600000
kernel config: https://syzkaller.appspot.com/x/.config?x=2964bf2f89c3f203
dashboard link: https://syzkaller.appspot.com/bug?extid=2f010f92f469db8b4a9e
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+2f010f...@syzkaller.appspotmail.com

audit: type=1800 audit(1567777228.313:278): pid=26132 uid=0 auid=4294967295
ses=4294967295 subj=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
op=collect_data cause=failed comm="syz-executor.3"
name="hugetlb.2MB.e_in_bytes" dev="sda1" ino=17442 res=0
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...!: (10499 ticks this GP) idle=7a2/1/0x4000000000000002
softirq=30174/30174 fqs=759
rcu: (t=10501 jiffies g=28025 q=22809)
rcu: rcu_preempt kthread starved for 8966 jiffies! g28025 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: RCU grace-period kthread stack dump:
rcu_preempt I29104 10 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2826 [inline]
__schedule+0x866/0x1dc0 kernel/sched/core.c:3474
schedule+0x92/0x1c0 kernel/sched/core.c:3518
schedule_timeout+0x4db/0xfc0 kernel/time/timer.c:1804
rcu_gp_kthread+0xd5c/0x2190 kernel/rcu/tree.c:2202
kthread+0x354/0x420 kernel/kthread.c:246
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
NMI backtrace for cpu 0
CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.70 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
nmi_cpu_backtrace.cold+0x63/0xa4 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x1b0/0x1f8 lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x189/0x1d5 kernel/rcu/tree.c:1340
print_cpu_stall kernel/rcu/tree.c:1478 [inline]
check_cpu_stall kernel/rcu/tree.c:1550 [inline]
__rcu_pending kernel/rcu/tree.c:3293 [inline]
rcu_pending kernel/rcu/tree.c:3336 [inline]
rcu_check_callbacks.cold+0x5e3/0xd90 kernel/rcu/tree.c:2682
update_process_times+0x32/0x80 kernel/time/timer.c:1636
tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:164
tick_sched_timer+0x47/0x130 kernel/time/tick-sched.c:1274
__run_hrtimer kernel/time/hrtimer.c:1398 [inline]
__hrtimer_run_queues+0x33b/0xdc0 kernel/time/hrtimer.c:1460
hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1518
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1067 [inline]
smp_apic_timer_interrupt+0x111/0x550 arch/x86/kernel/apic/apic.c:1092
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:893
</IRQ>
RIP: 0010:__read_once_size include/linux/compiler.h:193 [inline]
RIP: 0010:list_empty include/linux/list.h:203 [inline]
RIP: 0010:hhf_dequeue+0x4fe/0xa00 net/sched/sch_hhf.c:428
Code: 80 3c 20 00 0f 85 05 04 00 00 49 89 1f e9 66 fb ff ff e8 d5 dd dc fb
48 8b 45 b8 4c 8b 75 c8 42 80 3c 20 00 0f 85 13 04 00 00 <49> 8b 85 50 03
00 00 48 39 45 c8 0f 85 66 fb ff ff 45 31 ff e9 b0
RSP: 0018:ffff8880aa236998 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 1ffff11011149e22 RBX: ffff888088a4f078 RCX: ffffffff858e8a2e
RDX: 0000000000000100 RSI: ffffffff858e88bb RDI: ffff888088a4f080
RBP: ffff8880aa2369e8 R08: ffff8880aa226240 R09: 0000000000000006
R10: ffff8880aa226bb0 R11: 000000003a55b6e7 R12: dffffc0000000000
R13: ffff888088a4edc0 R14: ffff888088a4f110 R15: ffff888088a4f110
dequeue_skb net/sched/sch_generic.c:277 [inline]
qdisc_restart net/sched/sch_generic.c:380 [inline]
__qdisc_run+0x1e7/0x1960 net/sched/sch_generic.c:398
__dev_xmit_skb net/core/dev.c:3501 [inline]
__dev_queue_xmit+0x165c/0x2fe0 net/core/dev.c:3811
dev_queue_xmit+0x18/0x20 net/core/dev.c:3876
br_dev_queue_push_xmit+0x405/0x5d0 net/bridge/br_forward.c:56
br_nf_dev_queue_xmit+0x365/0x1610 net/bridge/br_netfilter_hooks.c:779
NF_HOOK include/linux/netfilter.h:289 [inline]
NF_HOOK include/linux/netfilter.h:283 [inline]
br_nf_post_routing+0x76f/0x12f0 net/bridge/br_netfilter_hooks.c:825
nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
nf_hook include/linux/netfilter.h:244 [inline]
NF_HOOK include/linux/netfilter.h:287 [inline]
br_forward_finish+0x215/0x400 net/bridge/br_forward.c:69
br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
br_nf_forward_finish+0x356/0x700 net/bridge/br_netfilter_hooks.c:553
NF_HOOK include/linux/netfilter.h:289 [inline]
NF_HOOK include/linux/netfilter.h:283 [inline]
br_nf_forward_ip net/bridge/br_netfilter_hooks.c:620 [inline]
br_nf_forward_ip+0x6b4/0x17c0 net/bridge/br_netfilter_hooks.c:564
nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
nf_hook include/linux/netfilter.h:244 [inline]
NF_HOOK include/linux/netfilter.h:287 [inline]
__br_forward+0x393/0xb30 net/bridge/br_forward.c:113
deliver_clone+0x61/0xc0 net/bridge/br_forward.c:129
br_flood+0x5fa/0x710 net/bridge/br_forward.c:238
br_handle_frame_finish+0xf72/0x1510 net/bridge/br_input.c:167
br_nf_hook_thresh+0x2e9/0x370 net/bridge/br_netfilter_hooks.c:1008
br_nf_pre_routing_finish_ipv6+0x6ed/0xd70
net/bridge/br_netfilter_ipv6.c:210
NF_HOOK include/linux/netfilter.h:289 [inline]
br_nf_pre_routing_ipv6+0x4a5/0x8ca net/bridge/br_netfilter_ipv6.c:240
br_nf_pre_routing+0x1016/0x159c net/bridge/br_netfilter_hooks.c:494
nf_hook_entry_hookfn include/linux/netfilter.h:119 [inline]
nf_hook_slow+0xbc/0x1e0 net/netfilter/core.c:511
nf_hook include/linux/netfilter.h:244 [inline]
NF_HOOK include/linux/netfilter.h:287 [inline]
br_handle_frame+0x985/0x1480 net/bridge/br_input.c:306
__netif_receive_skb_core+0xac1/0x3570 net/core/dev.c:4861
__netif_receive_skb_one_core+0xa8/0x1a0 net/core/dev.c:4930
__netif_receive_skb+0x2c/0x1d0 net/core/dev.c:5044
process_backlog+0x23a/0x7a0 net/core/dev.c:5855
napi_poll net/core/dev.c:6278 [inline]
net_rx_action+0x4f5/0x1070 net/core/dev.c:6344
__do_softirq+0x25c/0x921 kernel/softirq.c:292
run_ksoftirqd kernel/softirq.c:653 [inline]
run_ksoftirqd+0x8e/0x110 kernel/softirq.c:645
smpboot_thread_fn+0x6a3/0xa30 kernel/smpboot.c:164
kthread+0x354/0x420 kernel/kthread.c:246
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,
Sep 7, 2019, 12:05:09 AM9/7/19
to syzkaller...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 414510bc Linux 4.14.142
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=16d43849600000
kernel config: https://syzkaller.appspot.com/x/.config?x=9aa0b2ccd827f416
dashboard link: https://syzkaller.appspot.com/bug?extid=04f79b221ef00491c0c7
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+04f79b...@syzkaller.appspotmail.com

TCP: request_sock_TCPv6: Possible SYN flooding on port 20002. Sending
cookies. Check SNMP counters.
TCP: request_sock_TCPv6: Possible SYN flooding on port 20002. Sending
cookies. Check SNMP counters.
TCP: request_sock_TCPv6: Possible SYN flooding on port 20002. Sending
cookies. Check SNMP counters.
INFO: rcu_preempt self-detected stall on CPU
0-...: (1 GPs behind) idle=c36/140000000000001/0 softirq=200853/200857
fqs=5226
(t=10500 jiffies g=58785 c=58784 q=1659)
NMI backtrace for cpu 0
CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x138/0x197 lib/dump_stack.c:53
nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
print_cpu_stall kernel/rcu/tree.c:1542 [inline]
check_cpu_stall kernel/rcu/tree.c:1610 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
update_process_times+0x31/0x70 kernel/time/timer.c:1588
tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
__run_hrtimer kernel/time/hrtimer.c:1220 [inline]
__hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
</IRQ>
RIP: 0010:__list_add_valid+0x70/0xa0 lib/list_debug.c:35
RSP: 0018:ffff8880a9ceebe8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff10
RAX: dffffc0000000000 RBX: ffff888093e5c6b8 RCX: 0000000000000000
RDX: ffff888093e5c750 RSI: ffff888093e5c750 RDI: ffff888093e5c758
RBP: ffff8880a9ceec00 R08: 0000000000000000 R09: ffff8880a9ce2b50
R10: ffff8880a9ce2b30 R11: ffff8880a9ce21c0 R12: ffff888093e5c750
R13: ffff888093e5c6b8 R14: ffff888093e5c750 R15: ffff888093e5c740
__list_add include/linux/list.h:60 [inline]
list_add_tail include/linux/list.h:93 [inline]
list_move_tail include/linux/list.h:183 [inline]
hhf_dequeue+0x61b/0xa60 net/sched/sch_hhf.c:438
dequeue_skb net/sched/sch_generic.c:148 [inline]
qdisc_restart net/sched/sch_generic.c:241 [inline]
__qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
__dev_xmit_skb net/core/dev.c:3235 [inline]
__dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
br_dev_queue_push_xmit+0x367/0x530 net/bridge/br_forward.c:55
br_nf_dev_queue_xmit+0x307/0x1440 net/bridge/br_netfilter_hooks.c:776
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_post_routing+0xb80/0xf00 net/bridge/br_netfilter_hooks.c:822
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_forward_finish+0x1b7/0x320 net/bridge/br_forward.c:67
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_forward_finish+0x264/0x640 net/bridge/br_netfilter_hooks.c:550
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_forward_ip net/bridge/br_netfilter_hooks.c:617 [inline]
br_nf_forward_ip+0x5fc/0x11d0 net/bridge/br_netfilter_hooks.c:561
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
__br_forward+0x312/0x9c0 net/bridge/br_forward.c:111
deliver_clone+0x61/0xc0 net/bridge/br_forward.c:127
br_flood+0x43c/0x530 net/bridge/br_forward.c:222
br_handle_frame_finish+0xaf0/0x1830 net/bridge/br_input.c:210
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_pre_routing_finish_ipv6+0x621/0xc50
net/bridge/br_netfilter_ipv6.c:210
NF_HOOK include/linux/netfilter.h:250 [inline]
br_nf_pre_routing_ipv6+0x417/0x790 net/bridge/br_netfilter_ipv6.c:240
br_nf_pre_routing+0xdce/0x12c7 net/bridge/br_netfilter_hooks.c:491
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_handle_frame+0x80c/0x1110 net/bridge/br_input.c:348
__netif_receive_skb_core+0x789/0x2ca0 net/core/dev.c:4431
__netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4515
process_backlog+0x21f/0x730 net/core/dev.c:5197
napi_poll net/core/dev.c:5598 [inline]
net_rx_action+0x490/0xf80 net/core/dev.c:5664
__do_softirq+0x244/0x9a0 kernel/softirq.c:288
run_ksoftirqd kernel/softirq.c:670 [inline]
run_ksoftirqd+0x8c/0x1b0 kernel/softirq.c:662
smpboot_thread_fn+0x5f4/0x960 kernel/smpboot.c:164
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
INFO: rcu_sched detected stalls on CPUs/tasks:
0-...: (1 GPs behind) idle=c36/140000000000000/0 softirq=200771/200857
fqs=5210
(detected by 1, t=10529 jiffies, g=40490, c=40489, q=43)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
task: ffff8880a9ce21c0 task.stack: ffff8880a9ce8000
RIP: 0010:__list_add_valid+0x0/0xa0 lib/list_debug.c:23
RSP: 0018:ffff8880a9ceec08 EFLAGS: 00000246
RAX: 1ffff110127cb8eb RBX: ffff888093e5c6b8 RCX: 0000000000000000
RDX: ffff888093e5c750 RSI: ffff888093e5c750 RDI: ffff888093e5c6b8
RBP: ffff8880a9ceec60 R08: 0000000000000000 R09: ffff8880a9ce2b50
R10: ffff8880a9ce2b30 R11: ffff8880a9ce21c0 R12: dffffc0000000000
R13: ffff888093e5c4c0 R14: ffff888093e5c750 R15: ffff888093e5c740
FS: 0000000000000000(0000) GS:ffff8880aee00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b31f33000 CR3: 000000000766a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
dequeue_skb net/sched/sch_generic.c:148 [inline]
qdisc_restart net/sched/sch_generic.c:241 [inline]
__qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
__dev_xmit_skb net/core/dev.c:3235 [inline]
__dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
br_dev_queue_push_xmit+0x367/0x530 net/bridge/br_forward.c:55
br_nf_dev_queue_xmit+0x307/0x1440 net/bridge/br_netfilter_hooks.c:776
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_post_routing+0xb80/0xf00 net/bridge/br_netfilter_hooks.c:822
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_forward_finish+0x1b7/0x320 net/bridge/br_forward.c:67
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_forward_finish+0x264/0x640 net/bridge/br_netfilter_hooks.c:550
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_forward_ip net/bridge/br_netfilter_hooks.c:617 [inline]
br_nf_forward_ip+0x5fc/0x11d0 net/bridge/br_netfilter_hooks.c:561
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
__br_forward+0x312/0x9c0 net/bridge/br_forward.c:111
deliver_clone+0x61/0xc0 net/bridge/br_forward.c:127
br_flood+0x43c/0x530 net/bridge/br_forward.c:222
br_handle_frame_finish+0xaf0/0x1830 net/bridge/br_input.c:210
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_pre_routing_finish_ipv6+0x621/0xc50
net/bridge/br_netfilter_ipv6.c:210
NF_HOOK include/linux/netfilter.h:250 [inline]
br_nf_pre_routing_ipv6+0x417/0x790 net/bridge/br_netfilter_ipv6.c:240
br_nf_pre_routing+0xdce/0x12c7 net/bridge/br_netfilter_hooks.c:491
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_handle_frame+0x80c/0x1110 net/bridge/br_input.c:348
__netif_receive_skb_core+0x789/0x2ca0 net/core/dev.c:4431
__netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4515
process_backlog+0x21f/0x730 net/core/dev.c:5197
napi_poll net/core/dev.c:5598 [inline]
net_rx_action+0x490/0xf80 net/core/dev.c:5664
__do_softirq+0x244/0x9a0 kernel/softirq.c:288
run_ksoftirqd kernel/softirq.c:670 [inline]
run_ksoftirqd+0x8c/0x1b0 kernel/softirq.c:662
smpboot_thread_fn+0x5f4/0x960 kernel/smpboot.c:164
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: 54 49 89 fc e8 52 76 84 fe 4c 89 e6 48 c7 c7 c0 2e 9d 86 e8 73 fd ff
ff 41 5c 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <48> b8 00 00
00 00 00 fc ff df 55 48 89 e5 41 55 49 89 fd 48 8d
INFO: rcu_bh detected stalls on CPUs/tasks:
0-...: (1 GPs behind) idle=c36/140000000000000/0 softirq=198030/200857
fqs=5226
(detected by 1, t=10620 jiffies, g=161, c=160, q=40)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
task: ffff8880a9ce21c0 task.stack: ffff8880a9ce8000
RIP: 0010:hhf_dequeue+0x51a/0xa60 net/sched/sch_hhf.c:435
RSP: 0018:ffff8880a9ceec10 EFLAGS: 00000a07
RAX: 0000000000000000 RBX: ffff888093e5c6b8 RCX: 0000000000000000
RDX: 1ffff110127cb8ed RSI: ffff888093e5c750 RDI: ffff888093e5c76c
RBP: ffff8880a9ceec60 R08: 0000000000000000 R09: ffff8880a9ce2b50
R10: ffff8880a9ce2b30 R11: ffff8880a9ce21c0 R12: dffffc0000000000
R13: ffff888093e5c4c0 R14: 0000000000000001 R15: ffff888093e5c740
FS: 0000000000000000(0000) GS:ffff8880aee00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b31f33000 CR3: 000000000766a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
dequeue_skb net/sched/sch_generic.c:148 [inline]
qdisc_restart net/sched/sch_generic.c:241 [inline]
__qdisc_run+0x2b8/0xe00 net/sched/sch_generic.c:257
__dev_xmit_skb net/core/dev.c:3235 [inline]
__dev_queue_xmit+0x1571/0x25e0 net/core/dev.c:3493
dev_queue_xmit+0x18/0x20 net/core/dev.c:3558
br_dev_queue_push_xmit+0x367/0x530 net/bridge/br_forward.c:55
br_nf_dev_queue_xmit+0x307/0x1440 net/bridge/br_netfilter_hooks.c:776
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_post_routing+0xb80/0xf00 net/bridge/br_netfilter_hooks.c:822
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_forward_finish+0x1b7/0x320 net/bridge/br_forward.c:67
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_forward_finish+0x264/0x640 net/bridge/br_netfilter_hooks.c:550
NF_HOOK include/linux/netfilter.h:250 [inline]
NF_HOOK include/linux/netfilter.h:244 [inline]
br_nf_forward_ip net/bridge/br_netfilter_hooks.c:617 [inline]
br_nf_forward_ip+0x5fc/0x11d0 net/bridge/br_netfilter_hooks.c:561
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
__br_forward+0x312/0x9c0 net/bridge/br_forward.c:111
deliver_clone+0x61/0xc0 net/bridge/br_forward.c:127
br_flood+0x43c/0x530 net/bridge/br_forward.c:222
br_handle_frame_finish+0xaf0/0x1830 net/bridge/br_input.c:210
br_nf_hook_thresh+0x25b/0x2e0 net/bridge/br_netfilter_hooks.c:1005
br_nf_pre_routing_finish_ipv6+0x621/0xc50
net/bridge/br_netfilter_ipv6.c:210
NF_HOOK include/linux/netfilter.h:250 [inline]
br_nf_pre_routing_ipv6+0x417/0x790 net/bridge/br_netfilter_ipv6.c:240
br_nf_pre_routing+0xdce/0x12c7 net/bridge/br_netfilter_hooks.c:491
nf_hook_entry_hookfn include/linux/netfilter.h:108 [inline]
nf_hook_slow+0xaf/0x1b0 net/netfilter/core.c:467
nf_hook include/linux/netfilter.h:205 [inline]
NF_HOOK include/linux/netfilter.h:248 [inline]
br_handle_frame+0x80c/0x1110 net/bridge/br_input.c:348
__netif_receive_skb_core+0x789/0x2ca0 net/core/dev.c:4431
__netif_receive_skb+0x2c/0x1b0 net/core/dev.c:4515
process_backlog+0x21f/0x730 net/core/dev.c:5197
napi_poll net/core/dev.c:5598 [inline]
net_rx_action+0x490/0xf80 net/core/dev.c:5664
__do_softirq+0x244/0x9a0 kernel/softirq.c:288
run_ksoftirqd kernel/softirq.c:670 [inline]
run_ksoftirqd+0x8c/0x1b0 kernel/softirq.c:662
smpboot_thread_fn+0x5f4/0x960 kernel/smpboot.c:164
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: c0 01 00 00 8b 45 b8 48 8d 53 f0 48 39 d1 74 37 e8 3c 51 6d fc 49 8d
bd ac 02 00 00 8b 45 b8 48 89 fa 48 c1 ea 03 42 0f b6 0c 22 <48> 89 fa 83
e2 07 83 c2 03 38 ca 7c 08 84 c9 0f 85 d1 03 00 00

syzbot

unread,
Sep 7, 2019, 3:23:08 AM9/7/19
to syzkaller...@googlegroups.com
syzbot has found a reproducer for the following crash on:

HEAD commit: e7d2672c Linux 4.19.71
git tree: linux-4.19.y
console output: https://syzkaller.appspot.com/x/log.txt?x=10c87dc6600000
kernel config: https://syzkaller.appspot.com/x/.config?x=47881f21d9108add
dashboard link: https://syzkaller.appspot.com/bug?extid=2f010f92f469db8b4a9e
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=141080e6600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=174ff1e1600000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+2f010f...@syzkaller.appspotmail.com

IPv6: ADDRCONF(NETDEV_UP): hsr0: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
IPv6: ADDRCONF(NETDEV_UP): vxcan1: link is not ready
8021q: adding VLAN 0 to HW filter on device batadv0
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 1-...!: (10499 ticks this GP) idle=206/1/0x4000000000000002
softirq=12589/12589 fqs=8
rcu: (t=10501 jiffies g=5557 q=94)
rcu: rcu_preempt kthread starved for 10486 jiffies! g5557 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: RCU grace-period kthread stack dump:
rcu_preempt I29104 10 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2826 [inline]
__schedule+0x866/0x1dc0 kernel/sched/core.c:3474
schedule+0x92/0x1c0 kernel/sched/core.c:3518
schedule_timeout+0x4db/0xfc0 kernel/time/timer.c:1804
rcu_gp_kthread+0xd5c/0x2190 kernel/rcu/tree.c:2202
kthread+0x354/0x420 kernel/kthread.c:246
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
NMI backtrace for cpu 1
CPU: 1 PID: 18 Comm: ksoftirqd/1 Not tainted 4.19.71 #0
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0x8/0x50 kernel/kcov.c:100
Code: 66 ea ff 48 c7 05 44 12 cd 08 00 00 00 00 e9 ca e9 ff ff 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 48 8b 75 08 <65> 48 8b 04 25
40 ee 01 00 65 8b 15 28 87 96 7e 81 e2 00 01 1f 00
RSP: 0018:ffff8880aa2a6988 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000007 RBX: ffff888098d654f8 RCX: ffffffff858e8489
RDX: 0000000000000000 RSI: ffffffff858e893b RDI: ffff888098d655ac
RBP: ffff8880aa2a6988 R08: ffff8880aa2944c0 R09: 0000000000000006
R10: ffff8880aa294e30 R11: 00000000c638f6fe R12: dffffc0000000000
R13: ffff888098d65240 R14: 0000000000000000 R15: 0000000000000000
hhf_dequeue+0x56b/0xa00 net/sched/sch_hhf.c:437

syzbot

unread,
Sep 7, 2019, 10:00:07 AM9/7/19
to syzkaller...@googlegroups.com
syzbot has found a reproducer for the following crash on:

HEAD commit: 414510bc Linux 4.14.142
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=11c4d171600000
kernel config: https://syzkaller.appspot.com/x/.config?x=9aa0b2ccd827f416
dashboard link: https://syzkaller.appspot.com/bug?extid=04f79b221ef00491c0c7
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=133a2ba9600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12a5b85e600000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+04f79b...@syzkaller.appspotmail.com

IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
IPv6: ADDRCONF(NETDEV_CHANGE): team0: link becomes ready
IPv6: ADDRCONF(NETDEV_UP): vxcan1: link is not ready
8021q: adding VLAN 0 to HW filter on device batadv0
INFO: rcu_preempt self-detected stall on CPU
0-...: (1 GPs behind) idle=e12/140000000000001/0 softirq=18854/18855 fqs=12
(t=10500 jiffies g=1052 c=1051 q=166)
rcu_preempt kthread starved for 10476 jiffies! g1052 c1051 f0x0
RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
rcu_preempt R running task 29824 8 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2807 [inline]
__schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
schedule+0x92/0x1c0 kernel/sched/core.c:3427
schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
NMI backtrace for cpu 0
CPU: 0 PID: 2826 Comm: kworker/0:2 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x138/0x197 lib/dump_stack.c:53
nmi_cpu_backtrace.cold+0x57/0x94 lib/nmi_backtrace.c:101
nmi_trigger_cpumask_backtrace+0x141/0x189 lib/nmi_backtrace.c:62
arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
trigger_single_cpu_backtrace include/linux/nmi.h:158 [inline]
rcu_dump_cpu_stacks+0x186/0x1d2 kernel/rcu/tree.c:1396
print_cpu_stall kernel/rcu/tree.c:1542 [inline]
check_cpu_stall kernel/rcu/tree.c:1610 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold+0x43d/0xd0a kernel/rcu/tree.c:2792
update_process_times+0x31/0x70 kernel/time/timer.c:1588
tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
__run_hrtimer kernel/time/hrtimer.c:1220 [inline]
__hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1284
hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1318
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:23 [inline]
RIP: 0010:__sanitizer_cov_trace_pc+0xe/0x60 kernel/kcov.c:65
RSP: 0018:ffff8880aee06db8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff10
RAX: ffff88809de181c0 RBX: ffff88809ecc7438 RCX: 0000000000000000
RDX: 0000000000000007 RSI: ffff88809ecc74d0 RDI: ffff88809ecc74ec
RBP: ffff8880aee06e10 R08: 0000000000000000 R09: ffff88809de18bf0
R10: ffff88809de18bd0 R11: ffff88809de181c0 R12: dffffc0000000000
R13: ffff88809ecc7240 R14: 0000000000000002 R15: ffff88809ecc74c0
do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1015
</IRQ>
do_softirq.part.0+0x10e/0x160 kernel/softirq.c:332
do_softirq kernel/softirq.c:324 [inline]
__local_bh_enable_ip+0x154/0x1a0 kernel/softirq.c:185
local_bh_enable include/linux/bottom_half.h:32 [inline]
rcu_read_unlock_bh include/linux/rcupdate.h:725 [inline]
ip6_finish_output2+0x9f3/0x21b0 net/ipv6/ip6_output.c:121
ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
NF_HOOK_COND include/linux/netfilter.h:239 [inline]
ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
dst_output include/net/dst.h:462 [inline]
NF_HOOK include/linux/netfilter.h:250 [inline]
ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
process_one_work+0x863/0x1600 kernel/workqueue.c:2114
worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
INFO: rcu_sched detected stalls on CPUs/tasks:
0-...: (10502 ticks this GP) idle=e12/140000000000000/0
softirq=18855/18855 fqs=0
(detected by 1, t=10579 jiffies, g=775, c=774, q=9)
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 2826 Comm: kworker/0:2 Not tainted 4.14.142 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: ipv6_addrconf addrconf_dad_work
task: ffff88809de181c0 task.stack: ffff88809dd60000
RIP: 0010:__read_once_size include/linux/compiler.h:183 [inline]
RIP: 0010:list_empty include/linux/list.h:203 [inline]
RIP: 0010:hhf_dequeue+0x57/0xa60 net/sched/sch_hhf.c:426
RSP: 0018:ffff8880aee06dc0 EFLAGS: 00000246
RAX: ffffed1013d98e98 RBX: ffff88809ecc7438 RCX: 0000000000000000
RDX: 0000000000000100 RSI: ffff88809ecc74d0 RDI: ffff88809ecc7440
RBP: ffff8880aee06e10 R08: 0000000000000000 R09: ffff88809de18bf0
R10: ffff88809de18bd0 R11: ffff88809de181c0 R12: dffffc0000000000
R13: ffff88809ecc7240 R14: ffff88809ecc74d0 R15: ffff88809ecc74c0
FS: 0000000000000000(0000) GS:ffff8880aee00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000600 CR3: 000000000766a000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1015
</IRQ>
do_softirq.part.0+0x10e/0x160 kernel/softirq.c:332
do_softirq kernel/softirq.c:324 [inline]
__local_bh_enable_ip+0x154/0x1a0 kernel/softirq.c:185
local_bh_enable include/linux/bottom_half.h:32 [inline]
rcu_read_unlock_bh include/linux/rcupdate.h:725 [inline]
ip6_finish_output2+0x9f3/0x21b0 net/ipv6/ip6_output.c:121
ip6_finish_output+0x4f4/0xb50 net/ipv6/ip6_output.c:154
NF_HOOK_COND include/linux/netfilter.h:239 [inline]
ip6_output+0x20f/0x6d0 net/ipv6/ip6_output.c:171
dst_output include/net/dst.h:462 [inline]
NF_HOOK include/linux/netfilter.h:250 [inline]
ndisc_send_skb+0xb56/0x11e0 net/ipv6/ndisc.c:483
ndisc_send_ns+0x360/0x7e0 net/ipv6/ndisc.c:625
addrconf_dad_work+0xa40/0xff0 net/ipv6/addrconf.c:3996
process_one_work+0x863/0x1600 kernel/workqueue.c:2114
worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
Code: f8 48 c1 e8 03 4c 01 e0 48 89 45 c8 49 8d 85 90 02 00 00 48 89 45 d0
48 c1 e8 03 48 89 45 c0 e8 f0 55 6d fc 48 8b 45 c8 80 38 00 <0f> 85 53 07
00 00 49 8b 85 80 02 00 00 4d 89 fe 49 39 c7 0f 84
rcu_sched kthread starved for 10580 jiffies! g775 c774 f0x0
RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=0
rcu_sched I29824 9 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2807 [inline]
__schedule+0x7b8/0x1cd0 kernel/sched/core.c:3383
schedule+0x92/0x1c0 kernel/sched/core.c:3427
schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1744
rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255

syzbot

unread,
Dec 6, 2019, 6:58:01 PM12/6/19
to syzkaller...@googlegroups.com
syzbot suspects this bug was fixed by commit:

commit cc243e2427cef2a5dd7367cb0e0b846503350ffe
Author: Cong Wang <xiyou.w...@gmail.com>
Date: Sun Sep 8 20:40:51 2019 +0000

sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12388f36e00000
start commit: 414510bc Linux 4.14.142
git tree: linux-4.14.y
If the result looks correct, please mark the bug fixed by replying with:

#syz fix: sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

syzbot

unread,
Dec 6, 2019, 11:19:02 PM12/6/19
to syzkaller...@googlegroups.com
syzbot suspects this bug was fixed by commit:

commit a9e91767b921ee196ab693fa13a81b278f53ddac
Author: Cong Wang <xiyou.w...@gmail.com>
Date: Sun Sep 8 20:40:51 2019 +0000

sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1352459ce00000
start commit: e7d2672c Linux 4.19.71
git tree: linux-4.19.y
Reply all
Reply to author
Forward
0 new messages