INFO: rcu detected stall in tcp_keepalive_timer
9 views
Skip to first unread message
syzbot
Jan 5, 2020, 11:06:09 AM1/5/20
to syzkaller...@googlegroups.com
Hello,
syzbot found the following crash on:
HEAD commit: 84f5ad46 Linux 4.14.162
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12e779c1e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=67bcc84091a71c98
dashboard link: https://syzkaller.appspot.com/bug?extid=61263beb6e7b7bdab947
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+61263b...@syzkaller.appspotmail.com
INFO: rcu_sched detected stalls on CPUs/tasks:
(detected by 1, t=10502 jiffies, g=18896, c=18895, q=4)
All QSes seen, last rcu_sched kthread activity 9588
(4295002437-4294992849), jiffies_till_next_fqs=1, root ->qsmask 0x0
syz-executor.5 R running task 28144 3310 3253 0x8000000c
Call Trace:
<IRQ>
sched_show_task kernel/sched/core.c:5168 [inline]
sched_show_task.cold+0x2f0/0x351 kernel/sched/core.c:5143
print_other_cpu_stall kernel/rcu/tree.c:1501 [inline]
check_cpu_stall kernel/rcu/tree.c:1616 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold+0xcb1/0xd0a kernel/rcu/tree.c:2792
update_process_times+0x31/0x70 kernel/time/timer.c:1590
tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
__run_hrtimer kernel/time/hrtimer.c:1223 [inline]
__hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1287
hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1321
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
RIP: 0010:__save_stack_trace+0x70/0xd0 arch/x86/kernel/stacktrace.c:43
RSP: 0018:ffff8880aed077f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000001 RBX: ffff8880aed07890 RCX: 1ffff11015da0f06
RDX: dffffc0000000000 RSI: 1ffff11015da0f00 RDI: ffff8880aed07838
RBP: ffff8880aed07870 R08: 0000000000000001 R09: ffff8880aed07848
R10: ffff8880aed07818 R11: ffff8880aed07828 R12: 0000000000000000
R13: 0000000000000000 R14: ffff888092c74000 R15: ffff8880aa800ac0
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x45/0xd0 mm/kasan/kasan.c:447
set_track mm/kasan/kasan.c:459 [inline]
kasan_kmalloc mm/kasan/kasan.c:551 [inline]
kasan_kmalloc+0xce/0xf0 mm/kasan/kasan.c:529
kasan_slab_alloc+0xf/0x20 mm/kasan/kasan.c:489
slab_post_alloc_hook mm/slab.h:442 [inline]
slab_alloc_node mm/slab.c:3333 [inline]
kmem_cache_alloc_node_trace+0x13c/0x770 mm/slab.c:3659
__do_kmalloc_node mm/slab.c:3681 [inline]
__kmalloc_node_track_caller+0x3d/0x80 mm/slab.c:3696
__kmalloc_reserve.isra.0+0x40/0xe0 net/core/skbuff.c:137
__alloc_skb+0xcf/0x500 net/core/skbuff.c:205
alloc_skb include/linux/skbuff.h:980 [inline]
tcp_xmit_probe_skb+0x65/0x3b0 net/ipv4/tcp_output.c:3666
tcp_write_wakeup+0x187/0x5b0 net/ipv4/tcp_output.c:3731
tcp_keepalive_timer+0x67f/0xa80 net/ipv4/tcp_timer.c:691
call_timer_fn+0x161/0x670 kernel/time/timer.c:1279
expire_timers kernel/time/timer.c:1318 [inline]
__run_timers kernel/time/timer.c:1636 [inline]
__run_timers kernel/time/timer.c:1604 [inline]
run_timer_softirq+0x5b7/0x1520 kernel/time/timer.c:1649
__do_softirq+0x244/0x9a0 kernel/softirq.c:288
invoke_softirq kernel/softirq.c:368 [inline]
irq_exit+0x160/0x1b0 kernel/softirq.c:409
exiting_irq arch/x86/include/asm/apic.h:648 [inline]
smp_apic_timer_interrupt+0x146/0x5e0 arch/x86/kernel/apic/apic.c:1102
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:779
[inline]
RIP: 0010:lock_acquire+0x1ea/0x430 kernel/locking/lockdep.c:3997
RSP: 0018:ffff88806d8176d0 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff10
RAX: 1ffffffff0fe2d29 RBX: ffff888092c74000 RCX: 00000000eff106db
RDX: dffffc0000000000 RSI: ffff888092c748a8 RDI: 0000000000000282
RBP: ffff88806d817718 R08: 0000000000000000 R09: ffff888092c748c8
R10: ffff888092c748a8 R11: ffff888092c74000 R12: ffffffff87f88960
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
rcu_lock_acquire include/linux/rcupdate.h:242 [inline]
rcu_read_lock include/linux/rcupdate.h:629 [inline]
lock_page_memcg+0x39/0x220 mm/memcontrol.c:1669
page_remove_file_rmap mm/rmap.c:1211 [inline]
page_remove_rmap+0x19f/0x940 mm/rmap.c:1296
zap_pte_range mm/memory.c:1342 [inline]
zap_pmd_range mm/memory.c:1444 [inline]
zap_pud_range mm/memory.c:1473 [inline]
zap_p4d_range mm/memory.c:1494 [inline]
unmap_page_range+0xabd/0x19f0 mm/memory.c:1515
unmap_single_vma+0x15d/0x2c0 mm/memory.c:1560
unmap_vmas+0xac/0x170 mm/memory.c:1590
exit_mmap+0x285/0x4e0 mm/mmap.c:3062
__mmput kernel/fork.c:930 [inline]
mmput+0x114/0x440 kernel/fork.c:951
exit_mm kernel/exit.c:545 [inline]
do_exit+0x6fc/0x2c80 kernel/exit.c:841
do_group_exit+0x111/0x330 kernel/exit.c:951
get_signal+0x381/0x1cd0 kernel/signal.c:2413
do_signal+0x86/0x19a0 arch/x86/kernel/signal.c:814
exit_to_usermode_loop+0x15c/0x220 arch/x86/entry/common.c:160
prepare_exit_to_usermode+0x1b5/0x220 arch/x86/entry/common.c:199
retint_user+0x8/0x18
RIP: 0033:0x40439a
RSP: 002b:00007f7daa0e9c90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
RAX: 000000000000164a RBX: 0000000000000000 RCX: 000000000007b890
RDX: 000000000000017a RSI: 0000000000000000 RDI: 0000000000000001
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000004c1b1d R14: 00000000004d6ab8 R15: 00000000ffffffff
rcu_sched kthread starved for 9588 jiffies! g18896 c18895 f0x2
RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
rcu_sched R running task 29776 9 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2808 [inline]
__schedule+0x7b8/0x1cd0 kernel/sched/core.c:3384
schedule+0x92/0x1c0 kernel/sched/core.c:3428
schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1746
rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot found the following crash on:
HEAD commit: 84f5ad46 Linux 4.14.162
git tree: linux-4.14.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12e779c1e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=67bcc84091a71c98
dashboard link: https://syzkaller.appspot.com/bug?extid=61263beb6e7b7bdab947
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+61263b...@syzkaller.appspotmail.com
INFO: rcu_sched detected stalls on CPUs/tasks:
(detected by 1, t=10502 jiffies, g=18896, c=18895, q=4)
All QSes seen, last rcu_sched kthread activity 9588
(4295002437-4294992849), jiffies_till_next_fqs=1, root ->qsmask 0x0
syz-executor.5 R running task 28144 3310 3253 0x8000000c
Call Trace:
<IRQ>
sched_show_task kernel/sched/core.c:5168 [inline]
sched_show_task.cold+0x2f0/0x351 kernel/sched/core.c:5143
print_other_cpu_stall kernel/rcu/tree.c:1501 [inline]
check_cpu_stall kernel/rcu/tree.c:1616 [inline]
__rcu_pending kernel/rcu/tree.c:3390 [inline]
rcu_pending kernel/rcu/tree.c:3452 [inline]
rcu_check_callbacks.cold+0xcb1/0xd0a kernel/rcu/tree.c:2792
update_process_times+0x31/0x70 kernel/time/timer.c:1590
tick_sched_handle+0x85/0x160 kernel/time/tick-sched.c:161
tick_sched_timer+0x43/0x130 kernel/time/tick-sched.c:1219
__run_hrtimer kernel/time/hrtimer.c:1223 [inline]
__hrtimer_run_queues+0x270/0xbc0 kernel/time/hrtimer.c:1287
hrtimer_interrupt+0x1d8/0x5d0 kernel/time/hrtimer.c:1321
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1075 [inline]
smp_apic_timer_interrupt+0x11c/0x5e0 arch/x86/kernel/apic/apic.c:1100
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
RIP: 0010:__save_stack_trace+0x70/0xd0 arch/x86/kernel/stacktrace.c:43
RSP: 0018:ffff8880aed077f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
RAX: 0000000000000001 RBX: ffff8880aed07890 RCX: 1ffff11015da0f06
RDX: dffffc0000000000 RSI: 1ffff11015da0f00 RDI: ffff8880aed07838
RBP: ffff8880aed07870 R08: 0000000000000001 R09: ffff8880aed07848
R10: ffff8880aed07818 R11: ffff8880aed07828 R12: 0000000000000000
R13: 0000000000000000 R14: ffff888092c74000 R15: ffff8880aa800ac0
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
save_stack+0x45/0xd0 mm/kasan/kasan.c:447
set_track mm/kasan/kasan.c:459 [inline]
kasan_kmalloc mm/kasan/kasan.c:551 [inline]
kasan_kmalloc+0xce/0xf0 mm/kasan/kasan.c:529
kasan_slab_alloc+0xf/0x20 mm/kasan/kasan.c:489
slab_post_alloc_hook mm/slab.h:442 [inline]
slab_alloc_node mm/slab.c:3333 [inline]
kmem_cache_alloc_node_trace+0x13c/0x770 mm/slab.c:3659
__do_kmalloc_node mm/slab.c:3681 [inline]
__kmalloc_node_track_caller+0x3d/0x80 mm/slab.c:3696
__kmalloc_reserve.isra.0+0x40/0xe0 net/core/skbuff.c:137
__alloc_skb+0xcf/0x500 net/core/skbuff.c:205
alloc_skb include/linux/skbuff.h:980 [inline]
tcp_xmit_probe_skb+0x65/0x3b0 net/ipv4/tcp_output.c:3666
tcp_write_wakeup+0x187/0x5b0 net/ipv4/tcp_output.c:3731
tcp_keepalive_timer+0x67f/0xa80 net/ipv4/tcp_timer.c:691
call_timer_fn+0x161/0x670 kernel/time/timer.c:1279
expire_timers kernel/time/timer.c:1318 [inline]
__run_timers kernel/time/timer.c:1636 [inline]
__run_timers kernel/time/timer.c:1604 [inline]
run_timer_softirq+0x5b7/0x1520 kernel/time/timer.c:1649
__do_softirq+0x244/0x9a0 kernel/softirq.c:288
invoke_softirq kernel/softirq.c:368 [inline]
irq_exit+0x160/0x1b0 kernel/softirq.c:409
exiting_irq arch/x86/include/asm/apic.h:648 [inline]
smp_apic_timer_interrupt+0x146/0x5e0 arch/x86/kernel/apic/apic.c:1102
apic_timer_interrupt+0x96/0xa0 arch/x86/entry/entry_64.S:792
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:779
[inline]
RIP: 0010:lock_acquire+0x1ea/0x430 kernel/locking/lockdep.c:3997
RSP: 0018:ffff88806d8176d0 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff10
RAX: 1ffffffff0fe2d29 RBX: ffff888092c74000 RCX: 00000000eff106db
RDX: dffffc0000000000 RSI: ffff888092c748a8 RDI: 0000000000000282
RBP: ffff88806d817718 R08: 0000000000000000 R09: ffff888092c748c8
R10: ffff888092c748a8 R11: ffff888092c74000 R12: ffffffff87f88960
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
rcu_lock_acquire include/linux/rcupdate.h:242 [inline]
rcu_read_lock include/linux/rcupdate.h:629 [inline]
lock_page_memcg+0x39/0x220 mm/memcontrol.c:1669
page_remove_file_rmap mm/rmap.c:1211 [inline]
page_remove_rmap+0x19f/0x940 mm/rmap.c:1296
zap_pte_range mm/memory.c:1342 [inline]
zap_pmd_range mm/memory.c:1444 [inline]
zap_pud_range mm/memory.c:1473 [inline]
zap_p4d_range mm/memory.c:1494 [inline]
unmap_page_range+0xabd/0x19f0 mm/memory.c:1515
unmap_single_vma+0x15d/0x2c0 mm/memory.c:1560
unmap_vmas+0xac/0x170 mm/memory.c:1590
exit_mmap+0x285/0x4e0 mm/mmap.c:3062
__mmput kernel/fork.c:930 [inline]
mmput+0x114/0x440 kernel/fork.c:951
exit_mm kernel/exit.c:545 [inline]
do_exit+0x6fc/0x2c80 kernel/exit.c:841
do_group_exit+0x111/0x330 kernel/exit.c:951
get_signal+0x381/0x1cd0 kernel/signal.c:2413
do_signal+0x86/0x19a0 arch/x86/kernel/signal.c:814
exit_to_usermode_loop+0x15c/0x220 arch/x86/entry/common.c:160
prepare_exit_to_usermode+0x1b5/0x220 arch/x86/entry/common.c:199
retint_user+0x8/0x18
RIP: 0033:0x40439a
RSP: 002b:00007f7daa0e9c90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
RAX: 000000000000164a RBX: 0000000000000000 RCX: 000000000007b890
RDX: 000000000000017a RSI: 0000000000000000 RDI: 0000000000000001
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00000000004c1b1d R14: 00000000004d6ab8 R15: 00000000ffffffff
rcu_sched kthread starved for 9588 jiffies! g18896 c18895 f0x2
RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
rcu_sched R running task 29776 9 2 0x80000000
Call Trace:
context_switch kernel/sched/core.c:2808 [inline]
__schedule+0x7b8/0x1cd0 kernel/sched/core.c:3384
schedule+0x92/0x1c0 kernel/sched/core.c:3428
schedule_timeout+0x43e/0xe10 kernel/time/timer.c:1746
rcu_gp_kthread+0xbf4/0x1ec0 kernel/rcu/tree.c:2255
kthread+0x319/0x430 kernel/kthread.c:232
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot
May 4, 2020, 12:06:12 PM5/4/20
to syzkaller...@googlegroups.com
Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.
Crashes did not happen for a while, no reproducer and no activity.
Reply all
Reply to author
Forward
0 new messages