fs: sandboxed process brings host down

[Dmitry Vyukov]

Hello,

While running syzkaller fuzzer I hit the following problem. Supervisor
process sandboxes worker processes that do random activities with
CLONE_NEWUSER | CLONE_NEWNS | CLONE_NEWPID | CLONE_NEWUTS |
CLONE_NEWNET | CLONE_NEWIPC | CLONE_IO, setrlimit, chroot, etc.
Because of that worker process gains ability to bring whole machine
down (does not happen without the sandbox).

Here is extracted reproducer:

#include <sys/mount.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <errno.h>
#include <sched.h>
#include <linux/sched.h>
#include <syscall.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <memory.h>

int loop(void *arg);
void fail(const char *what);
bool write_file(const char* file, const char* what, ...);

int real_uid, real_gid;

int main()
{
int pid, status;
char stack[64<<10];

real_uid = getuid();
real_gid = getgid();
pid = clone(loop, &stack[sizeof(stack)-8], CLONE_NEWUSER |
CLONE_NEWNS | CLONE_NEWPID, 0, 0);
while (waitpid(pid, &status, __WALL) != pid) {}
return 0;
}

int loop(void *arg)
{
int i, pid, status;

write_file("/proc/self/setgroups", "deny");
if (!write_file("/proc/self/uid_map", "0 %d 1\n", real_uid))
fail("write of /proc/self/uid_map failed");
if (!write_file("/proc/self/gid_map", "0 %d 1\n", real_gid))
fail("write of /proc/self/gid_map failed");

if (mkdir("./syz-tmp", 0777))
fail("mkdir(syz-tmp) failed");
if (mount("", "./syz-tmp", "tmpfs", 0, NULL))
fail("mount(tmpfs) failed");
if (mkdir("./syz-tmp/newroot", 0777))
fail("mkdir failed");
if (mkdir("./syz-tmp/newroot/dev", 0700))
fail("mkdir failed");
if (mount("/dev", "./syz-tmp/newroot/dev", NULL, MS_BIND | MS_REC
| MS_PRIVATE, NULL))
fail("mount(dev) failed");
if (mkdir("./syz-tmp/pivot", 0777))
fail("mkdir failed");
if (syscall(SYS_pivot_root, "./syz-tmp", "./syz-tmp/pivot")) {
if (chdir("./syz-tmp"))
fail("chdir failed");
} else {
if (chdir("/"))
fail("chdir failed");
if (umount2("./pivot", MNT_DETACH))
fail("umount failed");
}
if (chroot("./newroot"))
fail("chroot failed");
if (chdir("/"))
fail("chdir failed");

for (i = 0;; i++) {
printf("%d\n", i);
pid = fork();
if (pid == -1)
fail("fork failed");
if (pid == 0) {
mount("/", "/", "",
MS_RDONLY|MS_SYNCHRONOUS|MS_BIND|MS_MOVE|MS_REC|MS_UNBINDABLE|MS_SLAVE,
0);
mount("/", "/", "",
MS_RDONLY|MS_SYNCHRONOUS|MS_BIND|MS_MOVE|MS_REC|MS_UNBINDABLE|MS_SLAVE,
0);
exit(0);
}
while (waitpid(pid, &status, __WALL) != pid) {
}
}
return 0;
}

bool write_file(const char* file, const char* what, ...)
{
char buf[1024];
va_list args;
va_start(args, what);
vsnprintf(buf, sizeof(buf), what, args);
va_end(args);
buf[sizeof(buf) - 1] = 0;
int len = strlen(buf);

int fd = open(file, O_WRONLY | O_CLOEXEC);
if (fd == -1)
return false;
if (write(fd, buf, len) != len) {
close(fd);
return false;
}
close(fd);
return true;
}

void fail(const char *what)
{
printf("%s (errno %d)\n", what, errno);
exit(1);
}

This program can be executed under a non-privileged user with the same
result. Each program iteration consumes hundreds of megs of kernel
memory, and this continues even when the program is killed. Soon
machine becomes completely unusable.

System seems to be trapped in dead loop trying to reclaim memory. OOM
killer and shrinker can't do anything useful; lockdep and rcu stall
detector are silent. The only thing that detects the issue is
workqueue stall detect:

[ 5835.871057] BUG: workqueue lockup - pool cpus=3 node=1 flags=0x0
nice=0 stuck for 5438s!

I've added NMI cpu traceback and lock dump into the workqueue lockup
detector and it reveals machine state somewhat. Below are 2 samples of
the output.

I am on commit 30f05309bde49295e02e45c7e615f73aa4e0ccc2 (Jan 20). But
I also tested the program on a 3.13 and 4.3 kernels. Result is the
same.

[ 451.551056] BUG: workqueue lockup - pool cpus=3 node=1 flags=0x0
nice=0 stuck for 54s!
[ 451.552104] Sending NMI to all CPUs:
[ 451.553965] NMI backtrace for cpu 0
[ 451.554422] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.0+ #275
[ 451.555167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 451.556156] task: ffffffff875bdc40 ti: ffffffff875a0000 task.ti:
ffffffff875a0000
[ 451.557087] RIP: 0010:[<ffffffff81262cc6>] [<ffffffff81262cc6>]
flat_send_IPI_mask+0x156/0x290
[ 451.558096] RSP: 0000:ffff88003ec07b78 EFLAGS: 00000046
[ 451.558717] RAX: 0000000000000000 RBX: 0000000000000c00 RCX: 0000000000000000
[ 451.559663] RDX: 0000000000000c00 RSI: 0000000000000000 RDI: ffffffffff5fc300
[ 451.560604] RBP: ffff88003ec07ba0 R08: 0000000000000001 R09: 0000000000000000
[ 451.561500] R10: ffffffff875bdc40 R11: 0000000000000001 R12: 0000000000000286
[ 451.562393] R13: 000000000f000000 R14: ffffffff87641d60 R15: 0000000000000002
[ 451.563398] FS: 0000000000000000(0000) GS:ffff88003ec00000(0000)
knlGS:0000000000000000
[ 451.564460] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 451.565255] CR2: 000000000045bb03 CR3: 000000006919f000 CR4: 00000000000006f0
[ 451.566285] Stack:
[ 451.566559] ffffffff87641d60 ffffffff882ce180 0000000000000040
fffffbfff1059638
[ 451.567450] ffff88006d71a4e0 ffff88003ec07bc0 ffffffff8125874e
ffffffff866ccb20
[ 451.568419] dffffc0000000000 ffff88003ec07c20 ffffffff829a017a
0000000000000001
[ 451.569504] Call Trace:
[ 451.569852] <IRQ> d [<ffffffff8125874e>] nmi_raise_cpu_backtrace+0x5e/0x80
[ 451.570761] [<ffffffff829a017a>] nmi_trigger_all_cpu_backtrace+0x49a/0x530
[ 451.571660] [<ffffffff812586f0>] ? irq_force_complete_move+0x60/0x60
[ 451.572559] [<ffffffff812587a4>] arch_trigger_all_cpu_backtrace+0x14/0x20
[ 451.573520] [<ffffffff813a9d3b>] wq_watchdog_timer_fn+0x46b/0x510
[ 451.574374] [<ffffffff813a998f>] ? wq_watchdog_timer_fn+0xbf/0x510
[ 451.575225] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 451.576143] [<ffffffff814b9e06>] call_timer_fn+0x176/0x550
[ 451.576892] [<ffffffff814b9d5e>] ? call_timer_fn+0xce/0x550
[ 451.577705] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 451.578592] [<ffffffff814b9c90>] ? process_timeout+0x20/0x20
[ 451.579305] [<ffffffff86332137>] ? _raw_spin_unlock_irq+0x27/0x70
[ 451.580219] [<ffffffff814bb325>] run_timer_softirq+0x5c5/0x9f0
[ 451.581039] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 451.581893] [<ffffffff814bad60>] ? msleep+0xe0/0xe0
[ 451.582555] [<ffffffff81361c0a>] __do_softirq+0x26a/0x920
[ 451.583344] [<ffffffff8136264f>] irq_exit+0x18f/0x1d0
[ 451.584015] [<ffffffff8125258e>] smp_apic_timer_interrupt+0x7e/0xa0
[ 451.584742] [<ffffffff863339ac>] apic_timer_interrupt+0x8c/0xa0
[ 451.585442] <EOI> d [<ffffffff81271096>] ? native_safe_halt+0x6/0x10
[ 451.586205] [<ffffffff811bdd42>] default_idle+0x52/0x2e0
[ 451.586836] [<ffffffff811bf37a>] arch_cpu_idle+0xa/0x10
[ 451.587476] [<ffffffff8143a058>] default_idle_call+0x48/0xa0
[ 451.588257] [<ffffffff8143a604>] cpu_startup_entry+0x554/0x710
[ 451.589032] [<ffffffff8630e452>] rest_init+0x192/0x1a0
[ 451.589792] [<ffffffff8830c780>] start_kernel+0x678/0x69e
[ 451.590507] [<ffffffff8830c108>] ? thread_info_cache_init+0xb/0xb
[ 451.591331] [<ffffffff8631efde>] ? memblock_reserve+0x59/0x5e
[ 451.592076] [<ffffffff8830b120>] ? early_idt_handler_array+0x120/0x120
[ 451.592925] [<ffffffff8830b342>] x86_64_start_reservations+0x2a/0x2c
[ 451.593764] [<ffffffff8830b49c>] x86_64_start_kernel+0x158/0x167
[ 451.594567] Code: 00 c3 5f ff 80 e6 10 75 e1 41 c1 e5 18 44 89 2c
25 10 c3 5f ff 44 89 fa 09 da 80 cf 04 41 83 ff 02 0f 44 d3 89 14 25
00 c3 5f ff <41> f7 c4 00 02 00 00 74 4a e8 fc 01 1f 00 48 c7 c7 68 58
64 87
[ 451.596894] NMI backtrace for cpu 1
[ 451.597334] CPU: 1 PID: 7282 Comm: syz-execprog Not tainted 4.4.0+ #275
[ 451.598144] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 451.599107] task: ffff880034290000 ti: ffff8800008c8000 task.ti:
ffff8800008c8000
[ 451.600118] RIP: 0010:[<ffffffff8144a794>] [<ffffffff8144a794>]
check_chain_key+0x284/0x5b0
[ 451.601220] RSP: 0000:ffff8800008cefe0 EFLAGS: 00000046
[ 451.601871] RAX: dffffc0000000000 RBX: ffff880034290818 RCX: ffff880034290000
[ 451.602704] RDX: 0000000000000003 RSI: ffff880034290820 RDI: 1ffff10006852103
[ 451.603539] RBP: ffff8800008cf040 R08: 0000000000000003 R09: 0000000000000001
[ 451.604372] R10: ffff880034290001 R11: 0000000000000000 R12: 0000000000000389
[ 451.605281] R13: 0000000e280a4389 R14: ffff880034290878 R15: dffffc0000000000
[ 451.606191] FS: 000000c82002c080(0000) GS:ffff88003ed00000(0000)
knlGS:0000000000000000
[ 451.607230] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 451.608018] CR2: 00007f66ed610433 CR3: 0000000016181000 CR4: 00000000000006e0
[ 451.608954] Stack:
[ 451.609220] 0000000000000004 ffff8800008cf038 ffffffff816733e4
ffffffff814545e9
[ 451.610158] ffff880034290820 ffffffff00000003 ffff880034290000
ffffffff88906188
[ 451.611195] ffff880034290878 ffffffff88906180 0000000e280a4000
ffffffff894f2ae0
[ 451.612206] Call Trace:
[ 451.612565] [<ffffffff816733e4>] ? domain_dirty_limits+0x224/0x540
[ 451.613468] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.614304] [<ffffffff814545e9>] __lock_acquire+0xea9/0x4700
[ 451.615109] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.615825] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.616761] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.617740] [<ffffffff8167dffe>] ? throttle_vm_writeout+0x13e/0x1c0
[ 451.618634] [<ffffffff82910030>] ? init_request_from_bio+0x1f0/0x1f0
[ 451.619535] [<ffffffff8145a28c>] lock_acquire+0x1dc/0x430
[ 451.620362] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 451.621167] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 451.621952] [<ffffffff86331ac3>] _raw_spin_lock+0x33/0x50
[ 451.622593] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 451.623402] [<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 451.624363] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 451.625185] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 451.626025] [<ffffffff81452edd>] ? trace_hardirqs_on+0xd/0x10
[ 451.626835] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 451.627708] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 451.628485] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 451.629330] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 451.630092] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 451.630871] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 451.631716] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 451.632502] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 451.633406] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 451.634231] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 451.634944] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 451.635716] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 451.636598] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 451.637470] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 451.638396] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.639291] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.640114] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 451.640917] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 451.641676] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 451.642456] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 451.643252] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 451.644069] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 451.644828] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 451.645568] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 451.646346] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 451.647045] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 451.647816] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 451.648609] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 451.649395] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 451.650199] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 451.650842] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 451.651591] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 451.652361] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 451.653018] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 451.653744] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 451.654506] [<ffffffff812705c4>] do_async_page_fault+0x14/
[ 451.655243] NMI backtrace for cpu 2
[ 451.655715] CPU: 2 PID: 7221 Comm: sshd Not tainted 4.4.0+ #275
[ 451.656497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 451.657530] task: ffff880036c5af80 ti: ffff880033010000 task.ti:
ffff880033010000
[ 451.658473] RIP: 0010:[<ffffffff8144cdbb>] [<ffffffff8144cdbb>]
__lock_is_held+0xbb/0x170
[ 451.659537] RSP: 0000:ffff880033017210 EFLAGS: 00000802
[ 451.660296] RAX: 1ffff10006d8b701 RBX: ffff880036c5b7f8 RCX: ffff880036c5b7a8
[ 451.661024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880036c5b808
[ 451.662068] RBP: ffff880033017240 R08: 0000000000000003 R09: 0000000000000001
[ 451.663097] R10: ffff880036c5af80 R11: 0000000000000001 R12: 0000000000000002
[ 451.664881] R13: dffffc0000000000 R14: ffff880036c5af80 R15: ffff88003dee4c18
[ 451.667037] FS: 00007f66ef26f7c0(0000) GS:ffff88006d600000(0000)
knlGS:0000000000000000
[ 451.668366] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 451.669172] CR2: 00007f4f9ea82470 CR3: 0000000034b2c000 CR4: 00000000000006e0
[ 451.670288] Stack:
[ 451.670584] 0000000000000000 0000000000000286 ffff88003dee4c18
ffff880036c5af80
[ 451.671647] 0000000000000001 ffff8800330173e8 ffff880033017268
ffffffff8145a631
[ 451.672687] ffff88003dee4c00 0000000000000080 ffffffffffffffff
ffff880033017290
[ 451.673729] Call Trace:
[ 451.674113] [<ffffffff8145a631>] lock_is_held+0x151/0x240
[ 451.674911] [<ffffffff816d8180>] __list_lru_count_one.isra.2+0x110/0x160
[ 451.675912] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 451.676778] [<ffffffff817b4f1f>] super_cache_count+0x10f/0x280
[ 451.677653] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 451.678540] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 451.679445] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 451.680452] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 451.681225] [<ffffffff8633371f>] ? retint_kernel+0x10/0x10
[ 451.681993] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 451.682837] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 451.683690] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 451.684445] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 451.685385] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 451.686252] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 451.687022] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 451.687867] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 451.688794] [<ffffffff863249f2>] ? preempt_schedule_common+0x42/0x70
[ 451.689696] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 451.690671] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 451.691522] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.692040] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.692850] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 451.693706] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 451.694527] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 451.695402] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 451.696255] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 451.697075] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 451.697904] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 451.698705] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 451.699508] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 451.700326] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 451.701139] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 451.701955] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 451.702780] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 451.703651] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 451.704475] [<ffffffff817eebcc>] ? poll_select_copy_remaining+0x25c/0x370
[ 451.705487] [<ffffffff817ee970>] ? set_fd_set+0x70/0x70
[ 451.706275] [<ffffffff82799042>] ? apparmor_file_permission+0x22/0x30
[ 451.707233] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 451.708072] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 451.708835] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 451.709683] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 451.710623] [<ffffffff812705c4>] do_async_page_fault+0x14/0xd0
[ 451.711497] [<ffffffff86334ef8>] async_page_fault+0x28/0x30
[ 451.712299] Code: 00 fc ff df 75 32 eb 70 41 83 c4 01 45 39 a6 20
08 00 00 7e 77 49 63 c4 48 8d 04 80 48 8d 1c c1 48 8d 7b 10 48 89 f8
48 c1 e8 03 <42> 80 3c 28 00 75 75 4c 3b 7b 10 74 40 48 8d 7b 22 48 89
f8 48
[ 451.714945] NMI backtrace for cpu 3
[ 451.715456] CPU: 3 PID: 1350 Comm: kworker/3:1 Not tainted 4.4.0+ #275
[ 451.716373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 451.717503] Workqueue: events pcpu_balance_workfn
[ 451.718176] task: ffff88006d3e4740 ti: ffff88006c010000 task.ti:
ffff88006c010000
[ 451.719224] RIP: 0010:[<ffffffff85b769a1>] [<ffffffff85b769a1>]
rpcauth_cache_shrink_count+0x1/0x70
[ 451.719943] RSP: 0018:ffff88006c017278 EFLAGS: 00000246
[ 451.720408] RAX: ffffffff85b769a0 RBX: 0000000000000000 RCX: dffffc0000000000
[ 451.720938] RDX: 0000000000000000 RSI: ffff88006c017348 RDI: ffffffff8813b140
[ 451.721471] RBP: ffff88006c0173b0 R08: 0000000000000004 R09: 0000000000000001
[ 451.722039] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[ 451.722568] R13: 0000000000000080 R14: 0000000000000000 R15: ffffffff8813b140
[ 451.723103] FS: 0000000000000000(0000) GS:ffff88006d700000(0000)
knlGS:0000000000000000
[ 451.723706] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 451.724149] CR2: 0000000000456800 CR3: 000000006919f000 CR4: 00000000000006e0
[ 451.724674] Stack:
[ 451.724835] ffff88006c0173b0 ffffffff8168f419 dffffc0000000000
ffff88006c017348
[ 451.725385] 0000000000000286 ffff88006c0172c8 1ffff1000d802e65
0000000000000013
[ 451.725931] 0000000000000080 ffff88003e831da0 ffff88006c0172d8
0000000000000000
[ 451.726481] Call Trace:
[ 451.726680] [<ffffffff8168f419>] ? shrink_slab.part.43+0x239/0x9f0
[ 451.727158] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 451.727612] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 451.728097] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 451.728506] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 451.728946] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 451.729412] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 451.729818] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 451.730415] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 451.730888] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 451.731318] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 451.731765] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 451.732261] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 451.732746] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 451.733281] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.733722] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 451.734169] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.734679] [<ffffffff863294f4>] ? mutex_lock_nested+0x6b4/0xa50
[ 451.735139] [<ffffffff829da8fd>] ? _find_next_bit.part.0+0xdd/0x110
[ 451.735622] [<ffffffff816c1e7d>] pcpu_populate_chunk+0x15d/0x910
[ 451.736086] [<ffffffff816c3ccc>] pcpu_balance_workfn+0xa9c/0xd60
[ 451.736539] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 451.737062] [<ffffffff816c3230>] ? __alloc_percpu+0x30/0x30
[ 451.737488] [<ffffffff8139fd36>] process_one_work+0x796/0x1440
[ 451.737934] [<ffffffff8139fc6a>] ? process_one_work+0x6ca/0x1440
[ 451.738472] [<ffffffff8139f5a0>] ? pwq_dec_nr_in_flight+0x2e0/0x2e0
[ 451.738953] [<ffffffff813a0abb>] worker_thread+0xdb/0xfc0
[ 451.739399] [<ffffffff86322cc4>] ? __schedule+0x964/0x1c50
[ 451.739822] [<ffffffff86332691>] ? _raw_spin_unlock_irqrestore+0x31/0xc0
[ 451.740431] [<ffffffff813b424f>] kthread+0x23f/0x2d0
[ 451.740809] [<ffffffff813a09e0>] ? process_one_work+0x1440/0x1440
[ 451.741274] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 451.741755] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 451.742245] [<ffffffff86332f6f>] ret_from_fork+0x3f/0x70
[ 451.742649] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 451.743139] Code: fb 48 89 d8 5b 41 5c 5d c3 31 db eb ef e8 48 80
be fb eb d0 e8 41 80 be fb eb a9 66 66 66 66 66 66 2e 0f 1f 84 00 00
00 00 00 55 <48> 89 e5 53 e8 b6 e6 9f fb 48 8b 05 ef 16 b3 03 48 c7 c7
30 df
[ 451.744560]
[ 451.744560] Showing all locks held in the system:
[ 451.745078] 1 lock held by kworker/3:2/3454:
[ 451.745740] #0: (&pool->manager_arb){+.+.+.}, at:
[<ffffffff813a12a7>] worker_thread+0x8c7/0xfc0
[ 451.747136] 2 locks held by getty/7203:
[ 451.747711] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.749072] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.750603] 2 locks held by getty/7204:
[ 451.751173] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.752522] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.753896] 2 locks held by getty/7205:
[ 451.754462] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.755786] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.757182] 2 locks held by getty/7206:
[ 451.757661] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.758904] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.760335] 2 locks held by getty/7207:
[ 451.760903] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.762243] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.763686] 2 locks held by getty/7208:
[ 451.764255] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 451.765586] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 451.767025] 1 lock held by sshd/7221:
[ 451.767564] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 451.768924] 1 lock held by syz-execprog/7282:
[ 451.769566] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 451.770979] 1 lock held by syz-executor/7321:
[ 451.771634] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 451.773066] 3 locks held by syz-executor/7322:
[ 451.773700] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 451.775074] #1: (namespace_sem){+++++.}, at: [<ffffffff818173c8>]
lock_mount+0xc8/0x520
[ 451.776345] #2: (pcpu_alloc_mutex){+.+.+.}, at:
[<ffffffff816c2ec8>] pcpu_alloc+0x898/0xba0



[ 3446.879074] BUG: workqueue lockup - pool cpus=3 node=1 flags=0x0
nice=0 stuck for 3049s!
[ 3446.880092] Sending NMI to all CPUs:
[ 3446.881852] NMI backtrace for cpu 0
[ 3446.882238] CPU: 0 PID: 2 Comm: kthreadd Not tainted 4.4.0+ #275
[ 3446.882908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 3446.883712] task: ffff88003df2df00 ti: ffff88003df50000 task.ti:
ffff88003df50000
[ 3446.884437] RIP: 0010:[<ffffffff81262cc6>] [<ffffffff81262cc6>]
flat_send_IPI_mask+0x156/0x290
[ 3446.885300] RSP: 0000:ffff88003ec07b78 EFLAGS: 00000046
[ 3446.885819] RAX: 0000000000000000 RBX: 0000000000000c00 RCX: 0000000000000000
[ 3446.886509] RDX: 0000000000000c00 RSI: 0000000000000000 RDI: ffffffffff5fc300
[ 3446.887204] RBP: ffff88003ec07ba0 R08: 0000000000000001 R09: 0000000000000000
[ 3446.887890] R10: ffff88003df2df00 R11: 0000000000000001 R12: 0000000000000286
[ 3446.888590] R13: 000000000f000000 R14: ffffffff87641d60 R15: 0000000000000002
[ 3446.889286] FS: 0000000000000000(0000) GS:ffff88003ec00000(0000)
knlGS:0000000000000000
[ 3446.890784] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3446.891796] CR2: 00007fc56755d433 CR3: 0000000016181000 CR4: 00000000000006f0
[ 3446.892820] Stack:
[ 3446.893138] ffffffff87641d60 ffffffff882ce180 0000000000000040
fffffbfff1059638
[ 3446.894861] ffff88006d71a4e0 ffff88003ec07bc0 ffffffff8125874e
ffffffff866ccb20
[ 3446.895944] dffffc0000000000 ffff88003ec07c20 ffffffff829a017a
0000000000000001
[ 3446.896981] Call Trace:
[ 3446.897350] <IRQ> d [<ffffffff8125874e>] nmi_raise_cpu_backtrace+0x5e/0x80
[ 3446.898349] [<ffffffff829a017a>] nmi_trigger_all_cpu_backtrace+0x49a/0x530
[ 3446.899356] [<ffffffff812586f0>] ? irq_force_complete_move+0x60/0x60
[ 3446.900273] [<ffffffff812587a4>] arch_trigger_all_cpu_backtrace+0x14/0x20
[ 3446.901316] [<ffffffff813a9d3b>] wq_watchdog_timer_fn+0x46b/0x510
[ 3446.902205] [<ffffffff813a998f>] ? wq_watchdog_timer_fn+0xbf/0x510
[ 3446.903103] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 3446.904077] [<ffffffff814b9e06>] call_timer_fn+0x176/0x550
[ 3446.904872] [<ffffffff814b9d5e>] ? call_timer_fn+0xce/0x550
[ 3446.905685] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 3446.906617] [<ffffffff814b9c90>] ? process_timeout+0x20/0x20
[ 3446.907457] [<ffffffff86332137>] ? _raw_spin_unlock_irq+0x27/0x70
[ 3446.908338] [<ffffffff814bb325>] run_timer_softirq+0x5c5/0x9f0
[ 3446.909190] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 3446.910129] [<ffffffff814bad60>] ? msleep+0xe0/0xe0
[ 3446.910937] [<ffffffff81452818>] ? mark_held_locks+0xc8/0x120
[ 3446.911760] [<ffffffff81361c0a>] __do_softirq+0x26a/0x920
[ 3446.912312] [<ffffffff8136264f>] irq_exit+0x18f/0x1d0
[ 3446.912909] [<ffffffff8125258e>] smp_apic_timer_interrupt+0x7e/0xa0
[ 3446.914014] [<ffffffff863339ac>] apic_timer_interrupt+0x8c/0xa0
[ 3446.914908] <EOI> d [<ffffffff814a1832>] ?
debug_lockdep_rcu_enabled.part.4+0x22/0x60
[ 3446.916292] [<ffffffff814a18e7>] debug_lockdep_rcu_enabled+0x77/0x90
[ 3446.917325] [<ffffffff8165f37d>] find_lock_task_mm+0x4d/0x2a0
[ 3446.918374] [<ffffffff8165f330>] ? oom_unkillable_task+0x360/0x360
[ 3446.919449] [<ffffffff8165f62a>] oom_badness+0x5a/0x390
[ 3446.919983] [<ffffffff816613e9>] out_of_memory+0x9e9/0x1060
[ 3446.920546] [<ffffffff8166125b>] ? out_of_memory+0x85b/0x1060
[ 3446.921177] [<ffffffff81660a00>] ? check_panic_on_oom+0xd0/0xd0
[ 3446.921868] [<ffffffff81671203>] __alloc_pages_nodemask+0x14b3/0x1760
[ 3446.922903] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 3446.923937] [<ffffffff82a43f00>] ? dynamic_emit_prefix+0x300/0x390
[ 3446.924560] [<ffffffff81756566>] ? alloc_debug_processing+0x166/0x190
[ 3446.925199] [<ffffffff817585a6>] ? ___slab_alloc+0x166/0x500
[ 3446.925754] [<ffffffff8134a8de>] ? copy_process.part.35+0x22e/0x57a0
[ 3446.926384] [<ffffffff8134a8de>] ? copy_process.part.35+0x22e/0x57a0
[ 3446.927016] [<ffffffff814a1aae>] ? rcu_read_lock_sched_held+0x9e/0x120
[ 3446.927672] [<ffffffff81671522>] alloc_kmem_pages_node+0x72/0x170
[ 3446.928139] [<ffffffff8134a901>] copy_process.part.35+0x251/0x57a0
[ 3446.928601] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 3446.929045] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.929526] [<ffffffff8134a6b0>] ? __cleanup_sighand+0x50/0x50
[ 3446.929963] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3446.930479] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.931300] [<ffffffff8135017c>] _do_fork+0x1bc/0xcb0
[ 3446.931728] [<ffffffff8134ffc0>] ? fork_idle+0x110/0x110
[ 3446.932159] [<ffffffff813b5eb6>] ? kthreadd+0x3a6/0x600
[ 3446.932554] [<ffffffff813b5f61>] ? kthreadd+0x451/0x600
[ 3446.932955] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.933447] [<ffffffff81350ca4>] kernel_thread+0x34/0x40
[ 3446.933850] [<ffffffff813b5fc0>] kthreadd+0x4b0/0x600
[ 3446.934239] [<ffffffff86332f6f>] ? ret_from_fork+0x3f/0x70
[ 3446.934665] [<ffffffff813b5b10>] ? kthread_create_on_cpu+0x150/0x150
[ 3446.935524] [<ffffffff813d5bde>] ? finish_t
[ 3446.936147] NMI backtrace for cpu 1
[ 3446.936572] CPU: 1 PID: 1524 Comm: kswapd0 Not tainted 4.4.0+ #275
[ 3446.937126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 3446.937810] task: ffff88003d904740 ti: ffff88003c328000 task.ti:
ffff88003c328000
[ 3446.938405] RIP: 0010:[<ffffffff8145196d>] [<ffffffff8145196d>]
mark_lock+0x11d/0xf00
[ 3446.938996] RSP: 0000:ffff88003c32f5f0 EFLAGS: 00000002
[ 3446.939420] RAX: 000000000000038a RBX: 0000000000000004 RCX: 0000000000000002
[ 3446.939961] RDX: 1ffff10007b209f6 RSI: ffff88003d904f90 RDI: ffff88003d904740
[ 3446.940506] RBP: ffff88003c32f630 R08: 1ffff10007b209f1 R09: 0000000000000000
[ 3446.941122] R10: ffff88003d904740 R11: ffff8800659799f8 R12: ffff88003d904f90
[ 3446.941695] R13: 0000000000000002 R14: ffff88003d904fb0 R15: ffff88003d904f90
[ 3446.942524] FS: 0000000000000000(0000) GS:ffff88003ed00000(0000)
knlGS:0000000000000000
[ 3446.943374] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3446.944209] CR2: 00007f20aa7be1fd CR3: 00000000631ec000 CR4: 00000000000006e0
[ 3446.945244] Stack:
[ 3446.945544] ffff88003d904f60 ffffffff00000002 ffff88003d904740
0000000000000389
[ 3446.946608] ffff88003d904f90 ffff88003d904fb2 0000000000000004
ffffffff894f2ae0
[ 3446.947665] ffff88003c32f7b8 ffffffff81454b47 ffff88003d904f60
ffffffff00000002
[ 3446.948718] Call Trace:
[ 3446.949082] [<ffffffff81454b47>] __lock_acquire+0x1407/0x4700
[ 3446.949906] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3446.950886] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3446.951976] [<ffffffff8145a28c>] lock_acquire+0x1dc/0x430
[ 3446.952781] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 3446.953783] [<ffffffff86331ac3>] _raw_spin_lock+0x33/0x50
[ 3446.954587] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 3446.955578] [<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 3446.956550] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 3446.957400] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 3446.958255] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 3446.959150] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 3446.959997] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 3446.960901] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 3446.961629] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 3446.962303] [<ffffffff81666800>] ? zone_watermark_ok_safe+0x1c0/0x280
[ 3446.962962] [<ffffffff816a12d2>] kswapd+0xa92/0x16e0
[ 3446.963499] [<ffffffff816a0840>] ? mem_cgroup_shrink_node_zone+0x5f0/0x5f0
[ 3446.964025] [<ffffffff86332137>] ? _raw_spin_unlock_irq+0x27/0x70
[ 3446.964482] [<ffffffff813d5bde>] ? finish_task_switch+0x14e/0x5f0
[ 3446.964938] [<ffffffff813d5bb0>] ? finish_task_switch+0x120/0x5f0
[ 3446.965400] [<ffffffff814396f0>] ? prepare_to_wait_event+0x410/0x410
[ 3446.965892] [<ffffffff86322cc4>] ? __schedule+0x964/0x1c50
[ 3446.966315] [<ffffffff86332691>] ? _raw_spin_unlock_irqrestore+0x31/0xc0
[ 3446.966816] [<ffffffff813b3a69>] ? __kthread_parkme+0x169/0x230
[ 3446.967266] [<ffffffff813b424f>] kthread+0x23f/0x2d0
[ 3446.967654] [<ffffffff816a0840>] ? mem_cgroup_shrink_node_zone+0x5f0/0x5f0
[ 3446.968172] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.968652] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.969140] [<ffffffff86332f6f>] ret_from_fork+0x3f/0x70
[ 3446.969542] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3446.970034] Code: ea 03 80 3c 02 00 0f 85 80 0d 00 00 49 85 5c 24
30 74 7b 41 bc 01 00 00 00 48 83 c4 18 44 89 e0 5b 41 5c 41 5d 41 5e
41 5f 5d c3 <4c> 89 f1 48 ba 00 00 00 00 00 fc ff df 48 c1 e9 03 0f b6
14 11
[ 3446.971543] NMI backtrace for cpu 2
[ 3446.971812] CPU: 2 PID: 7116 Comm: rsyslogd Not tainted 4.4.0+ #275
[ 3446.972284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 3446.972877] task: ffff8800638e17c0 ti: ffff880063080000 task.ti:
ffff880063080000
[ 3446.973461] RIP: 0010:[<ffffffff8145a85a>] [<ffffffff8145a85a>]
lock_release+0x13a/0xce0
[ 3446.974074] RSP: 0000:ffff8800630871a8 EFLAGS: 00000807
[ 3446.974471] RAX: dffffc0000000000 RBX: ffff880069e654b8 RCX: 0000000000000001
[ 3446.974998] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800638e1fe4
[ 3446.975551] RBP: ffff880063087250 R08: 0000000000000003 R09: 0000000000000001
[ 3446.976094] R10: ffff8800638e17c0 R11: 0000000000000001 R12: 0000000000000001
[ 3446.976629] R13: ffffffff879cbac0 R14: ffff8800638e17c0 R15: ffff8800630873e8
[ 3446.977163] FS: 00007f20a817c700(0000) GS:ffff88006d600000(0000)
knlGS:0000000000000000
[ 3446.977763] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3446.978197] CR2: 00007f4f9ea82470 CR3: 000000006521b000 CR4: 00000000000006e0
[ 3446.978724] Stack:
[ 3446.978884] ffff8800638e17c0 0000000000000080 0000000000000000
0000000000000001
[ 3446.979447] 0000000000000000 ffff880063087248 0000000000000286
0000000000000000
[ 3446.979997] ffffffff816d8096 ffff880000000000 0000000000000286
0000000000000002
[ 3446.980579] Call Trace:
[ 3446.980780] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 3446.982556] [<ffffffff816d815d>] ? __list_lru_count_one.isra.2+0xed/0x160
[ 3446.983185] [<ffffffff86331d0a>] _raw_spin_unlock+0x1a/0x30
[ 3446.983640] [<ffffffff816d815d>] __list_lru_count_one.isra.2+0xed/0x160
[ 3446.984178] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 3446.984687] [<ffffffff817b4f1f>] super_cache_count+0x10f/0x280
[ 3446.985172] [<ffffffff86331d12>] ? _raw_spin_unlock+0x22/0x30
[ 3446.985648] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 3446.986145] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 3446.986616] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 3446.987131] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 3446.987575] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 3446.988050] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 3446.988558] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 3446.989026] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 3446.989575] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 3446.990084] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 3446.990565] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 3446.991052] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 3446.991807] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 3446.992332] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 3446.992830] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 3446.993439] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3446.993994] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 3446.994615] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 3446.995128] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 3446.995655] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 3446.996169] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 3446.996731] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 3446.997228] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 3446.997731] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 3446.998210] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 3446.998765] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 3446.999260] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 3446.999787] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 3447.000656] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 3447.001381] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 3447.002078] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 3447.002556] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 3447.003427] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 3447.004473] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 3447.005648] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 3447.006872] [<ffffffff812705c4>] do_async_page_fault+0x14/0xd0
[ 3447.008085] [<ffffffff86334ef8>] async_page_fault+0x28/0x30
[ 3447.009236] Code: 0f 85 3d 08 00 00 65 4c 8b 34 25 c0 4e 01 00 49
8d be 24 08 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03
0f b6 14 02 <48> 89 f8 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 42 0b
00 00
[ 3447.012807] NMI backtrace for cpu 3
[ 3447.013408] CPU: 3 PID: 1350 Comm: kworker/3:1 Not tainted 4.4.0+ #275
[ 3447.014509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 3447.015835] Workqueue: events pcpu_balance_workfn
[ 3447.016516] task: ffff88006d3e4740 ti: ffff88006c010000 task.ti:
ffff88006c010000
[ 3447.017576] RIP: 0010:[<ffffffff8145ae2b>] [<ffffffff8145ae2b>]
lock_release+0x70b/0xce0
[ 3447.018739] RSP: 0018:ffff88006c017118 EFLAGS: 00000802
[ 3447.019497] RAX: 0000000000000000 RBX: 0000000000000005 RCX: ffff88006d3e502a
[ 3447.020506] RDX: 1ffff1000da7c9ec RSI: 0000000000000000 RDI: ffff88006d3e4f58
[ 3447.021502] RBP: ffff88006c0171c0 R08: 0000000000000005 R09: 0000000000000005
[ 3447.022107] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88006d3e4f68
[ 3447.022698] R13: ffff88006d3e5008 R14: 0000000000000004 R15: dffffc0000000000
[ 3447.023312] FS: 0000000000000000(0000) GS:ffff88006d700000(0000)
knlGS:0000000000000000
[ 3447.023918] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3447.024354] CR2: 0000000000456800 CR3: 000000006919f000 CR4: 00000000000006e0
[ 3447.024884] Stack:
[ 3447.025054] ffff88006d3e4740 0000000000000080 0000000000000000
0000000000000001
[ 3447.025602] 0000000000000000 ffff88006c0171b8 ffff88006d3e4f60
0000000000000000
[ 3447.026177] ffffffff816d8096 ffff880000000000 0000000000000286
ffff88006d3e4740
[ 3447.026725] Call Trace:
[ 3447.026917] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 3447.027446] [<ffffffff816d815d>] ? __list_lru_count_one.isra.2+0xed/0x160
[ 3447.027965] [<ffffffff86331d0a>] _raw_spin_unlock+0x1a/0x30
[ 3447.028395] [<ffffffff816d815d>] __list_lru_count_one.isra.2+0xed/0x160
[ 3447.028895] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 3447.029336] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 3447.029778] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 3447.030238] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 3447.030687] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 3447.031169] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 3447.031679] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 3447.032138] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 3447.032604] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 3447.033021] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 3447.033529] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 3447.033994] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 3447.034424] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 3447.034865] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 3447.035374] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 3447.035881] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 3447.036482] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 3447.036979] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 3447.037475] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3447.038049] [<ffffffff863294f4>] ? mutex_lock_nested+0x6b4/0xa50
[ 3447.038556] [<ffffffff829da8fd>] ? _find_next_bit.part.0+0xdd/0x110
[ 3447.039074] [<ffffffff816c1e7d>] pcpu_populate_chunk+0x15d/0x910
[ 3447.039570] [<ffffffff816c3ccc>] pcpu_balance_workfn+0xa9c/0xd60
[ 3447.040063] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 3447.040573] [<ffffffff816c3230>] ? __alloc_percpu+0x30/0x30
[ 3447.041509] [<ffffffff8139fd36>] process_one_work+0x796/0x1440
[ 3447.042356] [<ffffffff8139fc6a>] ? process_one_work+0x6ca/0x1440
[ 3447.043234] [<ffffffff8139f5a0>] ? pwq_dec_nr_in_flight+0x2e0/0x2e0
[ 3447.044151] [<ffffffff813a0abb>] worker_thread+0xdb/0xfc0
[ 3447.044933] [<ffffffff86322cc4>] ? __schedule+0x964/0x1c50
[ 3447.045734] [<ffffffff86332691>] ? _raw_spin_unlock_irqrestore+0x31/0xc0
[ 3447.046701] [<ffffffff813b424f>] kthread+0x23f/0x2d0
[ 3447.047448] [<ffffffff813a09e0>] ? process_one_work+0x1440/0x1440
[ 3447.048343] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3447.049272] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3447.050202] [<ffffffff86332f6f>] ret_from_fork+0x3f/0x70
[ 3447.050971] [<ffffffff813b4010>] ? kthread_create_on_node+0x3b0/0x3b0
[ 3447.051970] Code: 4f 89 e8 89 3b 30 00 8b 35 d3 7c 09 08 85 f6 0f
85 4a 02 00 00 48 8b 55 88 48 b8 00 00 00 00 00 fc ff df 48 c1 ea 03
0f b6 04 02 <84> c0 74 08 3c 03 0f 8e 0a 05 00 00 48 8b 45 d0 8b 75 c8
3b b0
[ 3447.054612]
[ 3447.054612] Showing all locks held in the system:
[ 3447.055511] 1 lock held by init/1:
[ 3447.056000] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 3447.057328] 6 locks held by kthreadd/2:
[ 3447.057874] #0: (oom_lock){+.+...}, at: [<ffffffff81670e3f>]
__alloc_pages_nodemask+0x10ef/0x1760
[ 3447.059234] #1: (rcu_read_lock){......}, at: [<ffffffff8166125b>]
out_of_memory+0x85b/0x1060
[ 3447.060547] #2: (rcu_read_lock){......}, at: [<ffffffff8165f330>]
find_lock_task_mm+0x0/0x2a0
[ 3447.061754] #3: (kernel/workqueue.c:5243){+.-.-.}, at:
[<ffffffff814b9d5e>] call_timer_fn+0xce/0x550
[ 3447.062523] #4: (rcu_read_lock){......}, at: [<ffffffff813a998f>]
wq_watchdog_timer_fn+0xbf/0x510
[ 3447.063284] #5: (tasklist_lock){.+.+..}, at: [<ffffffff8144f430>]
debug_show_all_locks+0x70/0x280
[ 3447.064053] 3 locks held by kworker/3:1/1350:
[ 3447.064387] #0: ("events"){.+.+.+}, at: [<ffffffff8139fc3a>]
process_one_work+0x69a/0x1440
[ 3447.065089] #1: (pcpu_balance_work){+.+.+.}, at:
[<ffffffff8139fc6a>] process_one_work+0x6ca/0x1440
[ 3447.065836] #2: (pcpu_alloc_mutex){+.+.+.}, at:
[<ffffffff816c32ea>] pcpu_balance_workfn+0xba/0xd60
[ 3447.066624] 1 lock held by kworker/3:2/3454:
[ 3447.066946] #0: (&pool->manager_arb){+.+.+.}, at:
[<ffffffff813a12a7>] worker_thread+0x8c7/0xfc0
[ 3447.067708] 1 lock held by rsyslogd/7116:
[ 3447.068021] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 3447.068730] 1 lock held by cron/7154:
[ 3447.069012] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 3447.069717] 1 lock held by sshd/7176:
[ 3447.069994] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 3447.070716] 2 locks held by getty/7203:
[ 3447.071014] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.071847] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.072720] 2 locks held by getty/7204:
[ 3447.073056] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.073846] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.074693] 2 locks held by getty/7205:
[ 3447.075022] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.075807] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.076660] 2 locks held by getty/7206:
[ 3447.076977] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.077765] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.078600] 2 locks held by getty/7207:
[ 3447.078915] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.079674] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.080794] 2 locks held by getty/7208:
[ 3447.081352] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 3447.082731] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 3447.084161] 1 lock held by syz-executor/7321:
[ 3447.084777] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 3447.086228] 3 locks held by syz-executor/7322:
[ 3447.086864] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 3447.088324] #1: (namespace_sem){+++++.}, at: [<ffffffff818173c8>]
lock_mount+0xc8/0x520
[ 3447.089568] #2: (pcpu_alloc_mutex){+.+.+.}, at:
[<ffffffff816c2ec8>] pcpu_alloc+0x898/0xba0



[ 5835.871057] BUG: workqueue lockup - pool cpus=3 node=1 flags=0x0
nice=0 stuck for 5438s!
[ 5835.872348] Sending NMI to all CPUs:
[ 5835.873396] NMI backtrace for cpu 0
[ 5835.873869] CPU: 0 PID: 7116 Comm: rsyslogd Not tainted 4.4.0+ #275
[ 5835.874674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 5835.875677] task: ffff8800638e17c0 ti: ffff880063080000 task.ti:
ffff880063080000
[ 5835.876487] RIP: 0010:[<ffffffff81262cc6>] [<ffffffff81262cc6>]
flat_send_IPI_mask+0x156/0x290
[ 5835.877499] RSP: 0000:ffff88003ec07b78 EFLAGS: 00000046
[ 5835.878090] RAX: 0000000000000000 RBX: 0000000000000c00 RCX: 0000000000000000
[ 5835.878871] RDX: 0000000000000c00 RSI: 0000000000000000 RDI: ffffffffff5fc300
[ 5835.879699] RBP: ffff88003ec07ba0 R08: 0000000000000001 R09: 0000000000000000
[ 5835.880578] R10: ffff8800638e17c0 R11: 0000000000000001 R12: 0000000000000286
[ 5835.881501] R13: 000000000f000000 R14: ffffffff87641d60 R15: 0000000000000002
[ 5835.882416] FS: 00007f20a817c700(0000) GS:ffff88003ec00000(0000)
knlGS:0000000000000000
[ 5835.883419] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5835.884093] CR2: 00007fc56755d433 CR3: 000000006521b000 CR4: 00000000000006f0
[ 5835.884852] Stack:
[ 5835.885103] ffffffff87641d60 ffffffff882ce180 0000000000000040
fffffbfff1059638
[ 5835.885892] ffff88006d71a4e0 ffff88003ec07bc0 ffffffff8125874e
ffffffff866ccb20
[ 5835.886797] dffffc0000000000 ffff88003ec07c20 ffffffff829a017a
0000000000000001
[ 5835.887777] Call Trace:
[ 5835.888073] <IRQ> d [<ffffffff8125874e>] nmi_raise_cpu_backtrace+0x5e/0x80
[ 5835.889074] [<ffffffff829a017a>] nmi_trigger_all_cpu_backtrace+0x49a/0x530
[ 5835.889955] [<ffffffff812586f0>] ? irq_force_complete_move+0x60/0x60
[ 5835.890780] [<ffffffff812587a4>] arch_trigger_all_cpu_backtrace+0x14/0x20
[ 5835.891613] [<ffffffff813a9d3b>] wq_watchdog_timer_fn+0x46b/0x510
[ 5835.892340] [<ffffffff813a998f>] ? wq_watchdog_timer_fn+0xbf/0x510
[ 5835.893078] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 5835.893834] [<ffffffff814b9e06>] call_timer_fn+0x176/0x550
[ 5835.894488] [<ffffffff814b9d5e>] ? call_timer_fn+0xce/0x550
[ 5835.895176] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 5835.895939] [<ffffffff814b9c90>] ? process_timeout+0x20/0x20
[ 5835.896618] [<ffffffff86332137>] ? _raw_spin_unlock_irq+0x27/0x70
[ 5835.897423] [<ffffffff814bb325>] run_timer_softirq+0x5c5/0x9f0
[ 5835.898133] [<ffffffff813a98d0>] ? show_workqueue_state+0x12a0/0x12a0
[ 5835.898828] [<ffffffff814bad60>] ? msleep+0xe0/0xe0
[ 5835.899398] [<ffffffff81452818>] ? mark_held_locks+0xc8/0x120
[ 5835.900045] [<ffffffff81361c0a>] __do_softirq+0x26a/0x920
[ 5835.900632] [<ffffffff8136264f>] irq_exit+0x18f/0x1d0
[ 5835.901196] [<ffffffff8125258e>] smp_apic_timer_interrupt+0x7e/0xa0
[ 5835.901879] [<ffffffff863339ac>] apic_timer_interrupt+0x8c/0xa0
[ 5835.902550] <EOI> d [<ffffffff8145a6a4>] ? lock_is_held+0x1c4/0x240
[ 5835.903327] [<ffffffff816d8180>] __list_lru_count_one.isra.2+0x110/0x160
[ 5835.904093] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 5835.904730] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 5835.905589] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 5835.906296] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 5835.907064] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 5835.908008] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 5835.909100] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 5835.910210] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 5835.911191] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 5835.911989] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 5835.912713] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 5835.913396] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 5835.914077] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 5835.914663] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 5835.915493] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 5835.916125] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 5835.916765] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 5835.918028] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 5835.919162] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 5835.919829] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 5835.920394] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 5835.921137] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 5835.921914] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 5835.922695] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 5835.923259] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 5835.923887] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 5835.924542] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 5835.925016] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 5835.925414] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 5835.925873] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 5835.926335] [<ffffffff816eb3e2>] handle_mm_fault+0x26
[ 5835.926871] NMI backtrace for cpu 1
[ 5835.927406] CPU: 1 PID: 1 Comm: init Not tainted 4.4.0+ #275
[ 5835.928165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 5835.929134] task: ffff88003df28000 ti: ffff88003df30000 task.ti:
ffff88003df30000
[ 5835.929930] RIP: 0010:[<ffffffff829c1e50>] [<ffffffff829c1e50>]
delay_tsc+0x60/0x70
[ 5835.930544] RSP: 0000:ffff88003df37200 EFLAGS: 00000206
[ 5835.931048] RAX: 00000dc830c34958 RBX: ffff880065979a60 RCX: 0000000000000028
[ 5835.931767] RDX: 00000dc800000000 RSI: 00000dc830c34930 RDI: 0000000000000001
[ 5835.932416] RBP: ffff88003df37200 R08: 0000000000000001 R09: 0000000000000001
[ 5835.933043] R10: ffff88003df28000 R11: 0000000000000000 R12: ffff880065979a70
[ 5835.933691] R13: 000000009a9d2d40 R14: ffff880065979a68 R15: 0000000000009043
[ 5835.934337] FS: 00007f894a17b7a0(0000) GS:ffff88003ed00000(0000)
knlGS:0000000000000000
[ 5835.934953] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5835.935420] CR2: 00007f20aa7be1fd CR3: 000000006919f000 CR4: 00000000000006e0
[ 5835.935982] Stack:
[ 5835.936153] ffff88003df37210 ffffffff829c1d5a ffff88003df37248
ffffffff81464709
[ 5835.936732] ffff880065979a60 0000000000000080 ffffffffffffffff
0000000000000001
[ 5835.937550] ffff88003df373e8 ffff88003df37268 ffffffff86331acb
ffffffff816d8096
[ 5835.938424] Call Trace:
[ 5835.938722] [<ffffffff829c1d5a>] __delay+0xa/0x10
[ 5835.939341] [<ffffffff81464709>] do_raw_spin_lock+0x149/0x2b0
[ 5835.940117] [<ffffffff86331acb>] _raw_spin_lock+0x3b/0x50
[ 5835.940734] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 5835.941493] [<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 5835.942476] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 5835.943385] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 5835.944288] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 5835.945025] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 5835.945697] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 5835.946546] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 5835.947141] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 5835.947767] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 5835.948269] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 5835.948701] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 5835.949351] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 5835.949941] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 5835.950368] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 5835.950835] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 5835.951338] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 5835.951853] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 5835.952388] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 5835.952930] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 5835.953595] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 5835.954357] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 5835.955025] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 5835.955749] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 5835.956455] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 5835.957277] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 5835.957924] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 5835.958579] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 5835.959261] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 5835.959760] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 5835.960310] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 5835.960824] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 5835.961458] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 5835.962369] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 5835.963073] [<ffffffff817eebcc>] ? poll_select_copy_remaining+0x25c/0x370
[ 5835.963776] [<ffffffff817ee970>] ? set_fd_set+0x70/0x70
[ 5835.964318] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 5835.965028] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 5835.965500] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 5835.966190] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 5835.966995] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 5835.967852] [<ffffffff812705c4>] do_async_page_fault+0x14/0xd0
[ 5835.968304] [<ffffffff86334ef8>] async_page_fault+0x28/0x30
[ 5835.968760] Code: 39 f9 73 27 65 ff 0d 20 30 65 7d f3 90 65 ff 05
17 30 65 7d 65 8b 0d e0 12 65 7d 44 39 c1 74 cd 48 29 c6 41 89 c8 48
01 f7 eb b3 <65> ff 0d f9 2f 65 7d 5d c3 0f 1f 80 00 00 00 00 48 81 3d
55 90
[ 5835.970216] NMI backtrace for cpu 2
[ 5835.970485] CPU: 2 PID: 7176 Comm: sshd Not tainted 4.4.0+ #275
[ 5835.971370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 5835.972538] task: ffff88003338af80 ti: ffff8800330d8000 task.ti:
ffff8800330d8000
[ 5835.973618] RIP: 0010:[<ffffffff829c1dfb>] [<ffffffff829c1dfb>]
delay_tsc+0xb/0x70
[ 5835.974713] RSP: 0000:ffff8800330df200 EFLAGS: 00000282
[ 5835.975501] RAX: 0000000000000001 RBX: ffff880065979a60 RCX: 000000000000002c
[ 5835.976532] RDX: 00000dc800000000 RSI: 00000dc830c34ea8 RDI: 0000000000000001
[ 5835.977630] RBP: ffff8800330df200 R08: 0000000000000002 R09: 0000000000000001
[ 5835.978654] R10: ffff88003338af80 R11: 0000000000000000 R12: ffff880065979a70
[ 5835.979683] R13: 000000009a9d2d40 R14: ffff880065979a68 R15: 000000000000991f
[ 5835.980706] FS: 00007fc5691bc7c0(0000) GS:ffff88006d600000(0000)
knlGS:0000000000000000
[ 5835.981867] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5835.982700] CR2: 00007f4f9ea82470 CR3: 000000003c2f6000 CR4: 00000000000006e0
[ 5835.983740] Stack:
[ 5835.984061] ffff8800330df210 ffffffff829c1d5a ffff8800330df248
ffffffff81464709
[ 5835.985125] ffff880065979a60 0000000000000080 ffffffffffffffff
0000000000000001
[ 5835.986196] ffff8800330df3e8 ffff8800330df268 ffffffff86331acb
ffffffff816d8096
[ 5835.987312] Call Trace:
[ 5835.987679] [<ffffffff829c1d5a>] __delay+0xa/0x10
[ 5835.988374] [<ffffffff81464709>] do_raw_spin_lock+0x149/0x2b0
[ 5835.989217] [<ffffffff86331acb>] _raw_spin_lock+0x3b/0x50
[ 5835.989995] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 5835.990982] [<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 5835.991957] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 5835.992806] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 5835.993667] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 5835.994534] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 5835.995415] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 5835.996333] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 5835.997113] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 5835.998030] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 5835.998910] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 5835.999707] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 5836.000681] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 5836.001587] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 5836.002397] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 5836.003244] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 5836.004187] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 5836.005109] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 5836.006114] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 5836.007101] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 5836.007788] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 5836.008308] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 5836.008746] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 5836.009226] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 5836.009694] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 5836.010150] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 5836.010977] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 5836.011804] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 5836.012633] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 5836.013403] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 5836.014262] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 5836.015139] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 5836.015988] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 5836.016857] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 5836.017747] [<ffffffff817eebcc>] ? poll_select_copy_remaining+0x25c/0x370
[ 5836.018739] [<ffffffff817ee970>] ? set_fd_set+0x70/0x70
[ 5836.019534] [<ffffffff81375386>] ? recalc_sigpending+0x76/0xa0
[ 5836.020392] [<ffffffff8137898d>] ? __set_task_blocked+0xfd/0x230
[ 5836.021270] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 5836.022099] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 5836.022844] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 5836.023674] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 5836.024544] [<ffffffff812705c4>] do_async_page_fault+0x14/0xd0
[ 5836.025395] [<ffffffff86334ef8>] async_page_fault+0x28/0x30
[ 5836.026216] Code: 48 c1 e0 02 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a
01 ff 15 d9 90 00 05 5d c3 0f 1f 80 00 00 00 00 55 65 ff 05 58 30 65
7d 48 89 e5 <65> 44 8b 05 1d 13 65 7d 0f ae e8 0f 31 48 c1 e2 20 48 89
d6 48
[ 5836.028212] NMI backtrace for cpu 3
[ 5836.028480] CPU: 3 PID: 7221 Comm: sshd Not tainted 4.4.0+ #275
[ 5836.028919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 5836.029519] task: ffff880036c5af80 ti: ffff880033010000 task.ti:
ffff880033010000
[ 5836.030082] RIP: 0010:[<ffffffff829c1e06>] [<ffffffff829c1e06>]
delay_tsc+0x16/0x70
[ 5836.030652] RSP: 0000:ffff880033017200 EFLAGS: 00000282
[ 5836.031053] RAX: 0000000000000001 RBX: ffff880065979a60 RCX: 000000000000002c
[ 5836.031599] RDX: 00000dc800000000 RSI: 00000dc830c35204 RDI: 0000000000000001
[ 5836.032136] RBP: ffff880033017200 R08: 0000000000000003 R09: 0000000000000001
[ 5836.032662] R10: ffff880036c5af80 R11: 0000000000000000 R12: ffff880065979a70
[ 5836.033193] R13: 000000009a9d2d40 R14: ffff880065979a68 R15: 000000000000915d
[ 5836.033718] FS: 00007f66ef26f7c0(0000) GS:ffff88006d700000(0000)
knlGS:0000000000000000
[ 5836.034317] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5836.034744] CR2: 000000000045bb03 CR3: 0000000034b2c000 CR4: 00000000000006e0
[ 5836.035280] Stack:
[ 5836.035446] ffff880033017210 ffffffff829c1d5a ffff880033017248
ffffffff81464709
[ 5836.035989] ffff880065979a60 0000000000000080 ffffffffffffffff
0000000000000001
[ 5836.036542] ffff8800330173e8 ffff880033017268 ffffffff86331acb
ffffffff816d8096
[ 5836.037097] Call Trace:
[ 5836.037290] [<ffffffff829c1d5a>] __delay+0xa/0x10
[ 5836.037701] [<ffffffff81464709>] do_raw_spin_lock+0x149/0x2b0
[ 5836.038141] [<ffffffff86331acb>] _raw_spin_lock+0x3b/0x50
[ 5836.038549] [<ffffffff816d8096>] ? __list_lru_count_one.isra.2+0x26/0x160
[ 5836.039071] [<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 5836.039573] [<ffffffff816d824f>] list_lru_count_one+0x7f/0xa0
[ 5836.040013] [<ffffffff817b4f7a>] super_cache_count+0x16a/0x280
[ 5836.040454] [<ffffffff8168f419>] shrink_slab.part.43+0x239/0x9f0
[ 5836.040902] [<ffffffff8178fbcf>] ? mem_cgroup_iter+0x38f/0xed0
[ 5836.041346] [<ffffffff8168f1e0>] ? current_may_throttle+0x210/0x210
[ 5836.041818] [<ffffffff8169dbaf>] shrink_zone+0xa2f/0xd80
[ 5836.042228] [<ffffffff8169d180>] ? shrink_lruvec+0x1080/0x1080
[ 5836.042668] [<ffffffff81270c45>] ? kvm_clock_get_cycles+0x25/0x30
[ 5836.043135] [<ffffffff814d67b6>] ? ktime_get+0x1c6/0x2a0
[ 5836.043551] [<ffffffff81587881>] ? __delayacct_freepages_start+0x41/0x80
[ 5836.044129] [<ffffffff8169e367>] do_try_to_free_pages+0x467/0x1090
[ 5836.044638] [<ffffffff8169df00>] ? shrink_zone+0xd80/0xd80
[ 5836.045111] [<ffffffff8169f18d>] try_to_free_pages+0x1fd/0x560
[ 5836.045610] [<ffffffff8169ef90>] ? do_try_to_free_pages+0x1090/0x1090
[ 5836.046158] [<ffffffff81670b60>] __alloc_pages_nodemask+0xe10/0x1760
[ 5836.046693] [<ffffffff8166fd50>] ? __alloc_pages_direct_compact+0x280/0x280
[ 5836.047279] [<ffffffff81453740>] ? debug_check_no_locks_freed+0x3c0/0x3c0
[ 5836.047984] [<ffffffff814545e9>] ? __lock_acquire+0xea9/0x4700
[ 5836.048731] [<ffffffff81652b17>] ? __page_cache_alloc+0x107/0x3e0
[ 5836.049398] [<ffffffff81653a52>] ? find_get_entry+0x182/0x350
[ 5836.050070] [<ffffffff817487b9>] alloc_pages_current+0xe9/0x450
[ 5836.050723] [<ffffffff816538d0>] ? wait_on_page_read+0x210/0x210
[ 5836.051474] [<ffffffff81652cdf>] __page_cache_alloc+0x2cf/0x3e0
[ 5836.052124] [<ffffffff8165b3b1>] ? filemap_fault+0xa01/0x11a0
[ 5836.052634] [<ffffffff8165b510>] filemap_fault+0xb60/0x11a0
[ 5836.053142] [<ffffffff8165b2ae>] ? filemap_fault+0x8fe/0x11a0
[ 5836.053652] [<ffffffff816e09d2>] __do_fault+0x1b2/0x320
[ 5836.054154] [<ffffffff816e0820>] ? do_page_mkwrite+0x310/0x310
[ 5836.054614] [<ffffffff816eb3b7>] ? handle_mm_fault+0x2637/0x4640
[ 5836.055142] [<ffffffff816eb3e2>] handle_mm_fault+0x2662/0x4640
[ 5836.055682] [<ffffffff816e8e86>] ? handle_mm_fault+0x106/0x4640
[ 5836.056229] [<ffffffff816e8d80>] ? __pmd_alloc+0x430/0x430
[ 5836.056706] [<ffffffff817eebcc>] ? poll_select_copy_remaining+0x25c/0x370
[ 5836.057318] [<ffffffff817ee970>] ? set_fd_set+0x70/0x70
[ 5836.057806] [<ffffffff82799042>] ? apparmor_file_permission+0x22/0x30
[ 5836.058365] [<ffffffff816d39d9>] ? vmacache_find+0x59/0x290
[ 5836.058862] [<ffffffff816f8770>] ? find_vma+0x30/0x150
[ 5836.059316] [<ffffffff8127eff6>] __do_page_fault+0x376/0x960
[ 5836.059771] [<ffffffff8127f738>] trace_do_page_fault+0xe8/0x420
[ 5836.060309] [<ffffffff812705c4>] do_async_page_fault+0x14/0xd0
[ 5836.060791] [<ffffffff86334ef8>] async_page_fault+0x28/0x30
[ 5836.061522] Code: f7 e2 48 8d 7a 01 ff 15 d9 90 00 05 5d c3 0f 1f
80 00 00 00 00 55 65 ff 05 58 30 65 7d 48 89 e5 65 44 8b 05 1d 13 65
7d 0f ae e8 <0f> 31 48 c1 e2 20 48 89 d6 48 09 c6 0f ae e8 0f 31 48 c1
e2 20
[ 5836.063057]
[ 5836.063057] Showing all locks held in the system:
[ 5836.063659] 1 lock held by kworker/3:2/3454:
[ 5836.064029] #0: (&pool->manager_arb){+.+.+.}, at:
[<ffffffff813a12a7>] worker_thread+0x8c7/0xfc0
[ 5836.064846] 6 locks held by rsyslogd/7116:
[ 5836.065207] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff8127ef26>]
__do_page_fault+0x2a6/0x960
[ 5836.066066] #1: (shrinker_rwsem){++++..}, at:
[<ffffffff8168f287>] shrink_slab.part.43+0xa7/0x9f0
[ 5836.066922] #2: (&(&lru->node[i].lock)->rlock){+.+.-.}, at:
[<ffffffff816d8096>] __list_lru_count_one.isra.2+0x26/0x160
[ 5836.068043] #3: (kernel/workqueue.c:5243){+.-.-.}, at:
[<ffffffff814b9d5e>] call_timer_fn+0xce/0x550
[ 5836.068804] #4: (rcu_read_lock){......}, at: [<ffffffff813a998f>]
wq_watchdog_timer_fn+0xbf/0x510
[ 5836.069584] #5: (tasklist_lock){.+.+..}, at: [<ffffffff8144f430>]
debug_show_all_locks+0x70/0x280
[ 5836.070446] 2 locks held by getty/7203:
[ 5836.070880] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.071757] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.072586] 2 locks held by getty/7204:
[ 5836.072877] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.073631] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.074929] 2 locks held by getty/7205:
[ 5836.075511] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.076839] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.078158] 2 locks held by getty/7206:
[ 5836.078476] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.079246] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.080191] 2 locks held by getty/7207:
[ 5836.080486] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.081407] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.082259] 2 locks held by getty/7208:
[ 5836.082553] #0: (&tty->ldisc_sem){++++++}, at:
[<ffffffff86331a47>] ldsem_down_read+0x37/0x40
[ 5836.083339] #1: (&ldata->atomic_read_lock){+.+...}, at:
[<ffffffff82cf8021>] n_tty_read+0x201/0x1d30
[ 5836.084210] 1 lock held by syz-executor/7321:
[ 5836.084642] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 5836.085674] 3 locks held by syz-executor/7322:
[ 5836.086047] #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at:
[<ffffffff8181738e>] lock_mount+0x8e/0x520
[ 5836.086825] #1: (namespace_sem){+++++.}, at: [<ffffffff818173c8>]
lock_mount+0xc8/0x520
[ 5836.087558] #2: (pcpu_alloc_mutex){+.+.+.}, at:
[<ffffffff816c2ec8>] pcpu_alloc+0x898/0xba0

[Al Viro]

On Fri, Jan 22, 2016 at 10:06:14PM +0100, Dmitry Vyukov wrote:
> Hello,
>
> While running syzkaller fuzzer I hit the following problem. Supervisor
> process sandboxes worker processes that do random activities with
> CLONE_NEWUSER | CLONE_NEWNS | CLONE_NEWPID | CLONE_NEWUTS |
> CLONE_NEWNET | CLONE_NEWIPC | CLONE_IO, setrlimit, chroot, etc.
> Because of that worker process gains ability to bring whole machine
> down (does not happen without the sandbox).

AFAICS, what you are doing is essentially mount --rbind / / in infinite
loop in luserns. Which ends up eating all memory. There's any number
of ways to do the same. We can play whack-a-mole with them until the
kernel is completely ossified with accounting code of different sorts.
Or one can disable userns and be done with that.

[Dmitry Vyukov]

My 2GB VM dies at around just 10-th iteration, is it normal?
Each iteration consumes several hundreds of megs of kernel memory. And
there seems to be exponential slowdown at around 5-th iteration.
I understand that there can be lots of forms of a local DoS. But there
seems to be something pathological about this particular one. And it
happens only with sandboxing that is meant to reduce DoS
possibilities...

[Kees Cook]

I think it's time for a runtime tunable to disable userns. For all the
distro users that are forced to run with kernels built with
CONFIG_USER_NS, there's no way for them to escape these USER_NS flaws
when they don't use the feature.

-Kees

--
Kees Cook
Chrome OS & Brillo Security

[Al Viro]

On Fri, Jan 22, 2016 at 10:38:40PM +0100, Dmitry Vyukov wrote:

> My 2GB VM dies at around just 10-th iteration, is it normal?
> Each iteration consumes several hundreds of megs of kernel memory. And
> there seems to be exponential slowdown at around 5-th iteration.
> I understand that there can be lots of forms of a local DoS. But there
> seems to be something pathological about this particular one. And it
> happens only with sandboxing that is meant to reduce DoS
> possibilities...

Sandboxing == giving attacker to do mount without being root.

As for exponential - sure, you double the amount of mounts on each step
(if not quadruple - I don't have your code in front of me right now,
but ISTR two mount --rbind in there). More obvious form would be
for i in `seq 64`; do mkdir /tmp/$i; mount --rbind / /tmp/$i; done - there
the entire tree would be visible (and visibly exponential by the number of
iterations).

I doubt that the first iteration chews hundreds of megs, BTW. If you
really get two mount --rbind per iteration, you would have about a million-fold
increase of the number of mounts after ten iterations, and having that chew
2Gb is quite plausible; in that case the first iteration would eat about 10K
or so...

[Robert Święcki]

2016-01-22 22:55 GMT+01:00 Al Viro <vi...@zeniv.linux.org.uk>:
> On Fri, Jan 22, 2016 at 10:38:40PM +0100, Dmitry Vyukov wrote:
>
>> My 2GB VM dies at around just 10-th iteration, is it normal?
>> Each iteration consumes several hundreds of megs of kernel memory. And
>> there seems to be exponential slowdown at around 5-th iteration.
>> I understand that there can be lots of forms of a local DoS. But there
>> seems to be something pathological about this particular one. And it
>> happens only with sandboxing that is meant to reduce DoS
>> possibilities...
>
> Sandboxing == giving attacker to do mount without being root.

I was discussing this initially with Dmitry and maybe I explained it
initially a bit incorrectly. I did not mean to suggest that using
CLONE_NEWUSER alone is a form of sandboxing. But rather, that when
used correctly (with dropping capabilities, rlimits and seccomp-bpf
filters) it could constitute a form of sandboxing.

What I suggested was to use CLONE_NEWUSER and friends to test the new
attack surface, which is enabled by using CLONE_NEW*. For regular
users the syscall(__NR_mount) returns early with EPERM, but when
CLONE_NEW* are used, a new, big attack surface opens up, reachable
from a level of a unprivileged user.

So, I guess, it's not about sandboxing but the newly reachable attack surface.

--
Robert Święcki