Incredible HD thrashing on 4.0
Kelly Dean
The problem seems to be triggered by individual qubes doing ordinary bursts of disk access, such as loading a program or accessing swap, which would normally take just a few seconds on Qubes 3.2, but dom0 then massively multiplies that I/O on Qubes 4.0, leading to disk thrashing that drags on for minutes at a time, and in some cases, more than an hour.
iotop in dom0 says the thrashing procs are e.g. [21.xvda-0] and [21.xvda-1], reading the disk at rates ranging from 10 to 50 MBps (max throughput of the disk is about 100). At this rate, for how prolonged the thrashing is, it could have read and re-read the entire virtual disk multiple times over, so there's something extremely inefficient going on.
Is there any solution other than installing a SSD? I'd prefer not to have to add hardware to solve a software performance regression.
Chris Laprise
The HDD mode avoids some thrashing.
Also, I remember installing a 4.0 release candidate on an external HDD
and didn't note unusual thrashing at the time.
--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
Unman
definitely not "minutes at a time". Is there something significant about
the disks that you cite, or are those just examples?
Kelly Dean
Unman writes:
> I don't recognise this on a somewhat under powered laptop with HDD -
> definitely not "minutes at a time". Is there something significant about
> the disks that you cite, or are those just examples?
System is a core i3 with 16GB RAM, and HD with about 100MB/s throughput.
Worst problems seem to be from swapping, and random times when I start a qube.
The swapping is unpredictable, but here's a typical best-case result for starting an ordinary app qube with fedora-28 template:
T+0: start qube. Brief burst of CPU & disk activity for a second, then mostly idle for 20 seconds.
T+20: heavy sustained disk thrashing starts.
T+40: pop-up notification that the domain has started. Thrashing continues.
T+60: thrashing abruptly stops.
That's only 1 minute, but when I'm unlucky, it can be several minutes.
How does that compare with your experience?
I don't have anything custom configured to run in the qube at startup, so all the activity is from the template's defaults. Nothing special about fedora-28 either; I get similar results from debian-9 and whonix-ws.
Chris Laprise
'xl info'.
Sphere
Same here for me, I hear lots of scratching sounds from the HDD whenever I do something in the laptop. Extremely worries me that the HDD might die soon because of it D:
Kelly Dean
Chris Laprise writes:
> Can Qubes access all of that RAM? Look at the total_memory figure from
> 'xl info'.
One additional data point: after the typical one-minute boot time for a qube, it's using no swap space, and dom0 is also using no swap space, even though both do have swap enabled. So, memory pressure isn't the problem.
Some other qubes are using some swap space, but they were idle while I was timing the boot of the test qube.
Marcus Linsner
Interestingly, I've just encountered this thrashing, but on SSD(it's just reading 192MiB/sec constantly), Qubes R4.0 up to date, inside a qube while compiling firefox: typing in any of 3 of its terminal windows does not even echo anything and the firefox compilation terminal is frozen; the swap (of 1G) was turned off a while ago (via swapoff); I used Qube Manager to Pause the offending cube and the thrashing stopped. I don't see much on logs.
Ok so I resumed the qube, the thrashing resumed for a few seconds then stopped and all terminals were alive again (I can type into them). The log spewed some new things (since the updatedb audit which was last while Paused), I'm including some long lines from before, note that the log after the unpause starts from "[ 6862.846945] INFO: rcu_sched self-detected stall on CPU", as follows:
[ 0.000000] Linux version 4.14.57-1.pvops.qubes.x86_64 (user@build-fedora4) (gcc version 6.4.1 20170727 (Red Hat 6.4.1-1) (GCC)) #1 SMP Mon Jul 23 16:28:54 UTC 2018
[ 0.000000] Command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 nopat
...
[ 2769.581919] audit: type=1101 audit(1534434741.005:133): pid=10290 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2769.582396] audit: type=1123 audit(1534434741.005:134): pid=10290 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd=737761706F6666202F6465762F7876646331 terminal=pts/3 res=success'
[ 2769.582525] audit: type=1110 audit(1534434741.006:135): pid=10290 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2769.583384] audit: type=1105 audit(1534434741.007:136): pid=10290 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2776.388700] audit: type=1106 audit(1534434747.812:137): pid=10290 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 2776.388735] audit: type=1104 audit(1534434747.812:138): pid=10290 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success'
[ 4093.008056] audit: type=1116 audit(1534436064.432:139): pid=29167 uid=0 auid=4294967295 ses=4294967295 msg='op=add-group id=982 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4093.030620] audit: type=1132 audit(1534436064.454:140): pid=29167 uid=0 auid=4294967295 ses=4294967295 msg='op=add-shadow-group id=982 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4093.304708] audit: type=1130 audit(1534436064.728:141): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=run-rfbdacad57c5f4bc183d36a7c402c9ae7 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4094.576065] audit: type=1130 audit(1534436065.999:142): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4094.576138] audit: type=1131 audit(1534436065.999:143): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4094.577822] audit: type=1131 audit(1534436066.001:144): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=run-rfbdacad57c5f4bc183d36a7c402c9ae7 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4219.569289] audit: type=1100 audit(1534436190.993:145): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.569317] audit: type=1101 audit(1534436190.993:146): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_succeed_if acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.569499] audit: type=1103 audit(1534436190.993:147): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.575980] audit: type=1105 audit(1534436190.999:148): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.596837] audit: type=1106 audit(1534436191.020:149): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4219.596882] audit: type=1104 audit(1534436191.020:150): pid=30881 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528350] audit: type=1100 audit(1534436507.950:151): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528387] audit: type=1101 audit(1534436507.950:152): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_succeed_if acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.528426] audit: type=1103 audit(1534436507.951:153): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.532307] audit: type=1105 audit(1534436507.956:154): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.555390] audit: type=1106 audit(1534436507.979:155): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_xauth acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4536.555432] audit: type=1104 audit(1534436507.979:156): pid=580 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/bin/su" hostname=? addr=? terminal=? res=success'
[ 4562.928666] audit: type=1116 audit(1534436534.352:157): pid=1009 uid=0 auid=4294967295 ses=4294967295 msg='op=add-group id=21 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4562.939364] audit: type=1132 audit(1534436534.363:158): pid=1009 uid=0 auid=4294967295 ses=4294967295 msg='op=add-shadow-group id=21 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'
[ 4563.070979] audit: type=1130 audit(1534436534.494:159): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.675418] audit: type=1130 audit(1534436535.099:160): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.675458] audit: type=1131 audit(1534436535.099:161): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=man-db-cache-update comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.676716] audit: type=1130 audit(1534436535.100:162): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4563.676754] audit: type=1131 audit(1534436535.100:163): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=run-r2587a42691cd4b91a079a19ad55b8789 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4570.160514] audit: type=1101 audit(1534436541.584:164): pid=1473 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4570.162254] audit: type=1123 audit(1534436541.584:165): pid=1473 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd="updatedb" terminal=pts/5 res=success'
[ 4570.162328] audit: type=1110 audit(1534436541.585:166): pid=1473 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4570.165188] audit: type=1105 audit(1534436541.588:167): pid=1473 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4580.680238] audit: type=1106 audit(1534436552.103:168): pid=1473 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 4580.680282] audit: type=1104 audit(1534436552.103:169): pid=1473 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/5 res=success'
[ 6862.846945] INFO: rcu_sched self-detected stall on CPU
[ 6862.846946] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 6862.846950] 0-...: (4 GPs behind) idle=3f6/1/0 softirq=1100880/1100880 fqs=0
[ 6862.846956] 2-...: (14 GPs behind) idle=844/0/0 softirq=1162047/1162047 fqs=0
[ 6862.846958] 3-...: (1 ticks this GP) idle=37e/1/0 softirq=1154645/1154645 fqs=0
[ 6862.846960] 4-...: (4 GPs behind) idle=fb2/1/0 softirq=1208765/1208765 fqs=0
[ 6862.846961] 5-...: (10 GPs behind) idle=f66/1/0 softirq=1157598/1157598 fqs=0
[ 6862.846963] 6-...: (14 GPs behind) idle=8dc/0/0 softirq=1263636/1263636 fqs=0
[ 6862.846965] 7-...: (14 GPs behind) idle=202/1/0 softirq=1156389/1156389 fqs=0
[ 6862.846966] 8-...: (14 GPs behind) idle=212/1/0 softirq=1136417/1136417 fqs=0
[ 6862.846968] 9-...: (136 GPs behind) idle=dea/1/0 softirq=1055775/1055775 fqs=0
[ 6862.846969] 10-...: (47 GPs behind) idle=b7a/1/0 softirq=1132407/1132407 fqs=0
[ 6862.846970] 11-...: (15 GPs behind) idle=efe/1/0 softirq=1283226/1283226 fqs=0
[ 6862.846970] (detected by 1, t=342825 jiffies, g=429028, c=429027, q=0)
[ 6862.846975] Sending NMI from CPU 1 to CPUs 0:
[ 6862.847001] NMI backtrace for cpu 0
[ 6862.847020] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6862.847021] task: ffffffff82212480 task.stack: ffffffff82200000
[ 6862.847025] RIP: 0010:blkif_copy_from_grant+0x21/0x70 [xen_blkfront]
[ 6862.847026] RSP: 0000:ffff88035d803d90 EFLAGS: 00000006
[ 6862.847027] RAX: ffffea0006b99380 RBX: ffff88035d803e58 RCX: ffff88035d803e58
[ 6862.847027] RDX: 0000000000000017 RSI: 0000000000000000 RDI: 0000000000024d1a
[ 6862.847028] RBP: 0000000000001000 R08: 0000000000001000 R09: 0000000000001000
[ 6862.847029] R10: 0000000000000040 R11: ffff88035b57b000 R12: 0000000000001000
[ 6862.847029] R13: ffffffffc00027e0 R14: 0000000000024d1b R15: 0000000000001000
[ 6862.847031] FS: 0000000000000000(0000) GS:ffff88035d800000(0000) knlGS:0000000000000000
[ 6862.847032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6862.847032] CR2: 00000000010d6640 CR3: 000000000220a004 CR4: 00000000003606f0
[ 6862.847034] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6862.847035] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6862.847035] Call Trace:
[ 6862.847037] <IRQ>
[ 6862.847041] gnttab_foreach_grant_in_range+0x66/0x100
[ 6862.847045] blkif_interrupt+0x2f7/0xd70 [xen_blkfront]
[ 6862.847047] __handle_irq_event_percpu+0x7b/0x190
[ 6862.847049] handle_irq_event_percpu+0x30/0x70
[ 6862.847051] handle_irq_event+0x36/0x60
[ 6862.847052] handle_edge_irq+0x73/0x190
[ 6862.847053] generic_handle_irq+0x24/0x30
[ 6862.847055] __evtchn_fifo_handle_events+0x187/0x1b0
[ 6862.847056] __xen_evtchn_do_upcall+0x42/0x80
[ 6862.847059] xen_evtchn_do_upcall+0x27/0x40
[ 6862.847061] xen_hvm_callback_vector+0x87/0x90
[ 6862.847062] </IRQ>
[ 6862.847063] RIP: 0010:native_safe_halt+0x2/0x10
[ 6862.847063] RSP: 0000:ffffffff82203e88 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[ 6862.847064] RAX: ffffffff8186b8c0 RBX: ffffffff82212480 RCX: 0000000000000000
[ 6862.847065] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 6862.847065] RBP: 0000000000000000 R08: 000000001bc5621b R09: ffffffff82974398
[ 6862.847066] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82212480
[ 6862.847066] R13: ffffffff82212480 R14: 0000000000000000 R15: 0000000000000000
[ 6862.847068] ? __sched_text_end+0x3/0x3
[ 6862.847071] default_idle+0x1a/0xf0
[ 6862.847073] do_idle+0x166/0x1d0
[ 6862.847075] cpu_startup_entry+0x6f/0x80
[ 6862.847077] start_kernel+0x682/0x6c3
[ 6862.847079] secondary_startup_64+0xa5/0xb0
[ 6862.847081] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 01 41 54 41 89 d4 8b 51 08 55 53 48 89 cb 48 8b 40 78 48 8b 04 d0 48 8b 40 08 <65> 48 8b 2c 25 40 5c 01 00 83 85 a0 19 00 00 01 48 2b 05 48 52
[ 6862.847546] 5-...: (10 GPs behind) idle=f66/1/0 softirq=1157598/1157598 fqs=0
[ 6862.847559] (t=342826 jiffies g=429028 c=429027 q=6)
[ 6862.847569] rcu_sched kthread starved for 342826 jiffies! g429028 c429027 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
[ 6862.847582] rcu_sched R running task 0 8 2 0x80000000
[ 6862.847595] Call Trace:
[ 6862.847601] ? __schedule+0x3dc/0x870
[ 6862.847607] schedule+0x32/0x80
[ 6862.847613] schedule_timeout+0x164/0x380
[ 6862.847619] ? call_timer_fn+0x130/0x130
[ 6862.847625] ? rcu_gp_kthread+0x577/0x980
[ 6862.847631] rcu_gp_kthread+0x577/0x980
[ 6862.847637] kthread+0xff/0x130
[ 6862.847643] ? _synchronize_rcu_expedited.constprop.65+0x370/0x370
[ 6862.847651] ? kthread_create_on_node+0x70/0x70
[ 6862.847658] ret_from_fork+0x35/0x40
[ 6862.847998] Sending NMI from CPU 1 to CPUs 2:
[ 6862.848023] NMI backtrace for cpu 2 skipped: idling at pc 0xffffffff8186bbe2
[ 6862.849023] Sending NMI from CPU 1 to CPUs 3:
[ 6862.849058] NMI backtrace for cpu 3
[ 6862.849072] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6862.849072] task: ffff88035ba85c40 task.stack: ffffc9000190c000
[ 6862.849074] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6862.849075] RSP: 0018:ffff88035d8c3e60 EFLAGS: 00000002
[ 6862.849076] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000017
[ 6862.849076] RDX: 0000000000000000 RSI: ffff88035d8c3e70 RDI: 0000000000000003
[ 6862.849077] RBP: ffff88035d8e1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6862.849077] R10: 0000000000000040 R11: ffff88035bbd9200 R12: ffff88035da21a40
[ 6862.849078] R13: 0000000000000001 R14: 0000000000100000 R15: ffff88035d8e1a84
[ 6862.849078] FS: 0000000000000000(0000) GS:ffff88035d8c0000(0000) knlGS:0000000000000000
[ 6862.849079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6862.849079] CR2: 00007c81d1d743b0 CR3: 000000000220a004 CR4: 00000000003606e0
[ 6862.849081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6862.849081] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6862.849081] Call Trace:
[ 6862.849082] <IRQ>
[ 6862.849084] ? xen_poll_irq_timeout+0x50/0x80
[ 6862.849085] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6862.849087] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6862.849088] ? rcu_process_callbacks+0x384/0xa40
[ 6862.849089] ? rebalance_domains+0x253/0x2b0
[ 6862.849090] ? __do_softirq+0x112/0x2b4
[ 6862.849092] ? irq_exit+0xee/0x100
[ 6862.849093] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6862.849094] ? xen_hvm_callback_vector+0x87/0x90
[ 6862.849094] </IRQ>
[ 6862.849095] ? __sched_text_end+0x3/0x3
[ 6862.849096] ? native_safe_halt+0x2/0x10
[ 6862.849097] ? default_idle+0x1a/0xf0
[ 6862.849098] ? do_idle+0x166/0x1d0
[ 6862.849099] ? cpu_startup_entry+0x6f/0x80
[ 6862.849100] ? start_secondary+0x1ae/0x200
[ 6862.849101] ? secondary_startup_64+0xa5/0xb0
[ 6862.849101] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6862.850051] Sending NMI from CPU 1 to CPUs 4:
[ 6863.047982] NMI backtrace for cpu 4
[ 6863.047984] CPU: 4 PID: 6264 Comm: top Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.047985] task: ffff8802d1151ec0 task.stack: ffffc9000452c000
[ 6863.047987] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.047988] RSP: 0000:ffffc9000452f780 EFLAGS: 00000202
[ 6863.047988] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000001d
[ 6863.047989] RDX: 0000000000000000 RSI: ffffc9000452f790 RDI: 0000000000000003
[ 6863.047989] RBP: ffff88035d921a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.047990] R10: 0000000000000040 R11: ffff88035b42c200 R12: ffff88035d8a1a40
[ 6863.047990] R13: 0000000000000001 R14: 0000000000140000 R15: ffff88035d921a84
[ 6863.047991] FS: 000079f4d5048900(0000) GS:ffff88035d900000(0000) knlGS:0000000000000000
[ 6863.047991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.047992] CR2: 000079f4d409eab0 CR3: 00000001aa87e001 CR4: 00000000003606e0
[ 6863.047993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.047995] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.047995] Call Trace:
[ 6863.047998] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.047999] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.048003] ? _raw_spin_lock+0x1d/0x20
[ 6863.048005] ? list_lru_count_one+0x28/0x70
[ 6863.048007] ? super_cache_count+0x7d/0xf0
[ 6863.048008] ? shrink_slab.part.50+0x12f/0x400
[ 6863.048011] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.048013] ? shrink_node+0x10e/0x320
[ 6863.048015] ? do_try_to_free_pages+0xe3/0x350
[ 6863.048016] ? try_to_free_pages+0x107/0x1d0
[ 6863.048018] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.048019] ? __switch_to_asm+0x34/0x70
[ 6863.048020] ? __switch_to_asm+0x40/0x70
[ 6863.048021] ? __switch_to_asm+0x40/0x70
[ 6863.048022] ? __switch_to_asm+0x40/0x70
[ 6863.048023] ? __switch_to_asm+0x40/0x70
[ 6863.048024] ? __switch_to_asm+0x40/0x70
[ 6863.048024] ? __switch_to_asm+0x40/0x70
[ 6863.048025] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.048027] ? __do_page_cache_readahead+0x129/0x340
[ 6863.048028] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.048029] ? find_get_entry+0x19/0x170
[ 6863.048030] ? filemap_fault+0x6b3/0x860
[ 6863.048030] ? filemap_fault+0x6b3/0x860
[ 6863.048031] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.048032] ? alloc_set_pte+0x343/0x520
[ 6863.048033] ? filemap_map_pages+0x2d5/0x510
[ 6863.048035] ? ext4_filemap_fault+0x2c/0x40
[ 6863.048036] ? __do_fault+0x1e/0xb5
[ 6863.048036] ? __handle_mm_fault+0xc40/0xda0
[ 6863.048038] ? handle_mm_fault+0xc6/0x1b0
[ 6863.048039] ? __do_page_fault+0x255/0x4f0
[ 6863.048040] ? do_page_fault+0x33/0x120
[ 6863.048041] ? page_fault+0x2f/0x50
[ 6863.048042] ? page_fault+0x45/0x50
[ 6863.048042] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.048980] Sending NMI from CPU 1 to CPUs 5:
[ 6863.049010] NMI backtrace for cpu 5
[ 6863.049012] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.049013] task: ffff88035ba83d80 task.stack: ffffc9000191c000
[ 6863.049015] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.049015] RSP: 0000:ffff88035d943cb0 EFLAGS: 00000002
[ 6863.049016] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000023
[ 6863.049017] RDX: 0000000000000000 RSI: ffff88035d943cc0 RDI: 0000000000000003
[ 6863.049018] RBP: ffff88035d961a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.049018] R10: 0000000000000040 R11: ffff88035b47d800 R12: ffff88035d8e1a40
[ 6863.049019] R13: 0000000000000001 R14: 0000000000180000 R15: ffff88035d961a84
[ 6863.049020] FS: 0000000000000000(0000) GS:ffff88035d940000(0000) knlGS:0000000000000000
[ 6863.049021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.049021] CR2: 00000000005d37c0 CR3: 000000000220a005 CR4: 00000000003606e0
[ 6863.049023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.049023] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.049024] Call Trace:
[ 6863.049024] <IRQ>
[ 6863.049026] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.049027] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.049028] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.049029] ? rcu_dump_cpu_stacks+0x44/0xc9
[ 6863.049030] ? rcu_check_callbacks+0x842/0x990
[ 6863.049031] ? irqtime_account_process_tick.isra.7+0xed/0x180
[ 6863.049032] ? tick_sched_do_timer+0x60/0x60
[ 6863.049033] ? update_process_times+0x28/0x50
[ 6863.049034] ? tick_sched_handle+0x23/0x60
[ 6863.049035] ? tick_sched_timer+0x34/0x70
[ 6863.049035] ? __hrtimer_run_queues+0xde/0x230
[ 6863.049036] ? hrtimer_interrupt+0xa6/0x1f0
[ 6863.049037] ? xen_timer_interrupt+0x1e/0x30
[ 6863.049038] ? __handle_irq_event_percpu+0x7b/0x190
[ 6863.049039] ? handle_irq_event_percpu+0x30/0x70
[ 6863.049040] ? handle_percpu_irq+0x37/0x50
[ 6863.049043] ? generic_handle_irq+0x24/0x30
[ 6863.049044] ? __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.049045] ? __xen_evtchn_do_upcall+0x42/0x80
[ 6863.049046] ? xen_evtchn_do_upcall+0x27/0x40
[ 6863.049047] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.049047] </IRQ>
[ 6863.049049] ? __sched_text_end+0x3/0x3
[ 6863.049050] ? native_safe_halt+0x2/0x10
[ 6863.049050] ? default_idle+0x1a/0xf0
[ 6863.049051] ? do_idle+0x166/0x1d0
[ 6863.049052] ? cpu_startup_entry+0x6f/0x80
[ 6863.049053] ? start_secondary+0x1ae/0x200
[ 6863.049054] ? secondary_startup_64+0xa5/0xb0
[ 6863.049055] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.050000] Sending NMI from CPU 1 to CPUs 6:
[ 6863.248957] NMI backtrace for cpu 6
[ 6863.248958] CPU: 6 PID: 4295 Comm: rustc Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.248959] task: ffff88019fbbdc40 task.stack: ffffc90006390000
[ 6863.248960] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.248961] RSP: 0000:ffffc90006393780 EFLAGS: 00000202
[ 6863.248961] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000029
[ 6863.248962] RDX: 0000000000000000 RSI: ffffc90006393790 RDI: 0000000000000003
[ 6863.248962] RBP: ffff88035d9a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.248962] R10: 0000000000000040 R11: ffff88035b4cf800 R12: 0000000000000001
[ 6863.248963] R13: 0000000000000100 R14: 00000000001c0000 R15: 0000000000000000
[ 6863.248963] FS: 00007ed99a4a1700(0000) GS:ffff88035d980000(0000) knlGS:0000000000000000
[ 6863.248964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.248964] CR2: 00007ed9a2119e50 CR3: 00000001516c6006 CR4: 00000000003606e0
[ 6863.248966] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.248966] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.248966] Call Trace:
[ 6863.248968] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.248969] ? __pv_queued_spin_lock_slowpath+0x22c/0x260
[ 6863.248971] ? _raw_spin_lock+0x1d/0x20
[ 6863.248972] ? list_lru_count_one+0x28/0x70
[ 6863.248973] ? super_cache_count+0x7d/0xf0
[ 6863.248974] ? shrink_slab.part.50+0x12f/0x400
[ 6863.248976] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.248977] ? shrink_node+0x10e/0x320
[ 6863.248978] ? do_try_to_free_pages+0xe3/0x350
[ 6863.248979] ? try_to_free_pages+0x107/0x1d0
[ 6863.248980] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.248981] ? __switch_to_asm+0x34/0x70
[ 6863.248982] ? __switch_to_asm+0x40/0x70
[ 6863.248983] ? __switch_to_asm+0x40/0x70
[ 6863.248983] ? __switch_to_asm+0x40/0x70
[ 6863.248984] ? __switch_to_asm+0x40/0x70
[ 6863.248985] ? __switch_to_asm+0x40/0x70
[ 6863.248986] ? __switch_to_asm+0x40/0x70
[ 6863.248987] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.248988] ? __do_page_cache_readahead+0x129/0x340
[ 6863.248993] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.248996] ? find_get_entry+0x19/0x170
[ 6863.248996] ? filemap_fault+0x6b3/0x860
[ 6863.248997] ? filemap_fault+0x6b3/0x860
[ 6863.248999] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.248999] ? alloc_set_pte+0x343/0x520
[ 6863.249004] ? filemap_map_pages+0x2d5/0x510
[ 6863.249006] ? ext4_filemap_fault+0x2c/0x40
[ 6863.249008] ? __do_fault+0x1e/0xb5
[ 6863.249009] ? __handle_mm_fault+0xc40/0xda0
[ 6863.249010] ? __switch_to_asm+0x34/0x70
[ 6863.249012] ? handle_mm_fault+0xc6/0x1b0
[ 6863.249014] ? __do_page_fault+0x255/0x4f0
[ 6863.249015] ? do_page_fault+0x33/0x120
[ 6863.249016] ? page_fault+0x2f/0x50
[ 6863.249018] ? page_fault+0x45/0x50
[ 6863.249018] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.249953] Sending NMI from CPU 1 to CPUs 7:
[ 6863.249977] NMI backtrace for cpu 7
[ 6863.249980] CPU: 7 PID: 266 Comm: systemd-journal Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.249981] task: ffff88033eba5c40 task.stack: ffffc90001f38000
[ 6863.249982] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.249983] RSP: 0000:ffffc90001f3b780 EFLAGS: 00000202
[ 6863.249983] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000002f
[ 6863.249984] RDX: 0000000000000000 RSI: ffffc90001f3b790 RDI: 0000000000000003
[ 6863.249984] RBP: ffff88035d9e1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.249984] R10: 0000000000000040 R11: ffff88035b52fe00 R12: ffff88035d821a40
[ 6863.249985] R13: 0000000000000001 R14: 0000000000200000 R15: ffff88035d9e1a84
[ 6863.249985] FS: 00007db8406e2940(0000) GS:ffff88035d9c0000(0000) knlGS:0000000000000000
[ 6863.249986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.249986] CR2: 00007db8401c7ab0 CR3: 0000000039bcc004 CR4: 00000000003606e0
[ 6863.249988] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.249988] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.249988] Call Trace:
[ 6863.249990] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.249991] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.249992] ? _raw_spin_lock+0x1d/0x20
[ 6863.249993] ? list_lru_count_one+0x28/0x70
[ 6863.249994] ? super_cache_count+0x7d/0xf0
[ 6863.249995] ? shrink_slab.part.50+0x12f/0x400
[ 6863.249996] ? shrink_node+0x10e/0x320
[ 6863.249997] ? do_try_to_free_pages+0xe3/0x350
[ 6863.249998] ? try_to_free_pages+0x107/0x1d0
[ 6863.250002] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.250003] ? __switch_to_asm+0x34/0x70
[ 6863.250005] ? __switch_to_asm+0x40/0x70
[ 6863.250006] ? __switch_to_asm+0x40/0x70
[ 6863.250007] ? __switch_to_asm+0x40/0x70
[ 6863.250009] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.250011] ? __do_page_cache_readahead+0x129/0x340
[ 6863.250012] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.250014] ? find_get_entry+0x19/0x170
[ 6863.250015] ? filemap_fault+0x6b3/0x860
[ 6863.250016] ? filemap_fault+0x6b3/0x860
[ 6863.250017] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.250018] ? alloc_set_pte+0x343/0x520
[ 6863.250020] ? filemap_map_pages+0x209/0x510
[ 6863.250021] ? ext4_filemap_fault+0x2c/0x40
[ 6863.250023] ? __do_fault+0x1e/0xb5
[ 6863.250024] ? __handle_mm_fault+0xc40/0xda0
[ 6863.250025] ? handle_mm_fault+0xc6/0x1b0
[ 6863.250027] ? __do_page_fault+0x255/0x4f0
[ 6863.250028] ? do_page_fault+0x33/0x120
[ 6863.250030] ? page_fault+0x2f/0x50
[ 6863.250031] ? page_fault+0x45/0x50
[ 6863.250032] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.250973] Sending NMI from CPU 1 to CPUs 8:
[ 6863.250993] NMI backtrace for cpu 8
[ 6863.251008] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.251009] task: ffff88035ba95c40 task.stack: ffffc90001934000
[ 6863.251011] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.251011] RSP: 0000:ffff88035da03e60 EFLAGS: 00000002
[ 6863.251012] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000035
[ 6863.251013] RDX: 0000000000000000 RSI: ffff88035da03e70 RDI: 0000000000000003
[ 6863.251013] RBP: ffff88035da21a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.251014] R10: 0000000000000040 R11: ffff88035b57b600 R12: 0000000000000001
[ 6863.251014] R13: 0000000000000100 R14: 0000000000240000 R15: 0000000000000000
[ 6863.251015] FS: 0000000000000000(0000) GS:ffff88035da00000(0000) knlGS:0000000000000000
[ 6863.251016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.251017] CR2: 00007f1cf2ac0510 CR3: 000000000220a005 CR4: 00000000003606e0
[ 6863.251018] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.251019] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.251019] Call Trace:
[ 6863.251020] <IRQ>
[ 6863.251025] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.251026] ? __pv_queued_spin_lock_slowpath+0x22c/0x260
[ 6863.251028] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.251029] ? rcu_process_callbacks+0x384/0xa40
[ 6863.251030] ? rebalance_domains+0x253/0x2b0
[ 6863.251032] ? __do_softirq+0x112/0x2b4
[ 6863.251033] ? irq_exit+0xee/0x100
[ 6863.251035] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6863.251036] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.251036] </IRQ>
[ 6863.251039] ? __sched_text_end+0x3/0x3
[ 6863.251040] ? native_safe_halt+0x2/0x10
[ 6863.251041] ? default_idle+0x1a/0xf0
[ 6863.251042] ? do_idle+0x166/0x1d0
[ 6863.251044] ? cpu_startup_entry+0x6f/0x80
[ 6863.251048] ? start_secondary+0x1ae/0x200
[ 6863.251049] ? secondary_startup_64+0xa5/0xb0
[ 6863.251050] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.251985] Sending NMI from CPU 1 to CPUs 9:
[ 6863.252003] NMI backtrace for cpu 9
[ 6863.252004] CPU: 9 PID: 435 Comm: systemd-logind Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.252005] task: ffff88033cfc5c40 task.stack: ffffc90001f58000
[ 6863.252006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x207/0x260
[ 6863.252006] RSP: 0000:ffffc90001f5b7c0 EFLAGS: 00000206
[ 6863.252007] RAX: 0000000000000000 RBX: ffff880039271280 RCX: 000000000000003b
[ 6863.252008] RDX: 0000000000001839 RSI: 0000000000000000 RDI: 0000000000000006
[ 6863.252008] RBP: ffff88035da61a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.252009] R10: 0000000000000040 R11: ffff88035b5f0400 R12: ffff88035d9a1a40
[ 6863.252010] R13: 0000000000000001 R14: 0000000000280000 R15: ffff88035da61a84
[ 6863.252011] FS: 0000717a785909c0(0000) GS:ffff88035da40000(0000) knlGS:0000000000000000
[ 6863.252011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.252012] CR2: 0000717a7803b6c0 CR3: 0000000034d76004 CR4: 00000000003606e0
[ 6863.252014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.252014] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.252014] Call Trace:
[ 6863.252017] _raw_spin_lock+0x1d/0x20
[ 6863.252018] list_lru_count_one+0x28/0x70
[ 6863.252020] super_cache_count+0x7d/0xf0
[ 6863.252021] shrink_slab.part.50+0x12f/0x400
[ 6863.252024] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.252025] shrink_node+0x10e/0x320
[ 6863.252027] do_try_to_free_pages+0xe3/0x350
[ 6863.252029] try_to_free_pages+0x107/0x1d0
[ 6863.252031] __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.252032] ? __switch_to_asm+0x34/0x70
[ 6863.252033] ? __switch_to_asm+0x40/0x70
[ 6863.252034] ? __switch_to_asm+0x40/0x70
[ 6863.252035] ? __switch_to_asm+0x40/0x70
[ 6863.252037] __alloc_pages_nodemask+0x270/0x2b0
[ 6863.252039] __do_page_cache_readahead+0x129/0x340
[ 6863.252041] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.252042] ? find_get_entry+0x19/0x170
[ 6863.252043] ? filemap_fault+0x6b3/0x860
[ 6863.252044] filemap_fault+0x6b3/0x860
[ 6863.252045] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.252046] ? alloc_set_pte+0x343/0x520
[ 6863.252048] ? filemap_map_pages+0x2d5/0x510
[ 6863.252050] ext4_filemap_fault+0x2c/0x40
[ 6863.252051] __do_fault+0x1e/0xb5
[ 6863.252052] __handle_mm_fault+0xc40/0xda0
[ 6863.252054] ? hrtimer_start_range_ns+0x18d/0x340
[ 6863.252055] handle_mm_fault+0xc6/0x1b0
[ 6863.252057] __do_page_fault+0x255/0x4f0
[ 6863.252058] do_page_fault+0x33/0x120
[ 6863.252060] ? page_fault+0x2f/0x50
[ 6863.252061] page_fault+0x45/0x50
[ 6863.252062] RIP: 7808db31:0x717a78384cc0
[ 6863.252063] RSP: 825f5aa0:00005eb4825e9e80 EFLAGS: 717a780b8f40
[ 6863.252063] Code: 08 85 c0 74 f7 4c 8b 7d 00 4d 85 ff 0f 84 af fe ff ff 41 0f 0d 0f e9 a6 fe ff ff 49 63 7c 24 40 ff 14 25 20 69 22 82 84 c0 75 a1 <f3> 90 83 ea 01 75 85 eb 98 48 83 3c 24 00 c6 43 01 00 74 18 c6
[ 6863.252999] Sending NMI from CPU 1 to CPUs 10:
[ 6863.253023] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.254011] Sending NMI from CPU 1 to CPUs 11:
[ 6863.254035] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.255028] rcu_sched kthread starved for 343233 jiffies! g429028 c429027 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=3
[ 6863.255029] rcu_sched R running task 0 8 2 0x80000000
[ 6863.255030] Call Trace:
[ 6863.255032] ? __schedule+0x3dc/0x870
[ 6863.255033] schedule+0x32/0x80
[ 6863.255034] schedule_timeout+0x164/0x380
[ 6863.255036] ? call_timer_fn+0x130/0x130
[ 6863.255037] ? rcu_gp_kthread+0x577/0x980
[ 6863.255037] rcu_gp_kthread+0x577/0x980
[ 6863.255039] kthread+0xff/0x130
[ 6863.255040] ? _synchronize_rcu_expedited.constprop.65+0x370/0x370
[ 6863.255041] ? kthread_create_on_node+0x70/0x70
[ 6863.255042] ret_from_fork+0x35/0x40
[ 6863.255054] Sending NMI from CPU 5 to CPUs 0:
[ 6863.255089] NMI backtrace for cpu 0
[ 6863.255090] CPU: 0 PID: 640 Comm: Xorg Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.255091] task: ffff88003267dc40 task.stack: ffffc900022f4000
[ 6863.255092] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.255093] RSP: 0000:ffffc900022f7780 EFLAGS: 00000202
[ 6863.255093] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000005
[ 6863.255094] RDX: 0000000000000000 RSI: ffffc900022f7790 RDI: 0000000000000003
[ 6863.255094] RBP: ffff88035d821a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.255094] R10: 0000000000000040 R11: ffff88035d54c600 R12: ffff88035d921a40
[ 6863.255095] R13: 0000000000000001 R14: 0000000000040000 R15: ffff88035d821a84
[ 6863.255096] FS: 000074f9a54f8ac0(0000) GS:ffff88035d800000(0000) knlGS:0000000000000000
[ 6863.255096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.255096] CR2: 0000000000578ed0 CR3: 0000000032788001 CR4: 00000000003606f0
[ 6863.255098] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.255098] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.255098] Call Trace:
[ 6863.255100] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.255101] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.255102] ? _raw_spin_lock+0x1d/0x20
[ 6863.255103] ? list_lru_count_one+0x28/0x70
[ 6863.255118] ? super_cache_count+0x7d/0xf0
[ 6863.255118] ? shrink_slab.part.50+0x12f/0x400
[ 6863.255120] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.255121] ? shrink_node+0x10e/0x320
[ 6863.255122] ? do_try_to_free_pages+0xe3/0x350
[ 6863.255123] ? try_to_free_pages+0x107/0x1d0
[ 6863.255125] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.255126] ? __switch_to_asm+0x34/0x70
[ 6863.255127] ? __switch_to_asm+0x40/0x70
[ 6863.255128] ? __switch_to_asm+0x40/0x70
[ 6863.255129] ? __switch_to_asm+0x40/0x70
[ 6863.255131] ? __switch_to_asm+0x40/0x70
[ 6863.255132] ? __switch_to_asm+0x40/0x70
[ 6863.255133] ? __switch_to_asm+0x40/0x70
[ 6863.255134] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.255135] ? __do_page_cache_readahead+0x129/0x340
[ 6863.255137] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.255138] ? find_get_entry+0x19/0x170
[ 6863.255139] ? filemap_fault+0x6b3/0x860
[ 6863.255140] ? filemap_fault+0x6b3/0x860
[ 6863.255141] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.255142] ? alloc_set_pte+0x343/0x520
[ 6863.255143] ? filemap_map_pages+0x209/0x510
[ 6863.255145] ? ext4_filemap_fault+0x2c/0x40
[ 6863.255146] ? __do_fault+0x1e/0xb5
[ 6863.255147] ? __handle_mm_fault+0xc40/0xda0
[ 6863.255148] ? handle_mm_fault+0xc6/0x1b0
[ 6863.255150] ? __do_page_fault+0x255/0x4f0
[ 6863.255151] ? do_page_fault+0x33/0x120
[ 6863.255152] ? page_fault+0x2f/0x50
[ 6863.255153] ? page_fault+0x45/0x50
[ 6863.255154] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.256086] Sending NMI from CPU 5 to CPUs 2:
[ 6863.256107] NMI backtrace for cpu 2
[ 6863.256108] CPU: 2 PID: 1 Comm: systemd Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.256108] task: ffff88035ba10000 task.stack: ffffc900018a4000
[ 6863.256110] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.256111] RSP: 0000:ffffc900018a7780 EFLAGS: 00000206
[ 6863.256111] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000011
[ 6863.256112] RDX: 0000000000000000 RSI: ffffc900018a7790 RDI: 0000000000000003
[ 6863.256112] RBP: ffff88035d8a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.256113] R10: 0000000000000040 R11: ffff88035bb8ac00 R12: ffff88035d821a40
[ 6863.256113] R13: 0000000000000001 R14: 00000000000c0000 R15: ffff88035d8a1a84
[ 6863.256114] FS: 00007773dbc39e40(0000) GS:ffff88035d880000(0000) knlGS:0000000000000000
[ 6863.256114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.256115] CR2: 0000618326569e11 CR3: 00000003587f8005 CR4: 00000000003606e0
[ 6863.256116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.256116] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.256116] Call Trace:
[ 6863.256119] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.256120] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.256122] ? _raw_spin_lock+0x1d/0x20
[ 6863.256123] ? list_lru_count_one+0x28/0x70
[ 6863.256124] ? super_cache_count+0x7d/0xf0
[ 6863.256125] ? shrink_slab.part.50+0x12f/0x400
[ 6863.256126] ? shrink_node+0x10e/0x320
[ 6863.256127] ? do_try_to_free_pages+0xe3/0x350
[ 6863.256128] ? try_to_free_pages+0x107/0x1d0
[ 6863.256129] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.256130] ? __switch_to_asm+0x34/0x70
[ 6863.256131] ? __switch_to_asm+0x40/0x70
[ 6863.256132] ? __switch_to_asm+0x40/0x70
[ 6863.256133] ? __switch_to_asm+0x40/0x70
[ 6863.256134] ? __switch_to_asm+0x40/0x70
[ 6863.256135] ? __switch_to_asm+0x40/0x70
[ 6863.256135] ? __switch_to_asm+0x40/0x70
[ 6863.256136] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.256138] ? __do_page_cache_readahead+0x129/0x340
[ 6863.256139] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.256140] ? find_get_entry+0x19/0x170
[ 6863.256141] ? filemap_fault+0x6b3/0x860
[ 6863.256141] ? filemap_fault+0x6b3/0x860
[ 6863.256142] ? page_add_file_rmap+0x18/0x1f0
[ 6863.256143] ? alloc_set_pte+0x343/0x520
[ 6863.256144] ? filemap_map_pages+0x2d5/0x510
[ 6863.256145] ? ext4_filemap_fault+0x2c/0x40
[ 6863.256146] ? __do_fault+0x1e/0xb5
[ 6863.256147] ? __handle_mm_fault+0xc40/0xda0
[ 6863.256148] ? timerfd_read+0xc7/0x280
[ 6863.256150] ? handle_mm_fault+0xc6/0x1b0
[ 6863.256151] ? __do_page_fault+0x255/0x4f0
[ 6863.256152] ? do_page_fault+0x33/0x120
[ 6863.256153] ? page_fault+0x2f/0x50
[ 6863.256154] ? page_fault+0x45/0x50
[ 6863.256154] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.257103] Sending NMI from CPU 5 to CPUs 4:
[ 6863.257117] NMI backtrace for cpu 4
[ 6863.257118] CPU: 4 PID: 6264 Comm: top Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.257118] task: ffff8802d1151ec0 task.stack: ffffc9000452c000
[ 6863.257119] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.257120] RSP: 0018:ffffc9000452f780 EFLAGS: 00000202
[ 6863.257120] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000001d
[ 6863.257121] RDX: 0000000000000000 RSI: ffffc9000452f790 RDI: 0000000000000003
[ 6863.257121] RBP: ffff88035d921a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.257122] R10: 0000000000000040 R11: ffff88035b42c200 R12: ffff88035d8a1a40
[ 6863.257122] R13: 0000000000000001 R14: 0000000000140000 R15: ffff88035d921a84
[ 6863.257123] FS: 000079f4d5048900(0000) GS:ffff88035d900000(0000) knlGS:0000000000000000
[ 6863.257123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.257123] CR2: 00007efb66abfb30 CR3: 00000001aa87e001 CR4: 00000000003606e0
[ 6863.257127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.257127] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.257128] Call Trace:
[ 6863.257129] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.257130] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.257131] ? _raw_spin_lock+0x1d/0x20
[ 6863.257132] ? list_lru_count_one+0x28/0x70
[ 6863.257133] ? super_cache_count+0x7d/0xf0
[ 6863.257134] ? shrink_slab.part.50+0x12f/0x400
[ 6863.257135] ? shrink_node+0x10e/0x320
[ 6863.257136] ? do_try_to_free_pages+0xe3/0x350
[ 6863.257137] ? try_to_free_pages+0x107/0x1d0
[ 6863.257138] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.257139] ? __switch_to_asm+0x34/0x70
[ 6863.257140] ? __switch_to_asm+0x40/0x70
[ 6863.257141] ? __switch_to_asm+0x40/0x70
[ 6863.257141] ? __switch_to_asm+0x40/0x70
[ 6863.257142] ? __switch_to_asm+0x40/0x70
[ 6863.257143] ? __switch_to_asm+0x40/0x70
[ 6863.257144] ? __switch_to_asm+0x40/0x70
[ 6863.257145] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.257146] ? __do_page_cache_readahead+0x129/0x340
[ 6863.257147] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.257148] ? find_get_entry+0x19/0x170
[ 6863.257149] ? filemap_fault+0x6b3/0x860
[ 6863.257150] ? filemap_fault+0x6b3/0x860
[ 6863.257151] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.257151] ? alloc_set_pte+0x343/0x520
[ 6863.257152] ? filemap_map_pages+0x2d5/0x510
[ 6863.257154] ? ext4_filemap_fault+0x2c/0x40
[ 6863.257155] ? __do_fault+0x1e/0xb5
[ 6863.257156] ? __handle_mm_fault+0xc40/0xda0
[ 6863.257157] ? handle_mm_fault+0xc6/0x1b0
[ 6863.257158] ? __do_page_fault+0x255/0x4f0
[ 6863.257159] ? do_page_fault+0x33/0x120
[ 6863.257160] ? page_fault+0x2f/0x50
[ 6863.257161] ? page_fault+0x45/0x50
[ 6863.257161] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.258115] NMI backtrace for cpu 5
[ 6863.258116] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.258117] Call Trace:
[ 6863.258118] <IRQ>
[ 6863.258119] dump_stack+0x5c/0x85
[ 6863.258120] nmi_cpu_backtrace+0xbf/0xd0
[ 6863.258122] ? irq_force_complete_move+0xe0/0xe0
[ 6863.258123] nmi_trigger_cpumask_backtrace+0x86/0xc0
[ 6863.258124] rcu_dump_cpu_stacks+0x92/0xc9
[ 6863.258125] rcu_check_callbacks+0x842/0x990
[ 6863.258126] ? irqtime_account_process_tick.isra.7+0xed/0x180
[ 6863.258128] ? tick_sched_do_timer+0x60/0x60
[ 6863.258128] update_process_times+0x28/0x50
[ 6863.258129] tick_sched_handle+0x23/0x60
[ 6863.258130] tick_sched_timer+0x34/0x70
[ 6863.258131] __hrtimer_run_queues+0xde/0x230
[ 6863.258132] hrtimer_interrupt+0xa6/0x1f0
[ 6863.258133] xen_timer_interrupt+0x1e/0x30
[ 6863.258135] __handle_irq_event_percpu+0x7b/0x190
[ 6863.258136] handle_irq_event_percpu+0x30/0x70
[ 6863.258137] handle_percpu_irq+0x37/0x50
[ 6863.258138] generic_handle_irq+0x24/0x30
[ 6863.258139] __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.258141] __xen_evtchn_do_upcall+0x42/0x80
[ 6863.258142] xen_evtchn_do_upcall+0x27/0x40
[ 6863.258143] xen_hvm_callback_vector+0x87/0x90
[ 6863.258144] </IRQ>
[ 6863.258145] RIP: 0010:native_safe_halt+0x2/0x10
[ 6863.258146] RSP: 0000:ffffc9000191feb8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[ 6863.258146] RAX: ffffffff8186b8c0 RBX: ffff88035ba83d80 RCX: 0000000000000000
[ 6863.258147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 6863.258147] RBP: 0000000000000005 R08: 000000001bba9704 R09: ffff88035d95b7e0
[ 6863.258148] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035ba83d80
[ 6863.258148] R13: ffff88035ba83d80 R14: 0000000000000000 R15: 0000000000000000
[ 6863.258149] ? __sched_text_end+0x3/0x3
[ 6863.258150] default_idle+0x1a/0xf0
[ 6863.258151] do_idle+0x166/0x1d0
[ 6863.258153] cpu_startup_entry+0x6f/0x80
[ 6863.258154] start_secondary+0x1ae/0x200
[ 6863.258159] secondary_startup_64+0xa5/0xb0
[ 6863.258162] Sending NMI from CPU 5 to CPUs 6:
[ 6863.258177] NMI backtrace for cpu 6
[ 6863.258178] CPU: 6 PID: 4295 Comm: rustc Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.258178] task: ffff88019fbbdc40 task.stack: ffffc90006390000
[ 6863.258179] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.258180] RSP: 0000:ffffc90006393780 EFLAGS: 00000202
[ 6863.258181] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000029
[ 6863.258181] RDX: 0000000000000000 RSI: ffffc90006393790 RDI: 0000000000000003
[ 6863.258181] RBP: ffff88035d9a1a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.258182] R10: 0000000000000040 R11: ffff88035b4cf800 R12: ffff88035d9e1a40
[ 6863.258182] R13: 0000000000000001 R14: 00000000001c0000 R15: ffff88035d9a1a84
[ 6863.258183] FS: 00007ed99a4a1700(0000) GS:ffff88035d980000(0000) knlGS:0000000000000000
[ 6863.258183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.258184] CR2: 00007cf603db71c0 CR3: 00000001516c6006 CR4: 00000000003606e0
[ 6863.258185] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.258185] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.258186] Call Trace:
[ 6863.258187] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.258188] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.258189] ? _raw_spin_lock+0x1d/0x20
[ 6863.258190] ? list_lru_count_one+0x28/0x70
[ 6863.258191] ? super_cache_count+0x7d/0xf0
[ 6863.258192] ? shrink_slab.part.50+0x12f/0x400
[ 6863.258194] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.258195] ? shrink_node+0x10e/0x320
[ 6863.258196] ? do_try_to_free_pages+0xe3/0x350
[ 6863.258197] ? try_to_free_pages+0x107/0x1d0
[ 6863.258198] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.258199] ? __switch_to_asm+0x34/0x70
[ 6863.258199] ? __switch_to_asm+0x40/0x70
[ 6863.258200] ? __switch_to_asm+0x40/0x70
[ 6863.258201] ? __switch_to_asm+0x40/0x70
[ 6863.258202] ? __switch_to_asm+0x40/0x70
[ 6863.258203] ? __switch_to_asm+0x40/0x70
[ 6863.258204] ? __switch_to_asm+0x40/0x70
[ 6863.258205] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.258206] ? __do_page_cache_readahead+0x129/0x340
[ 6863.258207] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.258208] ? find_get_entry+0x19/0x170
[ 6863.258209] ? filemap_fault+0x6b3/0x860
[ 6863.258209] ? filemap_fault+0x6b3/0x860
[ 6863.258210] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.258211] ? alloc_set_pte+0x343/0x520
[ 6863.258212] ? filemap_map_pages+0x2d5/0x510
[ 6863.258214] ? ext4_filemap_fault+0x2c/0x40
[ 6863.258215] ? __do_fault+0x1e/0xb5
[ 6863.258215] ? __handle_mm_fault+0xc40/0xda0
[ 6863.258216] ? __switch_to_asm+0x34/0x70
[ 6863.258217] ? handle_mm_fault+0xc6/0x1b0
[ 6863.258218] ? __do_page_fault+0x255/0x4f0
[ 6863.258219] ? do_page_fault+0x33/0x120
[ 6863.258220] ? page_fault+0x2f/0x50
[ 6863.258221] ? page_fault+0x45/0x50
[ 6863.258222] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.259175] Sending NMI from CPU 5 to CPUs 7:
[ 6863.260188] NMI backtrace for cpu 7
[ 6863.260189] CPU: 7 PID: 266 Comm: systemd-journal Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.260189] task: ffff88033eba5c40 task.stack: ffffc90001f38000
[ 6863.260189] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.260189] RSP: 0000:ffff88035d9c3cc0 EFLAGS: 00000046
[ 6863.260190] RAX: 0000000000000000 RBX: 000000000000000c RCX: ffff88035d9c3d14
[ 6863.260190] RDX: 0000000000000800 RSI: 0000000000000000 RDI: 0000000000000000
[ 6863.260190] RBP: ffff88035d420f40 R08: ffffc90000000000 R09: 00000000000121b2
[ 6863.260191] R10: 00000000000129b6 R11: ffffffff82977ded R12: 0000000000000004
[ 6863.260191] R13: ffff88035d9c3d14 R14: 0000000000000010 R15: ffffffff82977de0
[ 6863.260193] FS: 00007db8406e2940(0000) GS:ffff88035d9c0000(0000) knlGS:0000000000000000
[ 6863.260193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.260194] CR2: 00007db8401c7ab0 CR3: 0000000039bcc004 CR4: 00000000003606e0
[ 6863.260194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.260194] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.260194] Call Trace:
[ 6863.260194] <IRQ>
[ 6863.260195] ? domU_write_console+0x12d/0x1b0
[ 6863.260195] ? hvc_console_print+0xcc/0x120
[ 6863.260195] ? console_unlock+0x3f3/0x4e0
[ 6863.260195] ? wake_up_klogd_work_func+0x56/0x60
[ 6863.260195] ? irq_work_run_list+0x49/0x70
[ 6863.260196] ? tick_sched_do_timer+0x60/0x60
[ 6863.260196] ? update_process_times+0x3b/0x50
[ 6863.260196] ? tick_sched_handle+0x23/0x60
[ 6863.260196] ? tick_sched_timer+0x34/0x70
[ 6863.260196] ? __hrtimer_run_queues+0xde/0x230
[ 6863.260197] ? hrtimer_interrupt+0xa6/0x1f0
[ 6863.260197] ? xen_timer_interrupt+0x1e/0x30
[ 6863.260197] ? __handle_irq_event_percpu+0x7b/0x190
[ 6863.260197] ? handle_irq_event_percpu+0x30/0x70
[ 6863.260197] ? handle_percpu_irq+0x37/0x50
[ 6863.260198] ? generic_handle_irq+0x24/0x30
[ 6863.260198] ? __evtchn_fifo_handle_events+0x187/0x1b0
[ 6863.260198] ? __xen_evtchn_do_upcall+0x42/0x80
[ 6863.260198] ? xen_evtchn_do_upcall+0x27/0x40
[ 6863.260198] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.260199] </IRQ>
[ 6863.260199] ? xen_hypercall_sched_op+0x8/0x20
[ 6863.260199] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.260199] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.260199] ? _raw_spin_lock+0x1d/0x20
[ 6863.260200] ? list_lru_count_one+0x28/0x70
[ 6863.260200] ? super_cache_count+0x7d/0xf0
[ 6863.260200] ? shrink_slab.part.50+0x12f/0x400
[ 6863.260200] ? shrink_node+0x10e/0x320
[ 6863.260200] ? do_try_to_free_pages+0xe3/0x350
[ 6863.260201] ? try_to_free_pages+0x107/0x1d0
[ 6863.260201] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.260201] ? __switch_to_asm+0x34/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260201] ? __switch_to_asm+0x40/0x70
[ 6863.260202] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.260202] ? __do_page_cache_readahead+0x129/0x340
[ 6863.260202] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.260202] ? find_get_entry+0x19/0x170
[ 6863.260202] ? filemap_fault+0x6b3/0x860
[ 6863.260203] ? filemap_fault+0x6b3/0x860
[ 6863.260203] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.260203] ? alloc_set_pte+0x343/0x520
[ 6863.260203] ? filemap_map_pages+0x209/0x510
[ 6863.260203] ? ext4_filemap_fault+0x2c/0x40
[ 6863.260204] ? __do_fault+0x1e/0xb5
[ 6863.260204] ? __handle_mm_fault+0xc40/0xda0
[ 6863.260204] ? handle_mm_fault+0xc6/0x1b0
[ 6863.260204] ? __do_page_fault+0x255/0x4f0
[ 6863.260204] ? do_page_fault+0x33/0x120
[ 6863.260205] ? page_fault+0x2f/0x50
[ 6863.260205] ? page_fault+0x45/0x50
[ 6863.260205] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.260212] Sending NMI from CPU 5 to CPUs 8:
[ 6863.260225] NMI backtrace for cpu 8
[ 6863.260226] CPU: 8 PID: 921 Comm: gnome-terminal- Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.260227] task: ffff88008478bd80 task.stack: ffffc90002474000
[ 6863.260228] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.260228] RSP: 0000:ffff88035da03e60 EFLAGS: 00000002
[ 6863.260229] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000035
[ 6863.260229] RDX: 0000000000000000 RSI: ffff88035da03e70 RDI: 0000000000000003
[ 6863.260230] RBP: ffff88035da21a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.260230] R10: 0000000000000040 R11: ffff88035b57b600 R12: ffff88035d8e1a40
[ 6863.260231] R13: 0000000000000001 R14: 0000000000240000 R15: ffff88035da21a84
[ 6863.260231] FS: 00007cf606eb0ac0(0000) GS:ffff88035da00000(0000) knlGS:0000000000000000
[ 6863.260234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.260234] CR2: 00007cf605f5d5bf CR3: 000000012b34a003 CR4: 00000000003606e0
[ 6863.260236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.260236] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.260236] Call Trace:
[ 6863.260237] <IRQ>
[ 6863.260238] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.260239] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.260241] ? _raw_spin_lock_irqsave+0x32/0x40
[ 6863.260241] ? rcu_process_callbacks+0x384/0xa40
[ 6863.260243] ? rebalance_domains+0x253/0x2b0
[ 6863.260244] ? __do_softirq+0x112/0x2b4
[ 6863.260245] ? irq_exit+0xee/0x100
[ 6863.260246] ? xen_evtchn_do_upcall+0x2c/0x40
[ 6863.260247] ? xen_hvm_callback_vector+0x87/0x90
[ 6863.260247] </IRQ>
[ 6863.260249] ? __raw_callee_save___pv_queued_spin_unlock+0xc/0x12
[ 6863.260250] ? list_lru_count_one+0x47/0x70
[ 6863.260251] ? super_cache_count+0x64/0xf0
[ 6863.260252] ? shrink_slab.part.50+0x12f/0x400
[ 6863.260253] ? shrink_node+0x10e/0x320
[ 6863.260254] ? do_try_to_free_pages+0xe3/0x350
[ 6863.260255] ? try_to_free_pages+0x107/0x1d0
[ 6863.260256] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.260257] ? __switch_to_asm+0x34/0x70
[ 6863.260258] ? __switch_to_asm+0x40/0x70
[ 6863.260259] ? __switch_to_asm+0x40/0x70
[ 6863.260260] ? __switch_to_asm+0x40/0x70
[ 6863.260261] ? __switch_to_asm+0x40/0x70
[ 6863.260261] ? __switch_to_asm+0x40/0x70
[ 6863.260262] ? __switch_to_asm+0x40/0x70
[ 6863.260263] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.260264] ? __do_page_cache_readahead+0x129/0x340
[ 6863.260266] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.260267] ? find_get_entry+0x19/0x170
[ 6863.260267] ? filemap_fault+0x6b3/0x860
[ 6863.260268] ? filemap_fault+0x6b3/0x860
[ 6863.260269] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.260270] ? alloc_set_pte+0x343/0x520
[ 6863.260271] ? filemap_map_pages+0x2d5/0x510
[ 6863.260272] ? ext4_filemap_fault+0x2c/0x40
[ 6863.260273] ? __do_fault+0x1e/0xb5
[ 6863.260274] ? __handle_mm_fault+0xc40/0xda0
[ 6863.260275] ? handle_mm_fault+0xc6/0x1b0
[ 6863.260276] ? __do_page_fault+0x255/0x4f0
[ 6863.260277] ? do_page_fault+0x33/0x120
[ 6863.260278] ? page_fault+0x2f/0x50
[ 6863.260279] ? page_fault+0x45/0x50
[ 6863.260279] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.261224] Sending NMI from CPU 5 to CPUs 9:
[ 6863.261238] NMI backtrace for cpu 9
[ 6863.261239] CPU: 9 PID: 435 Comm: systemd-logind Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6863.261239] task: ffff88033cfc5c40 task.stack: ffffc90001f58000
[ 6863.261240] RIP: 0010:xen_hypercall_sched_op+0x8/0x20
[ 6863.261241] RSP: 0000:ffffc90001f5b780 EFLAGS: 00000202
[ 6863.261241] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000003b
[ 6863.261241] RDX: 0000000000000000 RSI: ffffc90001f5b790 RDI: 0000000000000003
[ 6863.261242] RBP: ffff88035da61a40 R08: ffff88035d0366c8 R09: 0000000000000000
[ 6863.261242] R10: 0000000000000040 R11: ffff88035b5f0400 R12: ffff88035d861a40
[ 6863.261243] R13: 0000000000000001 R14: 0000000000280000 R15: ffff88035da61a84
[ 6863.261243] FS: 0000717a785909c0(0000) GS:ffff88035da40000(0000) knlGS:0000000000000000
[ 6863.261244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6863.261244] CR2: 0000717a7803b6c0 CR3: 0000000034d76004 CR4: 00000000003606e0
[ 6863.261245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6863.261246] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6863.261246] Call Trace:
[ 6863.261247] ? xen_poll_irq_timeout+0x50/0x80
[ 6863.261248] ? __pv_queued_spin_lock_slowpath+0x1c2/0x260
[ 6863.261249] ? _raw_spin_lock+0x1d/0x20
[ 6863.261250] ? list_lru_count_one+0x28/0x70
[ 6863.261251] ? super_cache_count+0x7d/0xf0
[ 6863.261254] ? shrink_slab.part.50+0x12f/0x400
[ 6863.261256] ? get_free_grant+0x80/0x80 [xen_blkfront]
[ 6863.261257] ? shrink_node+0x10e/0x320
[ 6863.261258] ? do_try_to_free_pages+0xe3/0x350
[ 6863.261259] ? try_to_free_pages+0x107/0x1d0
[ 6863.261260] ? __alloc_pages_slowpath+0x3bf/0xe90
[ 6863.261261] ? __switch_to_asm+0x34/0x70
[ 6863.261262] ? __switch_to_asm+0x40/0x70
[ 6863.261262] ? __switch_to_asm+0x40/0x70
[ 6863.261263] ? __switch_to_asm+0x40/0x70
[ 6863.261265] ? __alloc_pages_nodemask+0x270/0x2b0
[ 6863.261266] ? __do_page_cache_readahead+0x129/0x340
[ 6863.261267] ? radix_tree_lookup_slot+0x1e/0x50
[ 6863.261268] ? find_get_entry+0x19/0x170
[ 6863.261269] ? filemap_fault+0x6b3/0x860
[ 6863.261269] ? filemap_fault+0x6b3/0x860
[ 6863.261270] ? page_add_file_rmap+0xbf/0x1f0
[ 6863.261271] ? alloc_set_pte+0x343/0x520
[ 6863.261272] ? filemap_map_pages+0x2d5/0x510
[ 6863.261273] ? ext4_filemap_fault+0x2c/0x40
[ 6863.261274] ? __do_fault+0x1e/0xb5
[ 6863.261275] ? __handle_mm_fault+0xc40/0xda0
[ 6863.261276] ? hrtimer_start_range_ns+0x18d/0x340
[ 6863.261277] ? handle_mm_fault+0xc6/0x1b0
[ 6863.261278] ? __do_page_fault+0x255/0x4f0
[ 6863.261279] ? do_page_fault+0x33/0x120
[ 6863.261280] ? page_fault+0x2f/0x50
[ 6863.261281] ? page_fault+0x45/0x50
[ 6863.261281] Code: cc cc cc b8 1c 00 00 00 0f 01 c1 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc b8 1d 00 00 00 0f 01 c1 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[ 6863.262236] Sending NMI from CPU 5 to CPUs 10:
[ 6863.262250] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffff8186bbe2
[ 6863.263248] Sending NMI from CPU 5 to CPUs 11:
[ 6863.263266] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffff8186bbe2
[ 6865.272330] audit: type=1701 audit(1534438836.696:170): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=435 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" sig=6 res=1
[ 6870.370170] cc1plus invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[ 6870.370197] cc1plus cpuset=/ mems_allowed=0
[ 6870.370209] CPU: 8 PID: 5092 Comm: cc1plus Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 6870.370225] Call Trace:
[ 6870.370236] dump_stack+0x5c/0x85
[ 6870.370248] dump_header+0x94/0x21c
[ 6870.370258] ? get_page_from_freelist+0x184/0xbd0
[ 6870.370270] oom_kill_process+0x212/0x410
[ 6870.370279] out_of_memory+0x131/0x4b0
[ 6870.370289] __alloc_pages_slowpath+0xda8/0xe90
[ 6870.370301] __alloc_pages_nodemask+0x270/0x2b0
[ 6870.370312] filemap_fault+0x54c/0x860
[ 6870.370322] ? alloc_set_pte+0x343/0x520
[ 6870.370332] ? filemap_map_pages+0x2d5/0x510
[ 6870.370345] ext4_filemap_fault+0x2c/0x40
[ 6870.370355] __do_fault+0x1e/0xb5
[ 6870.370364] __handle_mm_fault+0xc40/0xda0
[ 6870.370374] ? __switch_to_xtra+0x354/0x360
[ 6870.370384] handle_mm_fault+0xc6/0x1b0
[ 6870.370394] __do_page_fault+0x255/0x4f0
[ 6870.370404] do_page_fault+0x33/0x120
[ 6870.370414] ? page_fault+0x2f/0x50
[ 6870.370423] page_fault+0x45/0x50
[ 6870.370433] RIP: ce647960:0x79a0e57cb3f0
[ 6870.370441] RSP: ea817000:00007ffeb55d1f40 EFLAGS: 000000e4
[ 6870.370442] Mem-Info:
[ 6870.370461] active_anon:3357705 inactive_anon:24251 isolated_anon:0
[ 6870.370461] active_file:104 inactive_file:2 isolated_file:0
[ 6870.370461] unevictable:18392 dirty:0 writeback:2 unstable:0
[ 6870.370461] slab_reclaimable:10221 slab_unreclaimable:15302
[ 6870.370461] mapped:1245 shmem:1275 pagetables:10643 bounce:0
[ 6870.370461] free:30157 free_pcp:3654 free_cma:0
[ 6870.370523] Node 0 active_anon:13430820kB inactive_anon:97004kB active_file:416kB inactive_file:8kB unevictable:73568kB isolated(anon):0kB isolated(file):0kB mapped:4980kB dirty:0kB writeback:8kB shmem:5100kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 6870.370569] Node 0 DMA free:15908kB min:76kB low:92kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 6870.370621] lowmem_reserve[]: 0 3956 13655 13655 13655
[ 6870.370634] Node 0 DMA32 free:57708kB min:19556kB low:24444kB high:29332kB active_anon:3865036kB inactive_anon:39792kB active_file:620kB inactive_file:632kB unevictable:32436kB writepending:8kB present:4159452kB managed:4071632kB mlocked:32436kB kernel_stack:2512kB pagetables:15532kB bounce:0kB free_pcp:7576kB local_pcp:532kB free_cma:0kB
[ 6870.370686] lowmem_reserve[]: 0 0 9699 9699 9699
[ 6870.370699] Node 0 Normal free:47012kB min:47948kB low:59932kB high:71916kB active_anon:9565784kB inactive_anon:57212kB active_file:1692kB inactive_file:708kB unevictable:41132kB writepending:0kB present:10160128kB managed:9915580kB mlocked:41132kB kernel_stack:3552kB pagetables:27040kB bounce:0kB free_pcp:7040kB local_pcp:736kB free_cma:0kB
[ 6870.370753] lowmem_reserve[]: 0 0 0 0 0
[ 6870.370762] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 6870.370798] Node 0 DMA32: 662*4kB (UME) 256*8kB (UME) 314*16kB (UME) 331*32kB (UME) 181*64kB (UME) 89*128kB (UE) 32*256kB (UME) 13*512kB (UME) 1*1024kB (E) 0*2048kB 0*4096kB = 59160kB
[ 6870.370834] Node 0 Normal: 461*4kB (UME) 28*8kB (UME) 55*16kB (UM) 1385*32kB (UME) 5*64kB (M) 5*128kB (M) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48996kB
[ 6870.370872] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 6870.370894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 6870.370910] 1946 total pagecache pages
[ 6870.370918] 0 pages in swap cache
[ 6870.370926] Swap cache stats: add 3510, delete 3510, find 44/106
[ 6870.370937] Free swap = 0kB
[ 6870.370945] Total swap = 0kB
[ 6870.370952] 3583894 pages RAM
[ 6870.370959] 0 pages HighMem/MovableOnly
[ 6870.370966] 83114 pages reserved
[ 6870.370974] 0 pages cma reserved
[ 6870.370981] 0 pages hwpoisoned
[ 6870.370988] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 6870.371035] [ 266] 0 266 27000 204 45 4 0 0 systemd-journal
[ 6870.371054] [ 282] 0 282 30805 100 34 3 0 0 qubesdb-daemon
[ 6870.371073] [ 326] 0 326 23697 686 46 3 0 -1000 systemd-udevd
[ 6870.371091] [ 432] 0 432 3042 781 11 3 0 0 haveged
[ 6870.371108] [ 433] 81 433 13278 231 30 3 0 -900 dbus-daemon
[ 6870.371125] [ 435] 0 435 19358 189 42 3 0 0 systemd-logind
[ 6870.371143] [ 438] 0 438 10243 73 23 3 0 0 meminfo-writer
[ 6870.371162] [ 450] 0 450 34209 116 38 3 0 0 xl
[ 6870.371176] [ 454] 0 454 18919 152 43 3 0 0 qubes-gui
[ 6870.371194] [ 455] 0 455 16536 106 36 3 0 0 qrexec-agent
[ 6870.371211] [ 503] 0 503 73994 189 50 3 0 0 su
[ 6870.371226] [ 519] 0 519 52863 27 10 3 0 0 agetty
[ 6870.570122] [ 520] 0 520 52775 29 11 3 0 0 agetty
[ 6870.570136] [ 521] 1000 521 21958 334 45 3 0 0 systemd
[ 6870.570150] [ 527] 1000 527 34755 598 64 3 0 0 (sd-pam)
[ 6870.570163] [ 570] 1000 570 54160 86 14 3 0 0 bash
[ 6870.570177] [ 627] 1000 627 3500 30 12 3 0 0 xinit
[ 6870.570189] [ 640] 1000 640 323803 26706 180 4 0 0 Xorg
[ 6870.570205] [ 663] 1000 663 53597 67 12 4 0 0 qubes-session
[ 6870.570218] [ 675] 1000 675 13197 181 31 3 0 0 dbus-daemon
[ 6870.570231] [ 691] 1000 691 7242 119 17 3 0 0 ssh-agent
[ 6870.570244] [ 709] 1000 709 16562 105 38 3 0 0 qrexec-client-v
[ 6870.570257] [ 725] 1000 725 48074 134 31 3 0 0 dconf-service
[ 6870.570270] [ 731] 1000 731 428388 2600 194 5 0 0 gsd-xsettings
[ 6870.570282] [ 732] 1000 732 62744 1790 29 3 0 0 icon-sender
[ 6870.570295] [ 734] 1000 734 122405 246 40 3 0 0 gnome-keyring-d
[ 6870.570308] [ 735] 1000 735 120207 121 37 3 0 0 agent
[ 6870.570320] [ 748] 1000 748 438372 2843 212 5 0 0 nm-applet
[ 6870.570333] [ 755] 1000 755 128956 339 91 4 0 0 pulseaudio
[ 6870.570346] [ 757] 172 757 47723 83 29 3 0 0 rtkit-daemon
[ 6870.570360] [ 768] 998 768 657135 1585 91 6 1 0 polkitd
[ 6870.570372] [ 771] 1000 771 16528 100 34 3 0 0 qrexec-fork-ser
[ 6870.570385] [ 774] 1000 774 52238 17 11 3 0 0 sleep
[ 6870.570397] [ 848] 1000 848 87396 143 38 4 0 0 at-spi-bus-laun
[ 6870.570410] [ 854] 1000 854 13134 120 31 3 0 0 dbus-daemon
[ 6870.570423] [ 858] 1000 858 56364 197 46 3 0 0 at-spi2-registr
[ 6870.570436] [ 868] 1000 868 123835 208 46 3 0 0 gvfsd
[ 6870.570448] [ 902] 1000 902 105683 143 41 4 0 0 gvfsd-fuse
[ 6870.570462] [ 921] 1000 921 213915 6618 145 4 0 0 gnome-terminal-
[ 6870.570474] [ 925] 1000 925 206356 367 66 4 0 0 xdg-desktop-por
[ 6870.570486] [ 930] 1000 930 157249 145 39 4 0 0 xdg-document-po
[ 6870.570499] [ 933] 1000 933 117667 117 33 3 0 0 xdg-permission-
[ 6870.570511] [ 943] 1000 943 193292 1109 110 3 0 0 xdg-desktop-por
[ 6870.570524] [ 951] 1000 951 54193 171 16 3 0 0 bash
[ 6870.570536] [ 1012] 0 1012 252781 46241 218 4 0 0 packagekitd
[ 6870.570549] [ 2375] 1000 2375 54193 165 14 3 0 0 bash
[ 6870.570562] [ 6264] 1000 6264 63294 229 30 3 0 0 top
[ 6870.570579] [ 6748] 1000 6748 54193 160 13 3 0 0 bash
[ 6870.570595] [10247] 1000 10247 54193 161 13 3 0 0 bash
[ 6870.570607] [30948] 1000 30948 54193 158 16 3 0 0 bash
[ 6870.570620] [ 664] 1000 664 54193 156 15 4 0 0 bash
[ 6870.570633] [ 3687] 1000 3687 54023 213 14 3 0 0 watch
[ 6870.570645] [24682] 1000 24682 65858 761 34 3 0 0 rpmbuild
[ 6870.570657] [24781] 1000 24781 53597 75 13 3 0 0 sh
[ 6870.570668] [24795] 1000 24795 67125 4750 66 3 0 0 python2.7
[ 6870.570680] [24810] 1000 24810 31461 5860 62 3 0 0 python2.7
[ 6870.570692] [24848] 1000 24848 9587 583 23 3 0 0 gmake
[ 6870.570706] [24851] 1000 24851 9516 507 20 3 0 0 gmake
[ 6870.570719] [24952] 1000 24952 9521 484 22 3 0 0 gmake
[ 6870.570731] [24956] 1000 24956 9588 614 23 3 0 0 gmake
[ 6870.570743] [24997] 1000 24997 9145 139 21 3 0 0 gmake
[ 6870.570755] [25002] 1000 25002 218452 5840 215 4 0 0 cargo
[ 6870.570767] [ 1886] 1000 1886 9202 234 22 3 0 0 gmake
[ 6870.570781] [ 2121] 1000 2121 9284 298 22 3 0 0 gmake
[ 6870.570794] [ 4048] 1000 4048 755277 605078 1338 6 0 0 rustc
[ 6870.570807] [ 4540] 1000 4540 3006 320 11 3 0 0 ccache
[ 6870.570819] [ 4613] 1000 4613 3003 318 12 3 0 0 g++
[ 6870.570831] [ 4645] 1000 4645 2305 24 9 3 0 0 g++
[ 6870.570844] [ 4646] 1000 4646 498700 469312 952 5 0 0 cc1plus
[ 6870.570856] [ 4647] 1000 4647 37091 33394 79 4 0 0 as
[ 6870.770910] [ 4846] 1000 4846 2775 222 12 3 0 0 ccache
[ 6870.770924] [ 4861] 1000 4861 2773 220 10 3 0 0 g++
[ 6870.770937] [ 4877] 1000 4877 2305 23 10 3 0 0 g++
[ 6870.770950] [ 4878] 1000 4878 309707 283062 591 4 0 0 cc1plus
[ 6870.770963] [ 4879] 1000 4879 18182 14491 41 4 0 0 as
[ 6870.770974] [ 4897] 1000 4897 2780 202 11 3 0 0 ccache
[ 6870.770987] [ 4924] 1000 4924 2777 198 10 3 0 0 g++
[ 6870.771014] [ 4937] 1000 4937 2305 23 9 3 0 0 g++
[ 6870.771027] [ 4938] 1000 4938 306512 279904 587 4 0 0 cc1plus
[ 6870.771040] [ 4939] 1000 4939 9404 5674 23 3 0 0 as
[ 6870.771051] [ 4960] 1000 4960 2780 185 10 4 0 0 ccache
[ 6870.771064] [ 4967] 1000 4967 2814 200 11 3 0 0 ccache
[ 6870.771077] [ 4974] 1000 4974 2786 216 11 3 0 0 ccache
[ 6870.771090] [ 4979] 1000 4979 2813 184 11 3 0 0 ccache
[ 6870.771103] [ 4990] 1000 4990 2777 183 12 3 0 0 g++
[ 6870.771115] [ 4997] 1000 4997 2774 194 11 3 0 0 ccache
[ 6870.771128] [ 5006] 1000 5006 2812 187 10 3 0 0 g++
[ 6870.771141] [ 5009] 1000 5009 2813 201 10 3 0 0 g++
[ 6870.771154] [ 5012] 1000 5012 2785 216 11 3 0 0 g++
[ 6870.771166] [ 5015] 1000 5015 2305 22 9 3 0 0 g++
[ 6870.771179] [ 5016] 1000 5016 274112 247749 523 4 0 0 cc1plus
[ 6870.771192] [ 5017] 1000 5017 4982 1249 15 3 0 0 as
[ 6870.771203] [ 5020] 1000 5020 2772 191 10 3 0 0 g++
[ 6870.771216] [ 5023] 1000 5023 2305 23 10 3 0 0 g++
[ 6870.771230] [ 5024] 1000 5024 260164 241258 507 4 0 0 cc1plus
[ 6870.771244] [ 5025] 1000 5025 5511 1776 17 3 0 0 as
[ 6870.771256] [ 5026] 1000 5026 2305 23 9 3 0 0 g++
[ 6870.771268] [ 5027] 1000 5027 265514 241877 506 4 0 0 cc1plus
[ 6870.771280] [ 5028] 1000 5028 4983 1248 15 3 0 0 as
[ 6870.771294] [ 5031] 1000 5031 2305 23 8 3 0 0 g++
[ 6870.771306] [ 5032] 1000 5032 274929 251196 524 4 0 0 cc1plus
[ 6870.771318] [ 5033] 1000 5033 4983 1251 16 3 0 0 as
[ 6870.771329] [ 5034] 1000 5034 2305 24 9 3 0 0 g++
[ 6870.771341] [ 5035] 1000 5035 248405 222601 471 4 0 0 cc1plus
[ 6870.771354] [ 5036] 1000 5036 4982 1187 14 3 0 0 as
[ 6870.771364] [ 5051] 1000 5051 9145 136 21 3 0 0 gmake
[ 6870.771376] [ 5056] 1000 5056 2801 198 11 3 0 0 ccache
[ 6870.771389] [ 5066] 1000 5066 2799 196 10 3 0 0 g++
[ 6870.771401] [ 5071] 1000 5071 2795 175 10 3 0 0 ccache
[ 6870.771413] [ 5076] 1000 5076 2305 23 9 3 0 0 g++
[ 6870.771425] [ 5077] 1000 5077 175559 158120 341 3 0 0 cc1plus
[ 6870.771437] [ 5078] 1000 5078 4981 1252 17 3 0 0 as
[ 6870.771448] [ 5081] 1000 5081 2793 174 11 3 0 0 g++
[ 6870.771462] [ 5086] 1000 5086 2812 211 9 3 0 0 ccache
[ 6870.771474] [ 5091] 1000 5091 2305 23 10 3 0 0 g++
[ 6870.771486] [ 5092] 1000 5092 146716 125096 278 3 0 0 cc1plus
[ 6870.771498] [ 5093] 1000 5093 4982 1248 15 3 0 0 as
[ 6870.771508] [ 5096] 1000 5096 2811 210 11 3 0 0 g++
[ 6870.971546] [ 5101] 1000 5101 2305 23 9 3 0 0 g++
[ 6870.971560] [ 5102] 1000 5102 107775 89047 209 3 0 0 cc1plus
[ 6870.971573] [ 5103] 1000 5103 4983 1254 13 3 0 0 as
[ 6870.971584] [ 5118] 0 5118 7366 35 16 3 0 0 systemd-coredum
[ 6870.971597] [ 5120] 1000 5120 54023 213 11 3 0 0 watch
[ 6870.971610] [ 5121] 1000 5121 1465 16 7 3 0 0 sh
[ 6870.971620] Out of memory: Kill process 4048 (rustc) score 173 or sacrifice child
[ 6870.971643] Killed process 4048 (rustc) total-vm:3021108kB, anon-rss:2420312kB, file-rss:8kB, shmem-rss:0kB
[ 6871.117954] audit: type=1701 audit(1534438842.541:171): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=266 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" sig=6 res=1
[ 6871.322611] systemd[1]: Created slice system-systemd\x2dcoredump.slice.
[ 6871.338365] systemd[1]: Started Process Core Dump (PID 5118/UID 0).
[ 6871.340483] audit: type=1130 audit(1534438842.764:172): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@0-5118-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.357122] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
[ 6871.357205] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[ 6871.357751] audit: type=1130 audit(1534438842.781:173): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 6871.358161] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
[ 6871.358248] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[ 6871.358919] systemd[1]: Stopped Flush Journal to Persistent Storage.
[ 6871.358988] systemd[1]: Stopping Flush Journal to Persistent Storage...
[ 6871.358999] audit: type=1130 audit(1534438842.782:174): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359002] audit: type=1131 audit(1534438842.782:175): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359096] systemd[1]: Stopped Journal Service.
[ 6871.359138] audit: type=1130 audit(1534438842.783:176): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.359177] audit: type=1131 audit(1534438842.783:177): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 6871.364043] systemd[1]: Starting Journal Service...
[ 6871.396019] audit: type=1305 audit(1534438842.819:178): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 res=1
[ 6871.418446] systemd-journald[5125]: File /var/log/journal/12c5e521101c444594b96b53751551a8/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 6871.431865] systemd-coredump[5122]: MESSAGE=Process 266 (systemd-journal) of user 0 dumped core.
[ 6871.431884] systemd-coredump[5122]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.f4dc34a546c24e2aa7f99cdcd659015f.266.1534438842000000.lz4
[ 6871.431902] systemd-coredump[5122]: Stack trace of thread 266:
[ 6871.431912] systemd-coredump[5122]: #0 0x00007db8401c3ae4 journal_file_find_data_object_with_hash (libsystemd-shared-238.so)
[ 6871.431927] systemd-coredump[5122]: #1 0x00007db8401c3d59 journal_file_append_data (libsystemd-shared-238.so)
[ 6871.431941] systemd-coredump[5122]: #2 0x00007db8401c60d1 journal_file_append_entry (libsystemd-shared-238.so)
[ 6871.431955] systemd-coredump[5122]: #3 0x0000620290cfd6c8 dispatch_message_real (systemd-journald)
[ 6871.431967] systemd-coredump[5122]: #4 0x0000620290cf95e5 server_read_dev_kmsg (systemd-journald)
[ 6871.431980] systemd-coredump[5122]: #5 0x00007db8401b6cb8 source_dispatch (libsystemd-shared-238.so)
[ 6871.431992] systemd-coredump[5122]: #6 0x00007db8401b848c sd_event_dispatch (libsystemd-shared-238.so)
[ 6871.435899] systemd-coredum: 4 output lines suppressed due to ratelimiting
[ 6871.485869] audit: type=1130 audit(1534438842.909:179): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 6871.486357] audit: type=1130 audit(1534438842.910:180): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
So it looks like it ran out of memory and killed rustc, OR it was a side effect of qube Pause/Resume ?
10:28.79 Compiling style_traits v0.0.1 (file:///home/user/rpmbuild/BUILD/firefox-61.0.2/servo/components/style_traits)
...
13:09.57 /home/user/rpmbuild/BUILD/firefox-61.0.2/dom/crypto/CryptoKey.cpp:39:3: note: in expansion of macro 'NS_INTERFACE_MAP_ENTRY'
13:09.57 NS_INTERFACE_MAP_ENTRY(nsISupports)
13:09.57 ^~~~~~~~~~~~~~~~~~~~~~
21:52.81 error: Could not compile `style`.
21:52.82 To learn more, run the command again with --verbose.
21:52.87 gmake[4]: *** [/home/user/rpmbuild/BUILD/firefox-61.0.2/config/rules.mk:950: force-cargo-library-build] Error 101
21:52.89 gmake[3]: *** [/home/user/rpmbuild/BUILD/firefox-61.0.2/config/recurse.mk:73: toolkit/library/rust/target] Error 2
21:52.89 gmake[3]: *** Waiting for unfinished jobs....
Either way, I can't explain the constant reading speed of 192MiB/sec (according to dom0 xfce-panel plugin Disk Performance Monitor for /dev/sda while the qube was running)
$ cat /proc/meminfo
MemTotal: 7454500 kB
MemFree: 5635088 kB
MemAvailable: 6574676 kB
Buffers: 53832 kB
Cached: 1094368 kB
SwapCached: 0 kB
Active: 724832 kB
Inactive: 747696 kB
Active(anon): 233816 kB
Inactive(anon): 95768 kB
Active(file): 491016 kB
Inactive(file): 651928 kB
Unevictable: 73568 kB
Mlocked: 73568 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 292 kB
Writeback: 0 kB
AnonPages: 398016 kB
Mapped: 54320 kB
Shmem: 5256 kB
Slab: 134680 kB
SReclaimable: 74124 kB
SUnreclaim: 60556 kB
KernelStack: 4800 kB
PageTables: 10524 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3727248 kB
Committed_AS: 1332236 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 327644 kB
DirectMap2M: 14008320 kB
DirectMap1G: 0 kB
Hmm, mem total is 7.4Gig, it's set to 1400MB initial, and 14000MB max with "Include in memory balancing" enabled.
I'm kinda wondering if the OOM is the one that caused the huge disk reading, even though it doesn't make any sense...
Marcus Linsner
I resumed the firefox compilation and noticed that the memory jumped back to 14GB again - I was sure it was more than that 7.4GB before:
$ cat /proc/meminfo
MemTotal: 14003120 kB
MemFree: 4602448 kB
MemAvailable: 6622252 kB
Buffers: 186220 kB
Cached: 1986192 kB
SwapCached: 0 kB
Active: 7482024 kB
Inactive: 1448656 kB
Active(anon): 6667828 kB
Inactive(anon): 95780 kB
Active(file): 814196 kB
Inactive(file): 1352876 kB
Unevictable: 73568 kB
Mlocked: 73568 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 306392 kB
Writeback: 4684 kB
AnonPages: 6811888 kB
Mapped: 199164 kB
Shmem: 5340 kB
Slab: 239524 kB
SReclaimable: 177620 kB
SUnreclaim: 61904 kB
KernelStack: 5968 kB
PageTables: 28612 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 7001560 kB
Committed_AS: 8571548 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 327644 kB
DirectMap2M: 14008320 kB
DirectMap1G: 0 kB
Oh man, I'm hitting that disk thrashing again after just a few minutes: 202MiB/sec reading, 0.0 writing.
Paused qube, reading stopped.
Resumed qube sooner than before and it's still thrashing...
It'a a fedora 28 template-based VM.
I shut down another VM and I thought dom0 crashed because it froze for like 10 sec before the notification message told me that that VM stopped.
Marcus Linsner
Ok, I caught kswapd0 at 14% in a 'top' terminal on the offending qube, before the disk thrashing begun(which froze all terminals too) and then the only process at 100% after disk thrashing stopped! and here's the continuation of the log, btw the thrashing only stopped after OOM killed the rustc process(which, my guess was triggeding kswapd0 to use 100% cpu or what):
[ 6871.435899] systemd-coredum: 4 output lines suppressed due to ratelimiting
[ 6871.485869] audit: type=1130 audit(1534438842.909:179): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 6871.486357] audit: type=1130 audit(1534438842.910:180): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 7076.504392] kauditd_printk_skb: 5 callbacks suppressed
[ 7076.504393] audit: type=1101 audit(1534439047.928:186): pid=5658 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7076.504502] audit: type=1123 audit(1534439047.928:187): pid=5658 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd=636174202F6574632F67726F757073 terminal=pts/2 res=success'
[ 7076.505193] audit: type=1110 audit(1534439047.929:188): pid=5658 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7076.506823] audit: type=1105 audit(1534439047.930:189): pid=5658 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7076.508580] audit: type=1106 audit(1534439047.932:190): pid=5658 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7076.508681] audit: type=1104 audit(1534439047.932:191): pid=5658 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7078.730535] audit: type=1101 audit(1534439050.154:192): pid=5670 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7078.730868] audit: type=1123 audit(1534439050.154:193): pid=5670 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd=636174202F6574632F67726F7570 terminal=pts/2 res=success'
[ 7078.730905] audit: type=1110 audit(1534439050.154:194): pid=5670 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7078.731931] audit: type=1105 audit(1534439050.155:195): pid=5670 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success'
[ 7573.970907] kauditd_printk_skb: 2 callbacks suppressed
[ 7573.970908] audit: type=1101 audit(1534439545.394:198): pid=6724 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7573.970967] audit: type=1123 audit(1534439545.394:199): pid=6724 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success'
[ 7573.971028] audit: type=1110 audit(1534439545.394:200): pid=6724 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7573.971894] audit: type=1105 audit(1534439545.395:201): pid=6724 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7574.015405] audit: type=1106 audit(1534439545.439:202): pid=6724 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7574.015440] audit: type=1104 audit(1534439545.439:203): pid=6724 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7770.221883] audit: type=1101 audit(1534439741.645:204): pid=7130 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7770.222429] audit: type=1123 audit(1534439741.646:205): pid=7130 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success'
[ 7770.222529] audit: type=1110 audit(1534439741.646:206): pid=7130 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7770.227554] audit: type=1105 audit(1534439741.651:207): pid=7130 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7773.848220] audit: type=1106 audit(1534439745.271:208): pid=7130 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7773.848431] audit: type=1104 audit(1534439745.271:209): pid=7130 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7809.671640] audit: type=1101 audit(1534439781.095:210): pid=7217 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7809.672118] audit: type=1123 audit(1534439781.095:211): pid=7217 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success'
[ 7809.672152] audit: type=1110 audit(1534439781.095:212): pid=7217 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7809.673088] audit: type=1105 audit(1534439781.097:213): pid=7217 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7810.371225] audit: type=1106 audit(1534439781.794:214): pid=7217 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7810.371397] audit: type=1104 audit(1534439781.794:215): pid=7217 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7813.718144] audit: type=1101 audit(1534439785.142:216): pid=7234 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7813.718470] audit: type=1123 audit(1534439785.142:217): pid=7234 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd=646D657367202D2D636F6C6F723D616C77617973 terminal=pts/4 res=success'
[ 7813.718496] audit: type=1110 audit(1534439785.142:218): pid=7234 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7813.719407] audit: type=1105 audit(1534439785.143:219): pid=7234 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7817.505895] audit: type=1106 audit(1534439788.928:220): pid=7234 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 7817.506116] audit: type=1104 audit(1534439788.929:221): pid=7234 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 8251.964372] audit: type=1101 audit(1534440223.388:222): pid=8119 uid=1000 auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 8251.964572] audit: type=1123 audit(1534440223.388:223): pid=8119 uid=1000 auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success'
[ 8251.965245] audit: type=1110 audit(1534440223.388:224): pid=8119 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 8251.969268] audit: type=1105 audit(1534440223.393:225): pid=8119 uid=0 auid=1000 ses=1 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 8253.023385] audit: type=1106 audit(1534440224.447:226): pid=8119 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 8253.023615] audit: type=1104 audit(1534440224.447:227): pid=8119 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
[ 9739.590829] watchdog: BUG: soft lockup - CPU#11 stuck for 163s! [kswapd0:102]
[ 9739.590913] Modules linked in: binfmt_misc fuse ip6table_filter ip6_tables xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c xen_netfront intel_rapl crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel intel_rapl_perf pcspkr u2mfn(O) xen_gntdev xen_gntalloc xenfs xen_privcmd xen_blkback xen_evtchn xen_blkfront
[ 9739.591307] CPU: 11 PID: 102 Comm: kswapd0 Tainted: G O 4.14.57-1.pvops.qubes.x86_64 #1
[ 9739.591393] task: ffff88035654dc40 task.stack: ffffc90001f08000
[ 9739.591472] RIP: 0010:__raw_callee_save___pv_queued_spin_unlock+0x1/0x12
[ 9739.591536] RSP: 0000:ffffc90001f0bcb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff0c
[ 9739.591609] RAX: ffff880034d179e0 RBX: ffff8803591a4d00 RCX: 0000000000000000
[ 9739.591691] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8803591a4d00
[ 9739.591762] RBP: 0000000000000018 R08: 28f5c28f5c28f5c3 R09: 00000000000000bc
[ 9739.591836] R10: ffffc90001f0bdd0 R11: 0000000000000000 R12: ffffc90001f0bd68
[ 9739.591911] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88035a9b0458
[ 9739.591983] FS: 0000000000000000(0000) GS:ffff88035dac0000(0000) knlGS:0000000000000000
[ 9739.592056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9739.592121] CR2: 0000000000eea000 CR3: 000000000220a006 CR4: 00000000003606e0
[ 9739.592202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9739.592270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 9739.592337] Call Trace:
[ 9739.592376] list_lru_count_one+0x47/0x70
[ 9739.592425] super_cache_count+0x64/0xf0
[ 9739.592465] shrink_slab.part.50+0x12f/0x400
[ 9739.592522] shrink_node+0x10e/0x320
[ 9739.592586] kswapd+0x355/0x720
[ 9739.592633] kthread+0xff/0x130
[ 9739.592677] ? mem_cgroup_shrink_node+0x170/0x170
[ 9739.592729] ? kthread_create_on_node+0x70/0x70
[ 9739.592781] ret_from_fork+0x35/0x40
[ 9739.592831] Code: 90 90 90 90 90 90 90 90 90 90 51 52 56 57 41 50 41 51 41 52 41 53 e8 cf 04 00 00 41 5b 41 5a 41 59 41 58 5f 5e 5a 59 c3 66 90 52 <b8> 01 00 00 00 31 d2 f0 0f b0 17 3c 01 75 02 5a c3 56 0f b6 f0
[ 9744.021459] audit: type=1701 audit(1534441715.445:228): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=326 comm="systemd-udevd" exe="/usr/lib/systemd/systemd-udevd" sig=6 res=1
[ 9781.154297] sh invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[ 9781.154323] sh cpuset=/ mems_allowed=0
[ 9781.154333] CPU: 6 PID: 13108 Comm: sh Tainted: G O L 4.14.57-1.pvops.qubes.x86_64 #1
[ 9781.154349] Call Trace:
[ 9781.154358] dump_stack+0x5c/0x85
[ 9781.154367] dump_header+0x94/0x21c
[ 9781.154376] ? get_page_from_freelist+0x184/0xbd0
[ 9781.154386] oom_kill_process+0x212/0x410
[ 9781.154394] out_of_memory+0x131/0x4b0
[ 9781.154403] __alloc_pages_slowpath+0xda8/0xe90
[ 9781.154414] __alloc_pages_nodemask+0x270/0x2b0
[ 9781.154424] filemap_fault+0x54c/0x860
[ 9781.154433] ? alloc_set_pte+0x343/0x520
[ 9781.154442] ? filemap_map_pages+0x2d5/0x510
[ 9781.154454] ext4_filemap_fault+0x2c/0x40
[ 9781.154463] __do_fault+0x1e/0xb5
[ 9781.154471] __handle_mm_fault+0xc40/0xda0
[ 9781.154481] handle_mm_fault+0xc6/0x1b0
[ 9781.154489] __do_page_fault+0x255/0x4f0
[ 9781.154499] do_page_fault+0x33/0x120
[ 9781.154507] ? page_fault+0x2f/0x50
[ 9781.154515] page_fault+0x45/0x50
[ 9781.154523] RIP: 4d5aa6d8: (null)
[ 9781.154531] RSP: 0000:000060b33d2792e0 EFLAGS: 60b33d04550f
[ 9781.154537] Mem-Info:
[ 9781.154554] active_anon:3359753 inactive_anon:23957 isolated_anon:0
[ 9781.154554] active_file:192 inactive_file:352 isolated_file:0
[ 9781.154554] unevictable:18392 dirty:0 writeback:1 unstable:0
[ 9781.154554] slab_reclaimable:10075 slab_unreclaimable:15506
[ 9781.154554] mapped:1227 shmem:1351 pagetables:10544 bounce:0
[ 9781.154554] free:30606 free_pcp:1856 free_cma:0
[ 9781.154612] Node 0 active_anon:13439012kB inactive_anon:95828kB active_file:768kB inactive_file:828kB unevictable:73568kB isolated(anon):0kB isolated(file):0kB mapped:4648kB dirty:0kB writeback:4kB shmem:5404kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 9781.154654] Node 0 DMA free:15908kB min:76kB low:92kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 9781.154697] lowmem_reserve[]: 0 3956 13655 13655 13655
[ 9781.154715] Node 0 DMA32 free:58312kB min:19556kB low:24444kB high:29332kB active_anon:3878024kB inactive_anon:38876kB active_file:224kB inactive_file:776kB unevictable:32436kB writepending:0kB present:4159452kB managed:4071632kB mlocked:32436kB kernel_stack:2384kB pagetables:14536kB bounce:0kB free_pcp:948kB local_pcp:24kB free_cma:0kB
[ 9781.154764] lowmem_reserve[]: 0 0 9699 9699 9699
[ 9781.154775] Node 0 Normal free:48204kB min:47948kB low:59932kB high:71916kB active_anon:9560988kB inactive_anon:56952kB active_file:3120kB inactive_file:2216kB unevictable:41132kB writepending:4kB present:10160128kB managed:9915580kB mlocked:41132kB kernel_stack:3712kB pagetables:27640kB bounce:0kB free_pcp:6484kB local_pcp:632kB free_cma:0kB
[ 9781.154829] lowmem_reserve[]: 0 0 0 0 0
[ 9781.154838] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 9781.154869] Node 0 DMA32: 278*4kB (UME) 206*8kB (UME) 230*16kB (UME) 267*32kB (UME) 131*64kB (UME) 65*128kB (UME) 38*256kB (UME) 19*512kB (UME) 5*1024kB (UE) 1*2048kB (M) 0*4096kB = 58312kB
[ 9781.154909] Node 0 Normal: 1113*4kB (UMEH) 541*8kB (UMEH) 479*16kB (UMEH) 437*32kB (UMEH) 177*64kB (UME) 37*128kB (ME) 10*256kB (ME) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 49564kB
[ 9781.154944] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 9781.154959] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 9781.154975] 1803 total pagecache pages
[ 9781.154982] 0 pages in swap cache
[ 9781.154990] Swap cache stats: add 3510, delete 3510, find 44/106
[ 9781.155011] Free swap = 0kB
[ 9781.155040] Total swap = 0kB
[ 9781.155048] 3583894 pages RAM
[ 9781.155055] 0 pages HighMem/MovableOnly
[ 9781.155062] 83114 pages reserved
[ 9781.155069] 0 pages cma reserved
[ 9781.155076] 0 pages hwpoisoned
[ 9781.155083] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 9781.155106] [ 282] 0 282 30805 100 34 3 0 0 qubesdb-daemon
[ 9781.155123] [ 326] 0 326 23697 686 46 3 0 -1000 systemd-udevd
[ 9781.155139] [ 432] 0 432 3042 781 11 3 0 0 haveged
[ 9781.155155] [ 433] 81 433 13278 230 30 3 0 -900 dbus-daemon
[ 9781.155171] [ 438] 0 438 10243 73 23 3 0 0 meminfo-writer
[ 9781.155187] [ 450] 0 450 34209 116 38 3 0 0 xl
[ 9781.155202] [ 454] 0 454 18997 203 43 3 0 0 qubes-gui
[ 9781.155218] [ 455] 0 455 16536 106 36 3 0 0 qrexec-agent
[ 9781.155234] [ 503] 0 503 73994 189 50 3 0 0 su
[ 9781.355141] [ 519] 0 519 52863 27 10 3 0 0 agetty
[ 9781.355161] [ 520] 0 520 52775 29 11 3 0 0 agetty
[ 9781.355179] [ 521] 1000 521 21958 338 45 3 0 0 systemd
[ 9781.355196] [ 527] 1000 527 34755 598 64 3 0 0 (sd-pam)
[ 9781.355216] [ 570] 1000 570 54160 86 14 3 0 0 bash
[ 9781.355234] [ 627] 1000 627 3500 30 12 3 0 0 xinit
[ 9781.355253] [ 640] 1000 640 323803 26712 180 4 0 0 Xorg
[ 9781.355276] [ 663] 1000 663 53597 67 12 4 0 0 qubes-session
[ 9781.355296] [ 675] 1000 675 13197 181 31 3 0 0 dbus-daemon
[ 9781.355315] [ 691] 1000 691 7242 119 17 3 0 0 ssh-agent
[ 9781.355333] [ 709] 1000 709 16562 127 38 3 0 0 qrexec-client-v
[ 9781.355352] [ 725] 1000 725 48107 148 31 3 0 0 dconf-service
[ 9781.355371] [ 731] 1000 731 428388 2600 194 5 0 0 gsd-xsettings
[ 9781.355389] [ 732] 1000 732 62744 1790 29 3 0 0 icon-sender
[ 9781.355408] [ 734] 1000 734 122405 245 40 3 0 0 gnome-keyring-d
[ 9781.355426] [ 735] 1000 735 120207 121 37 3 0 0 agent
[ 9781.355444] [ 748] 1000 748 438372 2843 212 5 0 0 nm-applet
[ 9781.355462] [ 755] 1000 755 128956 339 91 4 0 0 pulseaudio
[ 9781.355482] [ 757] 172 757 47723 87 29 3 0 0 rtkit-daemon
[ 9781.355500] [ 768] 998 768 657135 1588 91 6 1 0 polkitd
[ 9781.355522] [ 771] 1000 771 16528 100 34 3 0 0 qrexec-fork-ser
[ 9781.355541] [ 774] 1000 774 52238 17 11 3 0 0 sleep
[ 9781.355559] [ 848] 1000 848 87396 143 38 4 0 0 at-spi-bus-laun
[ 9781.355578] [ 854] 1000 854 13134 120 31 3 0 0 dbus-daemon
[ 9781.355597] [ 858] 1000 858 56364 197 46 3 0 0 at-spi2-registr
[ 9781.355616] [ 868] 1000 868 123835 221 46 3 0 0 gvfsd
[ 9781.355634] [ 902] 1000 902 105683 143 41 4 0 0 gvfsd-fuse
[ 9781.355651] [ 921] 1000 921 214263 6889 146 4 0 0 gnome-terminal-
[ 9781.355664] [ 925] 1000 925 206356 367 66 4 0 0 xdg-desktop-por
[ 9781.355676] [ 930] 1000 930 157249 162 39 4 0 0 xdg-document-po
[ 9781.355688] [ 933] 1000 933 117667 117 33 3 0 0 xdg-permission-
[ 9781.355700] [ 943] 1000 943 193292 1125 110 3 0 0 xdg-desktop-por
[ 9781.355713] [ 951] 1000 951 54193 171 16 3 0 0 bash
[ 9781.355725] [ 1012] 0 1012 257186 47659 220 4 0 0 packagekitd
[ 9781.355742] [ 2375] 1000 2375 54193 165 14 3 0 0 bash
[ 9781.355755] [ 6264] 1000 6264 63294 229 30 3 0 0 top
[ 9781.355768] [ 6748] 1000 6748 54193 161 13 3 0 0 bash
[ 9781.355780] [10247] 1000 10247 54193 161 13 3 0 0 bash
[ 9781.355793] [30948] 1000 30948 54193 160 16 3 0 0 bash
[ 9781.355805] [ 664] 1000 664 54193 156 15 4 0 0 bash
[ 9781.355818] [ 3687] 1000 3687 54023 213 14 3 0 0 watch
[ 9781.355830] [ 5125] 0 5125 26967 181 46 3 0 0 systemd-journal
[ 9781.355843] [ 5136] 0 5136 19325 174 40 3 0 0 systemd-logind
[ 9781.355855] [ 9347] 1000 9347 65858 760 38 4 0 0 rpmbuild
[ 9781.355868] [ 9446] 1000 9446 53597 74 12 3 0 0 sh
[ 9781.355878] [ 9476] 1000 9476 67165 4754 68 3 0 0 python2.7
[ 9781.554909] [ 9489] 1000 9489 30476 4870 62 3 0 0 python2.7
[ 9781.554927] [ 9527] 1000 9527 9587 582 23 3 0 0 gmake
[ 9781.554940] [ 9530] 1000 9530 9516 504 22 3 0 0 gmake
[ 9781.554953] [ 9633] 1000 9633 9521 484 20 3 0 0 gmake
[ 9781.554965] [ 9637] 1000 9637 9654 680 22 3 0 0 gmake
[ 9781.554978] [ 9677] 1000 9677 9145 139 22 3 0 0 gmake
[ 9781.554991] [ 9683] 1000 9683 53977 4945 46 4 0 0 cargo
[ 9781.555018] [ 9955] 1000 9955 738373 611629 1350 6 0 0 rustc
[ 9781.555033] [12689] 1000 12689 9175 203 19 3 0 0 gmake
[ 9781.555045] [12775] 1000 12775 9145 137 21 3 0 0 gmake
[ 9781.555063] [12822] 1000 12822 9172 204 22 4 0 0 gmake
[ 9781.555076] [12882] 1000 12882 2943 241 11 3 0 0 ccache
[ 9781.555088] [12895] 1000 12895 2941 239 11 3 0 0 g++
[ 9781.555101] [12905] 1000 12905 2305 24 8 3 0 0 g++
[ 9781.555113] [12906] 1000 12906 446815 416509 848 4 0 0 cc1plus
[ 9781.555126] [12907] 1000 12907 7225 3466 19 3 0 0 as
[ 9781.555137] [12911] 1000 12911 9144 171 22 3 0 0 gmake
[ 9781.555149] [12918] 1000 12918 2799 254 12 3 0 0 ccache
[ 9781.555162] [12925] 1000 12925 2797 196 11 3 0 0 ccache
[ 9781.555174] [12933] 1000 12933 2795 194 10 4 0 0 g++
[ 9781.555187] [12936] 1000 12936 2797 253 10 3 0 0 g++
[ 9781.555199] [12944] 1000 12944 2305 23 8 3 0 0 g++
[ 9781.555211] [12945] 1000 12945 295307 277152 575 4 0 0 cc1plus
[ 9781.555224] [12946] 1000 12946 18940 15243 42 3 0 0 as
[ 9781.555234] [12947] 1000 12947 2305 24 9 3 0 0 g++
[ 9781.555247] [12948] 1000 12948 361103 334097 695 4 0 0 cc1plus
[ 9781.555260] [12949] 1000 12949 9074 5344 22 3 0 0 as
[ 9781.555271] [12958] 1000 12958 2766 184 10 3 0 0 ccache
[ 9781.555286] [12970] 1000 12970 2780 216 10 3 0 0 ccache
[ 9781.555298] [12980] 1000 12980 2764 182 10 3 0 0 g++
[ 9781.555310] [12988] 1000 12988 2305 23 9 3 0 0 g++
[ 9781.555322] [12989] 1000 12989 263737 244263 513 4 0 0 cc1plus
[ 9781.555334] [12990] 1000 12990 9502 5774 23 3 0 0 as
[ 9781.555345] [12991] 1000 12991 2779 214 11 3 0 0 g++
[ 9781.555357] [12998] 1000 12998 2791 188 10 3 0 0 ccache
[ 9781.555369] [13001] 1000 13001 2305 24 10 3 0 0 g++
[ 9781.555381] [13002] 1000 13002 318726 292631 610 4 0 0 cc1plus
[ 9781.555393] [13003] 1000 13003 4982 1252 13 3 0 0 as
[ 9781.555404] [13006] 1000 13006 2789 186 11 4 0 0 g++
[ 9781.555416] [13011] 1000 13011 2305 24 9 3 0 0 g++
[ 9781.555428] [13012] 1000 13012 245657 226072 475 4 0 0 cc1plus
[ 9781.555440] [13013] 1000 13013 6862 3134 17 3 0 0 as
[ 9781.555453] [13021] 1000 13021 9145 137 22 4 0 0 gmake
[ 9781.555465] [13023] 1000 13023 9144 136 21 3 0 0 gmake
[ 9781.555477] [13026] 1000 13026 2781 232 11 3 0 0 ccache
[ 9781.555489] [13027] 1000 13027 2786 183 9 3 0 0 ccache
[ 9781.555502] [13040] 1000 13040 2762 179 11 3 0 0 ccache
[ 9781.555515] [13041] 1000 13041 2793 188 11 3 0 0 ccache
[ 9781.555527] [13048] 1000 13048 2761 178 10 3 0 0 g++
[ 9781.754559] [13051] 1000 13051 2784 181 10 3 0 0 g++
[ 9781.754574] [13054] 1000 13054 2791 186 11 3 0 0 g++
[ 9781.754590] [13057] 1000 13057 2779 231 10 3 0 0 g++
[ 9781.754607] [13062] 1000 13062 2304 23 9 3 0 0 g++
[ 9781.754624] [13063] 1000 13063 213546 192190 412 4 0 0 cc1plus
[ 9781.754641] [13064] 1000 13064 4981 1221 16 3 0 0 as
[ 9781.754655] [13065] 1000 13065 2305 22 9 3 0 0 g++
[ 9781.754672] [13066] 1000 13066 209797 189104 403 4 0 0 cc1plus
[ 9781.754689] [13067] 1000 13067 4981 1219 15 3 0 0 as
[ 9781.754702] [13068] 1000 13068 2304 23 9 3 0 0 g++
[ 9781.754717] [13069] 1000 13069 208141 188217 402 3 0 0 cc1plus
[ 9781.754729] [13070] 1000 13070 4981 1223 13 3 0 0 as
[ 9781.754740] [13073] 1000 13073 2305 24 9 3 0 0 g++
[ 9781.754754] [13074] 1000 13074 204312 186045 395 3 0 0 cc1plus
[ 9781.754770] [13075] 1000 13075 4982 1248 16 3 0 0 as
[ 9781.754783] [13080] 1000 13080 2762 176 10 3 0 0 ccache
[ 9781.754796] [13089] 1000 13089 2761 174 11 3 0 0 g++
[ 9781.754808] [13094] 1000 13094 2304 22 8 3 0 0 g++
[ 9781.754821] [13095] 1000 13095 104070 81528 194 3 0 0 cc1plus
[ 9781.754835] [13096] 1000 13096 4981 1222 16 3 0 0 as
[ 9781.754846] [13107] 1000 13107 54023 213 11 3 0 0 watch
[ 9781.754859] [13108] 1000 13108 53597 59 14 3 0 0 sh
[ 9781.754870] [13109] 0 13109 11239 48 23 3 0 0 systemd-coredum
[ 9781.754883] Out of memory: Kill process 9955 (rustc) score 175 or sacrifice child
[ 9781.754904] Killed process 9955 (rustc) total-vm:2953492kB, anon-rss:2446516kB, file-rss:0kB, shmem-rss:0kB
[ 9781.838565] oom_reaper: reaped process 9955 (rustc), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 9781.944947] audit: type=1701 audit(1534441753.368:229): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=5125 comm="systemd-journal" exe="/usr/lib/systemd/systemd-journald" sig=6 res=1
[ 9782.129302] systemd[1]: Started Process Core Dump (PID 13109/UID 0).
[ 9782.131966] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
[ 9782.132076] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[ 9782.133213] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
[ 9782.133287] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 2.
[ 9782.133437] systemd[1]: Stopped Flush Journal to Persistent Storage.
[ 9782.133479] systemd[1]: Stopping Flush Journal to Persistent Storage...
[ 9782.133498] systemd[1]: Stopped Journal Service.
[ 9782.135777] systemd[1]: Starting Journal Service...
[ 9782.136153] audit: type=1130 audit(1534441753.555:230): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@1-13109-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.136189] audit: type=1131 audit(1534441753.556:231): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 9782.136220] audit: type=1131 audit(1534441753.557:232): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.136251] audit: type=1130 audit(1534441753.557:233): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.136281] audit: type=1131 audit(1534441753.557:234): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.153162] audit: type=1305 audit(1534441753.576:235): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 res=1
[ 9782.168207] systemd-journald[13113]: File /var/log/journal/12c5e521101c444594b96b53751551a8/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 9782.202464] systemd-coredump[13111]: MESSAGE=Process 5125 (systemd-journal) of user 0 dumped core.
[ 9782.202488] systemd-coredump[13111]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.f4dc34a546c24e2aa7f99cdcd659015f.5125.1534441753000000.lz4
[ 9782.202518] systemd-coredump[13111]: Stack trace of thread 5125:
[ 9782.202531] systemd-coredump[13111]: #0 0x00007bd86c2981f7 fsync (libpthread.so.0)
[ 9782.202546] systemd-coredump[13111]: #1 0x00007bd86ca3607b journal_file_set_online (libsystemd-shared-238.so)
[ 9782.202564] systemd-coredump[13111]: #2 0x00007bd86ca3716a journal_file_append_object (libsystemd-shared-238.so)
[ 9782.202583] systemd-coredump[13111]: #3 0x00007bd86ca37dc3 journal_file_append_data (libsystemd-shared-238.so)
[ 9782.202602] systemd-coredump[13111]: #4 0x00007bd86ca3a0d1 journal_file_append_entry (libsystemd-shared-238.so)
[ 9782.202622] systemd-coredump[13111]: #5 0x0000569e8af706c8 dispatch_message_real (systemd-journald)
[ 9782.202638] systemd-coredump[13111]: #6 0x0000569e8af6c5e5 server_read_dev_kmsg (systemd-journald)
[ 9782.206525] systemd-coredum: 6 output lines suppressed due to ratelimiting
[ 9782.219312] systemd[1]: Started Journal Service.
[ 9782.219459] audit: type=1130 audit(1534441753.643:236): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.233297] audit: type=1130 audit(1534441753.657:237): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 9782.327795] audit: type=1130 audit(1534441753.751:238): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Marcus Linsner
The good news is that I've realized that the OOM triggering was legit: I had firefox set to use 12 cores at once and 14GiB of RAM was clearly not enough! (8 and no ccache was good though - did compile it twice like so)
The bad news is that I still don't know why the disk-read thrashing was happening for me, but I will default to blame the OOM (even though no swap was active, ie. I swapoff-ed the swap partition earlier) due to previous experience with OOM triggering on bare-metal hardware: I seem to remember SSD disk activity led being full-on during an impending OOM and everything freezing!
brenda...@gmail.com
> The good news is that I've realized that the OOM triggering was legit: I had firefox set to use 12 cores at once and 14GiB of RAM was clearly not enough! (8 and no ccache was good though - did compile it twice like so)
>
> The bad news is that I still don't know why the disk-read thrashing was happening for me, but I will default to blame the OOM (even though no swap was active, ie. I swapoff-ed the swap partition earlier) due to previous experience with OOM triggering on bare-metal hardware: I seem to remember SSD disk activity led being full-on during an impending OOM and everything freezing!
Maybe this applies:
https://askubuntu.com/questions/432809/why-is-kswapd0-running-on-a-computer-with-no-swap
[[if kswapd0 is taking any CPU and you do not have swap, the system is nearly out of RAM and is trying to deal with the situation by (in practise) swapping pages from executables. The correct fix is to reduce workload, add swap or (preferably) install more RAM. Adding swap will improve performance because kernel will have more options about what to swap to disk. Without swap the kernel is practically forced to swap application code.]]
This could be a reason you only see reads hammering the drive, maybe?
Also worth remembering: every read is decrypting block(s) which takes some CPU (even on systems with AES-NI support).
Brendan
Marcus Linsner
Thank you Brendan! The following comment(from the webpage that you linked) explained the constant disk-reading best for me:
"For example, consider a case where you have zero swap and system is nearly running out of RAM. The kernel will take memory from e.g. Firefox (it can do this because Firefox is running executable code that has been loaded from disk - the code can be loaded from disk again if needed). If Firefox then needs to access that RAM again N seconds later, the CPU generates "hard fault" which forces Linux to free some RAM (e.g. take some RAM from another process), load the missing data from disk and then allow Firefox to continue as usual. This is pretty similar to normal swapping and kswapd0 does it. " - Mikko Rantalainen Feb 15 at 13:08
$ sysctl vm.swappiness
vm.swappiness = 60
In retrospect, I apologize for hijacking this thread, because it now appears to me that my issue is totally different from the OP(even though the subject still applies):
Marcus Linsner
> "For example, consider a case where you have zero swap and system is nearly running out of RAM. The kernel will take memory from e.g. Firefox (it can do this because Firefox is running executable code that has been loaded from disk - the code can be loaded from disk again if needed). If Firefox then needs to access that RAM again N seconds later, the CPU generates "hard fault" which forces Linux to free some RAM (e.g. take some RAM from another process), load the missing data from disk and then allow Firefox to continue as usual. This is pretty similar to normal swapping and kswapd0 does it. " - Mikko Rantalainen Feb 15 at 13:08
Good news: no more disk thrashing with this patch [1] (also attached) and I'm keeping track of how to properly get rid of this disk thrashing in this [2].
Bad news: I made the patch and I've no idea how good it is(since I am noob :D) and what are the side-effects of using it. Likely a better patch can be made! (but none who know how to do it right have answered/helped yet :D so ... it's, for me, better than nothing)
I'm not going to post here anymore, to allow OP to be answered (since, it seems to be a different issue)
[1] https://github.com/constantoverride/qubes-linux-kernel/blob/acd686a5019c7ab6ec10dc457bdee4830e2d741f/patches.addon/le9b.patch
[2] https://stackoverflow.com/q/52067753/10239615