Allocating(too fast)fails and triggers OOM-killer before MemTotal reaches AppVM's set Max memory
Marcus Linsner
Initial Memory: 2400 MB
Max memory 24000 MB
[v] Include in memory balancing
has no swap enabled (swapon reports nothing) and uses an official kernel(eg. not compiled by me):
Linux dev01-w-s-f-fdr28 4.19.12-3.pvops.qubes.x86_64 #1 SMP Wed Dec 26 22:31:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Then I try this in its `gnome-terminal`:
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=1 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2188636 kB for each of the 1 concurrent threads.
MemTotal before: 2774252 kB
stress: info: [21239] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: info: [21239] successful run completed in 10s
real 0m10.048s
user 0m9.509s
sys 0m0.467s
exit code: 0
MemTotal afterwards: 8606644 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2165272 kB for each of the 2 concurrent threads.
MemTotal before: 2764324 kB
stress: info: [23838] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [23838] (415) <-- worker 23840 got signal 9
stress: WARN: [23838] (417) now reaping child worker processes
stress: FAIL: [23838] (451) failed run completed in 1s
real 0m0.636s
user 0m0.123s
sys 0m0.710s
exit code: 1
MemTotal afterwards: 3135016 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=1 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2184124 kB for each of the 1 concurrent threads.
MemTotal before: 2756864 kB
stress: info: [24126] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: info: [24126] successful run completed in 10s
real 0m10.077s
user 0m9.518s
sys 0m0.486s
exit code: 0
MemTotal afterwards: 8580864 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 5131272 kB for each of the 2 concurrent threads.
MemTotal before: 5698076 kB
stress: info: [24371] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [24371] (415) <-- worker 24373 got signal 9
stress: WARN: [24371] (417) now reaping child worker processes
stress: FAIL: [24371] (451) failed run completed in 1s
real 0m1.584s
user 0m0.567s
sys 0m2.378s
exit code: 1
MemTotal afterwards: 3887140 kB
So, you can see that it fails with 2 workers(/threads) because apparently it doesn't get to hot plug enough RAM before the OOM-killer triggers:
[ 1321.957837] gnome-terminal- invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[ 1321.957903] gnome-terminal- cpuset=/ mems_allowed=0
[ 1321.957916] CPU: 1 PID: 906 Comm: gnome-terminal- Tainted: G O 4.19.12-3.pvops.qubes.x86_64 #1
[ 1321.957932] Call Trace:
[ 1321.957942] dump_stack+0x5c/0x7b
[ 1321.957951] dump_header+0x6b/0x29c
[ 1321.957959] oom_kill_process+0x27b/0x280
[ 1321.957966] ? oom_badness+0xd9/0x130
[ 1321.957973] out_of_memory+0x12f/0x4e0
[ 1321.957984] __alloc_pages_slowpath+0xa07/0xe00
[ 1321.957994] __alloc_pages_nodemask+0x29e/0x2e0
[ 1321.958005] filemap_fault+0x52a/0x850
[ 1321.958013] ? filemap_map_pages+0x2d5/0x520
[ 1321.958023] ext4_filemap_fault+0x2c/0x3b
[ 1321.958031] __do_fault+0x1f/0xc0
[ 1321.958039] do_fault+0x367/0x4e0
[ 1321.958046] ? __switch_to_asm+0x40/0x70
[ 1321.958054] __handle_mm_fault+0x705/0x780
[ 1321.958062] handle_mm_fault+0xfc/0x1f0
[ 1321.958070] __do_page_fault+0x255/0x4f0
[ 1321.958077] do_page_fault+0x32/0x110
[ 1321.958085] ? page_fault+0x8/0x30
[ 1321.958092] page_fault+0x1e/0x30
[ 1321.958100] RIP: 0033:0x7d75d1c6ce30
[ 1321.958110] Code: Bad RIP value.
[ 1321.958117] RSP: 002b:00007ffcabc38288 EFLAGS: 00010246
[ 1321.958126] RAX: 0000000000000000 RBX: 000061f5f6d25050 RCX: 0000000000000001
[ 1321.958139] RDX: 000061f5f6405250 RSI: 000061f5f6405140 RDI: 000061f5f6d25050
[ 1321.958153] RBP: 00007ffcabc382b0 R08: 000061f5f690b030 R09: 0000000000000006
[ 1321.958165] R10: 000061f5f690b040 R11: 00007ffcabc382e0 R12: 0000000000000000
[ 1321.958179] R13: 000061f5f664db10 R14: 00007d75d0f0e3c0 R15: 000061f5f6c41000
[ 1321.958197] Mem-Info:
[ 1321.958204] active_anon:874613 inactive_anon:616 isolated_anon:0
active_file:158 inactive_file:67 isolated_file:0
unevictable:1933 dirty:55 writeback:7 unstable:0
slab_reclaimable:6100 slab_unreclaimable:8979
mapped:721 shmem:748 pagetables:3823 bounce:0
free:40265 free_pcp:15 free_cma:0
[ 1321.961366] Node 0 active_anon:3498452kB inactive_anon:2464kB active_file:592kB inactive_file:212kB unevictable:7732kB isolated(anon):0kB isolated(file):0kB mapped:2884kB dirty:220kB writeback:28kB shmem:2992kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[ 1321.961408] Node 0 DMA free:15908kB min:44kB low:56kB high:68kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1321.961449] lowmem_reserve[]: 0 3952 23480 23480 23480
[ 1321.961459] Node 0 DMA32 free:88996kB min:11368kB low:15412kB high:19456kB active_anon:2613496kB inactive_anon:44kB active_file:348kB inactive_file:392kB unevictable:6428kB writepending:128kB present:4159456kB managed:2722652kB mlocked:6428kB kernel_stack:64kB pagetables:5452kB bounce:0kB free_pcp:72kB local_pcp:24kB free_cma:0kB
[ 1321.961503] lowmem_reserve[]: 0 0 19528 19528 19528
[ 1321.961513] Node 0 Normal free:56156kB min:56168kB low:76164kB high:96160kB active_anon:884308kB inactive_anon:2420kB active_file:456kB inactive_file:484kB unevictable:1304kB writepending:560kB present:20400128kB managed:1148580kB mlocked:1304kB kernel_stack:3184kB pagetables:9840kB bounce:0kB free_pcp:24kB local_pcp:0kB free_cma:0kB
[ 1321.961561] lowmem_reserve[]: 0 0 0 0 0
[ 1321.961568] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 1321.961595] Node 0 DMA32: 296*4kB (UME) 185*8kB (UME) 111*16kB (ME) 93*32kB (UME) 64*64kB (UME) 38*128kB (UME) 22*256kB (ME) 1*512kB (U) 0*1024kB 1*2048kB (M) 16*4096kB (U) = 90104kB
[ 1321.961625] Node 0 Normal: 285*4kB (UME) 229*8kB (UME) 57*16kB (UME) 30*32kB (UME) 15*64kB (UME) 5*128kB (UME) 4*256kB (UME) 2*512kB (E) 2*1024kB (E) 1*2048kB (M) 11*4096kB (U) = 57644kB
[ 1321.961657] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1321.966614] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1321.966630] 943 total pagecache pages
[ 1321.966637] 0 pages in swap cache
[ 1321.966644] Swap cache stats: add 0, delete 0, find 0/0
[ 1321.966653] Free swap = 0kB
[ 1321.966659] Total swap = 0kB
[ 1321.966666] 6143895 pages RAM
[ 1321.966673] 0 pages HighMem/MovableOnly
[ 1321.966680] 5172110 pages reserved
[ 1321.966686] 0 pages cma reserved
[ 1321.966693] 0 pages hwpoisoned
[ 1321.966703] Tasks state (memory values in pages):
[ 1321.966711] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1321.966735] [ 227] 0 227 29076 209 237568 0 0 systemd-journal
[ 1321.966752] [ 238] 0 238 30814 102 167936 0 0 qubesdb-daemon
[ 1321.966767] [ 264] 0 264 23716 629 212992 0 -1000 systemd-udevd
[ 1321.966783] [ 508] 81 508 13266 237 147456 0 -900 dbus-daemon
[ 1321.966798] [ 511] 0 511 3042 782 69632 0 0 haveged
[ 1321.966812] [ 514] 0 514 19355 194 184320 0 0 systemd-logind
[ 1321.966827] [ 518] 0 518 10249 75 122880 0 0 meminfo-writer
[ 1321.966842] [ 533] 0 533 34218 118 180224 0 0 xl
[ 1321.966856] [ 576] 0 576 16543 106 176128 0 0 qrexec-agent
[ 1321.966871] [ 580] 0 580 18925 146 196608 0 0 qubes-gui
[ 1321.966887] [ 589] 0 589 22821 175 221184 0 0 qubes-gui-runus
[ 1321.966902] [ 679] 0 679 52863 27 69632 0 0 agetty
[ 1321.966915] [ 682] 1000 682 21962 329 208896 0 0 systemd
[ 1321.966929] [ 683] 0 683 52775 28 73728 0 0 agetty
[ 1321.968715] [ 697] 1000 697 34789 585 294912 0 0 (sd-pam)
[ 1321.968733] [ 730] 1000 730 3500 30 69632 0 0 xinit
[ 1321.968747] [ 820] 1000 820 201789 14993 614400 0 0 Xorg
[ 1321.968762] [ 848] 1000 848 53597 70 81920 0 0 qubes-session
[ 1321.968778] [ 859] 1000 859 13202 161 147456 0 0 dbus-daemon
[ 1321.968793] [ 885] 1000 885 7257 120 94208 0 0 ssh-agent
[ 1321.968808] [ 886] 1000 886 87403 143 180224 0 0 at-spi-bus-laun
[ 1321.968824] [ 901] 1000 901 13141 114 143360 0 0 dbus-daemon
[ 1321.968839] [ 904] 1000 904 56371 195 200704 0 0 at-spi2-registr
[ 1321.968855] [ 906] 1000 906 208554 3695 577536 0 0 gnome-terminal-
[ 1321.968870] [ 912] 1000 912 123842 220 212992 0 0 gvfsd
[ 1321.968884] [ 917] 1000 917 89304 144 184320 0 0 gvfsd-fuse
[ 1321.968899] [ 926] 1000 926 205856 368 294912 0 0 xdg-desktop-por
[ 1321.968914] [ 930] 1000 930 157254 159 192512 0 0 xdg-document-po
[ 1321.968929] [ 939] 1000 939 16568 104 172032 0 0 qrexec-client-v
[ 1321.968944] [ 942] 1000 942 117672 116 172032 0 0 xdg-permission-
[ 1321.968960] [ 956] 1000 956 48112 151 147456 0 0 dconf-service
[ 1321.968975] [ 965] 1000 965 63004 2334 143360 0 0 icon-sender
[ 1321.968990] [ 971] 1000 971 122411 246 184320 0 0 gnome-keyring-d
[ 1321.969005] [ 978] 1000 978 176076 1274 495616 0 0 gsd-xsettings
[ 1321.970365] [ 980] 1000 980 120212 120 184320 0 0 agent
[ 1321.970380] [ 986] 1000 986 193424 1221 495616 0 0 xdg-desktop-por
[ 1321.970397] [ 996] 1000 996 128978 340 405504 0 0 pulseaudio
[ 1321.970413] [ 998] 172 998 47729 83 155648 0 0 rtkit-daemon
[ 1321.970428] [ 999] 1000 999 186112 1544 552960 0 0 nm-applet
[ 1321.970449] [ 1005] 998 1005 546542 1542 364544 0 0 polkitd
[ 1321.970463] [ 1023] 1000 1023 16534 101 167936 0 0 qrexec-fork-ser
[ 1321.970478] [ 1025] 1000 1025 52238 16 65536 0 0 sleep
[ 1321.970492] [ 1048] 1000 1048 54375 355 86016 0 0 bash
[ 1321.970506] [ 1088] 0 1088 80071 244 286720 0 0 sudo
[ 1321.970520] [ 1093] 0 1093 53876 63 81920 0 0 dmesg
[ 1321.970533] [ 1096] 1000 1096 54419 397 81920 0 0 bash
[ 1321.970547] [ 1188] 1000 1188 54414 360 90112 0 0 bash
[ 1321.970561] [ 1318] 1000 1318 53923 106 77824 0 0 watch
[ 1321.970578] [ 24371] 1000 24371 2000 20 61440 0 0 stress
[ 1321.970592] [ 24372] 1000 24372 1284819 416405 3403776 0 0 stress
[ 1321.970605] [ 24373] 1000 24373 1284819 424851 3473408 0 0 stress
[ 1321.970619] Out of memory: Kill process 24373 (stress) score 437 or sacrifice child
[ 1321.970636] Killed process 24373 (stress) total-vm:5139276kB, anon-rss:1699404kB, file-rss:0kB, shmem-rss:0kB
[ 1322.014107] oom_reaper: reaped process 24373 (stress), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
I can actually get it to work with 2 workers too, but most of the time it doesn't work.
another non-working 2 threads example:
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2186356 kB for each of the 2 concurrent threads.
MemTotal before: 2796088 kB
stress: info: [32116] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [32116] (415) <-- worker 32117 got signal 9
stress: WARN: [32116] (417) now reaping child worker processes
stress: FAIL: [32116] (451) failed run completed in 2s
real 0m1.781s
user 0m0.163s
sys 0m0.758s
exit code: 1
MemTotal afterwards: 6371684 kB
So, if I run the commands one after the other without pause, then it eventually works (see the one that has exit code 0 the first):
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2276704 kB for each of the 2 concurrent threads.
MemTotal before: 2873412 kB
stress: info: [753] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [753] (415) <-- worker 756 got signal 9
stress: WARN: [753] (417) now reaping child worker processes
stress: FAIL: [753] (451) failed run completed in 0s
real 0m0.405s
user 0m0.116s
sys 0m0.556s
exit code: 1
MemTotal afterwards: 2894940 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2306084 kB for each of the 2 concurrent threads.
MemTotal before: 2894940 kB
stress: info: [824] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [824] (415) <-- worker 826 got signal 9
stress: WARN: [824] (417) now reaping child worker processes
stress: FAIL: [824] (451) failed run completed in 1s
real 0m0.534s
user 0m0.130s
sys 0m0.651s
exit code: 1
MemTotal afterwards: 2894940 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2262092 kB for each of the 2 concurrent threads.
MemTotal before: 2851440 kB
stress: info: [875] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [875] (415) <-- worker 877 got signal 9
stress: WARN: [875] (417) now reaping child worker processes
stress: FAIL: [875] (451) failed run completed in 1s
real 0m0.843s
user 0m0.192s
sys 0m1.054s
exit code: 1
MemTotal afterwards: 3248752 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2970956 kB for each of the 2 concurrent threads.
MemTotal before: 3559536 kB
stress: info: [943] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [943] (415) <-- worker 946 got signal 9
stress: WARN: [943] (417) now reaping child worker processes
stress: FAIL: [943] (451) failed run completed in 1s
real 0m1.009s
user 0m0.221s
sys 0m1.086s
exit code: 1
MemTotal afterwards: 3559536 kB
^[[A[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB forh of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2970904 kB for each of the 2 concurrent threads.
MemTotal before: 3559536 kB
stress: info: [984] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [984] (415) <-- worker 989 got signal 9
stress: WARN: [984] (417) now reaping child worker processes
stress: FAIL: [984] (451) failed run completed in 0s
real 0m0.706s
user 0m0.178s
sys 0m0.897s
exit code: 1
MemTotal afterwards: 3559536 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2970356 kB for each of the 2 concurrent threads.
MemTotal before: 3559536 kB
stress: info: [1041] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1041] (415) <-- worker 1046 got signal 9
stress: WARN: [1041] (417) now reaping child worker processes
stress: FAIL: [1041] (451) failed run completed in 0s
real 0m0.488s
user 0m0.145s
sys 0m0.736s
exit code: 1
MemTotal afterwards: 3559536 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 2970372 kB for each of the 2 concurrent threads.
MemTotal before: 3559536 kB
stress: info: [1069] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
stress: FAIL: [1069] (415) <-- worker 1070 got signal 9
stress: WARN: [1069] (417) now reaping child worker processes
stress: FAIL: [1069] (451) failed run completed in 1s
real 0m0.575s
user 0m0.130s
sys 0m0.676s
exit code: 1
MemTotal afterwards: 3569776 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3084252 kB for each of the 2 concurrent threads.
MemTotal before: 3677868 kB
stress: info: [1094] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
stress: FAIL: [1094] (415) <-- worker 1097 got signal 9
stress: WARN: [1094] (417) now reaping child worker processes
stress: FAIL: [1094] (451) failed run completed in 1s
real 0m0.717s
user 0m0.185s
sys 0m0.985s
exit code: 1
MemTotal afterwards: 4011176 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3423284 kB for each of the 2 concurrent threads.
MemTotal before: 4011176 kB
stress: info: [1119] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[A
stress: FAIL: [1119] (415) <-- worker 1121 got signal 9
stress: WARN: [1119] (417) now reaping child worker processes
stress: FAIL: [1119] (451) failed run completed in 0s
real 0m0.812s
user 0m0.144s
sys 0m0.775s
exit code: 1
MemTotal afterwards: 4011176 kB
^[[A
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3424136 kB for each of the 2 concurrent threads.
MemTotal before: 4011176 kB
stress: info: [1144] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
stress: FAIL: [1144] (415) <-- worker 1146 got signal 9
stress: WARN: [1144] (417) now reaping child worker processes
stress: FAIL: [1144] (451) failed run completed in 0s
real 0m0.516s
user 0m0.128s
sys 0m0.777s
exit code: 1
^[[AMemTotal afterwards: 4011176 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3421196 kB for each of the 2 concurrent threads.
MemTotal before: 4011176 kB
stress: info: [1169] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[Astress: FAIL: [1169] (415) <-- worker 1171 got signal 9
stress: WARN: [1169] (417) now reaping child worker processes
stress: FAIL: [1169] (451) failed run completed in 1s
real 0m0.912s
user 0m0.158s
sys 0m0.744s
exit code: 1
MemTotal afterwards: 4011176 kB
^[[A
^[[A
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3423984 kB for each of the 2 concurrent threads.
MemTotal before: 4011176 kB
stress: info: [1194] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
stress: FAIL: [1194] (415) <-- worker 1196 got signal 9
stress: WARN: [1194] (417) now reaping child worker processes
stress: FAIL: [1194] (451) failed run completed in 1s
real 0m0.735s
user 0m0.179s
sys 0m0.961s
exit code: 1
MemTotal afterwards: 4095144 kB
^[[A
^[[A[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB forh of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3621624 kB for each of the 2 concurrent threads.
MemTotal before: 4215976 kB
stress: info: [1218] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
stress: FAIL: [1218] (415) <-- worker 1219 got signal 9
stress: WARN: [1218] (417) now reaping child worker processes
stress: FAIL: [1218] (451) failed run completed in 1s
real 0m0.678s
user 0m0.174s
sys 0m0.908s
exit code: 1
^[[A
MemTotal afterwards: 4560528 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3972448 kB for each of the 2 concurrent threads.
MemTotal before: 4560528 kB
stress: info: [1244] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[Astress: FAIL: [1244] (415) <-- worker 1245 got signal 9
stress: WARN: [1244] (417) now reaping child worker processes
stress: FAIL: [1244] (451) failed run completed in 1s
real 0m0.715s
user 0m0.186s
sys 0m0.906s
exit code: 1
^[[AMemTotal afterwards: 4560528 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3973028 kB for each of the 2 concurrent threads.
MemTotal before: 4560528 kB
stress: info: [1270] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[Astress: FAIL: [1270] (415) <-- worker 1272 got signal 9
stress: WARN: [1270] (417) now reaping child worker processes
stress: FAIL: [1270] (451) failed run completed in 0s
real 0m0.684s
user 0m0.222s
sys 0m1.006s
exit code: 1
MemTotal afterwards: 4560528 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3970396 kB for each of the 2 concurrent threads.
MemTotal before: 4560528 kB
stress: info: [1298] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[A
stress: FAIL: [1298] (415) <-- worker 1300 got signal 9
stress: WARN: [1298] (417) now reaping child worker processes
stress: FAIL: [1298] (451) failed run completed in 0s
real 0m0.797s
user 0m0.186s
sys 0m1.069s
exit code: 1
^[[AMemTotal afterwards: 4560528 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3971044 kB for each of the 2 concurrent threads.
MemTotal before: 4560528 kB
stress: info: [1325] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[Astress: FAIL: [1325] (415) <-- worker 1327 got signal 9
stress: WARN: [1325] (417) now reaping child worker processes
stress: FAIL: [1325] (451) failed run completed in 1s
real 0m0.616s
user 0m0.171s
sys 0m0.911s
exit code: 1
MemTotal afterwards: 4560528 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 3971396 kB for each of the 2 concurrent threads.
MemTotal before: 4560528 kB
stress: info: [1351] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
^[[A
^[[A
^[[A
^[[A
^[[A
^[[A
^[[A
stress: info: [1351] successful run completed in 10s
real 0m10.092s
user 0m16.627s
sys 0m1.699s
exit code: 0
MemTotal afterwards: 16381624 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 15745188 kB for each of the 2 concurrent threads.
MemTotal before: 16325356 kB
stress: info: [1557] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1557] (415) <-- worker 1558 got signal 9
stress: WARN: [1557] (417) now reaping child worker processes
stress: FAIL: [1557] (451) failed run completed in 6s
real 0m6.457s
user 0m1.788s
sys 0m7.825s
exit code: 1
MemTotal afterwards: 17467788 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16362060 kB for each of the 2 concurrent threads.
MemTotal before: 16942744 kB
stress: info: [1659] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1659] (415) <-- worker 1660 got signal 9
stress: WARN: [1659] (417) now reaping child worker processes
stress: FAIL: [1659] (451) failed run completed in 5s
real 0m5.528s
user 0m1.662s
sys 0m7.539s
exit code: 1
MemTotal afterwards: 18674020 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 18044952 kB for each of the 2 concurrent threads.
MemTotal before: 18626916 kB
stress: info: [1765] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1765] (415) <-- worker 1766 got signal 9
stress: WARN: [1765] (417) now reaping child worker processes
stress: FAIL: [1765] (451) failed run completed in 6s
real 0m5.576s
user 0m1.793s
sys 0m7.949s
exit code: 1
MemTotal afterwards: 16133144 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 15495816 kB for each of the 2 concurrent threads.
MemTotal before: 16077172 kB
stress: info: [1873] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1873] (415) <-- worker 1874 got signal 9
stress: WARN: [1873] (417) now reaping child worker processes
stress: FAIL: [1873] (451) failed run completed in 5s
real 0m4.955s
user 0m1.526s
sys 0m7.215s
exit code: 1
MemTotal afterwards: 15483816 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 14830968 kB for each of the 2 concurrent threads.
MemTotal before: 15415408 kB
stress: info: [1982] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [1982] (415) <-- worker 1983 got signal 9
stress: WARN: [1982] (417) now reaping child worker processes
stress: FAIL: [1982] (451) failed run completed in 6s
real 0m5.820s
user 0m1.502s
sys 0m7.361s
exit code: 1
MemTotal afterwards: 18178180 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 17531964 kB for each of the 2 concurrent threads.
MemTotal before: 18120688 kB
stress: info: [2076] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2076] (415) <-- worker 2078 got signal 9
stress: WARN: [2076] (417) now reaping child worker processes
stress: FAIL: [2076] (451) failed run completed in 6s
real 0m6.745s
user 0m1.940s
sys 0m8.962s
exit code: 1
MemTotal afterwards: 19714468 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19071624 kB for each of the 2 concurrent threads.
MemTotal before: 19661220 kB
stress: info: [2192] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2192] (415) <-- worker 2194 got signal 9
stress: WARN: [2192] (417) now reaping child worker processes
stress: FAIL: [2192] (451) failed run completed in 8s
real 0m7.024s
user 0m1.907s
sys 0m9.509s
exit code: 1
MemTotal afterwards: 19833112 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19173460 kB for each of the 2 concurrent threads.
MemTotal before: 19757336 kB
stress: info: [2318] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2318] (415) <-- worker 2319 got signal 9
stress: WARN: [2318] (417) now reaping child worker processes
stress: FAIL: [2318] (451) failed run completed in 6s
real 0m6.321s
user 0m1.864s
sys 0m8.540s
exit code: 1
MemTotal afterwards: 20153860 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19497672 kB for each of the 2 concurrent threads.
MemTotal before: 20084228 kB
stress: info: [2432] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2432] (415) <-- worker 2433 got signal 9
stress: WARN: [2432] (417) now reaping child worker processes
stress: FAIL: [2432] (451) failed run completed in 7s
real 0m7.878s
user 0m2.029s
sys 0m8.969s
exit code: 1
MemTotal afterwards: 18375064 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 17152152 kB for each of the 2 concurrent threads.
MemTotal before: 17743404 kB
stress: info: [2548] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2548] (415) <-- worker 2549 got signal 9
stress: WARN: [2548] (417) now reaping child worker processes
stress: FAIL: [2548] (451) failed run completed in 7s
real 0m6.798s
user 0m1.804s
sys 0m8.239s
exit code: 1
MemTotal afterwards: 16923772 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16078864 kB for each of the 2 concurrent threads.
MemTotal before: 16672344 kB
stress: info: [2650] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2650] (415) <-- worker 2651 got signal 9
stress: WARN: [2650] (417) now reaping child worker processes
stress: FAIL: [2650] (451) failed run completed in 5s
real 0m5.389s
user 0m1.636s
sys 0m7.452s
exit code: 1
MemTotal afterwards: 16641436 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16044304 kB for each of the 2 concurrent threads.
MemTotal before: 16641436 kB
stress: info: [2762] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2762] (415) <-- worker 2764 got signal 9
stress: WARN: [2762] (417) now reaping child worker processes
stress: FAIL: [2762] (451) failed run completed in 2s
real 0m2.784s
user 0m0.759s
sys 0m3.714s
exit code: 1
MemTotal afterwards: 16641436 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16047480 kB for each of the 2 concurrent threads.
MemTotal before: 16641436 kB
stress: info: [2818] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2818] (415) <-- worker 2819 got signal 9
stress: WARN: [2818] (417) now reaping child worker processes
stress: FAIL: [2818] (451) failed run completed in 4s
real 0m4.341s
user 0m1.464s
sys 0m6.742s
exit code: 1
MemTotal afterwards: 17788520 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 17190152 kB for each of the 2 concurrent threads.
MemTotal before: 17788520 kB
stress: info: [2910] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2910] (415) <-- worker 2912 got signal 9
stress: WARN: [2910] (417) now reaping child worker processes
stress: FAIL: [2910] (451) failed run completed in 3s
real 0m2.689s
user 0m0.847s
sys 0m4.064s
exit code: 1
MemTotal afterwards: 17788520 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 17190632 kB for each of the 2 concurrent threads.
MemTotal before: 17788520 kB
stress: info: [2970] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [2970] (415) <-- worker 2972 got signal 9
stress: WARN: [2970] (417) now reaping child worker processes
stress: FAIL: [2970] (451) failed run completed in 4s
real 0m3.804s
user 0m1.219s
sys 0m5.570s
exit code: 1
MemTotal afterwards: 17540948 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16942580 kB for each of the 2 concurrent threads.
MemTotal before: 17540948 kB
stress: info: [3052] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3052] (415) <-- worker 3054 got signal 9
stress: WARN: [3052] (417) now reaping child worker processes
stress: FAIL: [3052] (451) failed run completed in 2s
real 0m2.676s
user 0m0.863s
sys 0m4.028s
exit code: 1
MemTotal afterwards: 17540948 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 16943380 kB for each of the 2 concurrent threads.
MemTotal before: 17540948 kB
stress: info: [3112] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3112] (415) <-- worker 3113 got signal 9
stress: WARN: [3112] (417) now reaping child worker processes
stress: FAIL: [3112] (451) failed run completed in 6s
real 0m5.321s
user 0m1.263s
sys 0m6.314s
exit code: 1
MemTotal afterwards: 19906576 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19264896 kB for each of the 2 concurrent threads.
MemTotal before: 19851280 kB
stress: info: [3200] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3200] (415) <-- worker 3202 got signal 9
stress: WARN: [3200] (417) now reaping child worker processes
stress: FAIL: [3200] (451) failed run completed in 6s
real 0m6.677s
user 0m2.033s
sys 0m9.527s
exit code: 1
MemTotal afterwards: 19844072 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19184644 kB for each of the 2 concurrent threads.
MemTotal before: 19760356 kB
stress: info: [3324] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3324] (415) <-- worker 3326 got signal 9
stress: WARN: [3324] (417) now reaping child worker processes
stress: FAIL: [3324] (451) failed run completed in 7s
real 0m6.355s
user 0m1.900s
sys 0m8.847s
exit code: 1
MemTotal afterwards: 20103872 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19462908 kB for each of the 2 concurrent threads.
MemTotal before: 20048576 kB
stress: info: [3440] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3440] (415) <-- worker 3442 got signal 9
stress: WARN: [3440] (417) now reaping child worker processes
stress: FAIL: [3440] (451) failed run completed in 7s
real 0m6.720s
user 0m2.024s
sys 0m9.092s
exit code: 1
MemTotal afterwards: 19857992 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 19218008 kB for each of the 2 concurrent threads.
MemTotal before: 19803044 kB
stress: info: [3560] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3560] (415) <-- worker 3562 got signal 9
stress: WARN: [3560] (417) now reaping child worker processes
stress: FAIL: [3560] (451) failed run completed in 6s
real 0m6.065s
user 0m1.927s
sys 0m8.394s
exit code: 1
MemTotal afterwards: 19167804 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$ timeout=10s threads=2 alloc="$(awk '/MemAvailable/{printf "%d\n", $2 + 4000;}' < /proc/meminfo)"; echo "Will alloc: $alloc kB for each of the $threads concurrent threads."; echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' < /proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m $threads --timeout $timeout ; echo "exit code: $?" ; awk '/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' < /proc/meminfo
Will alloc: 18531704 kB for each of the 2 concurrent threads.
MemTotal before: 19116296 kB
stress: info: [3676] dispatching hogs: 0 cpu, 0 io, 2 vm, 0 hdd
stress: FAIL: [3676] (415) <-- worker 3677 got signal 9
stress: WARN: [3676] (417) now reaping child worker processes
stress: FAIL: [3676] (451) failed run completed in 5s
real 0m5.587s
user 0m1.921s
sys 0m7.925s
exit code: 1
MemTotal afterwards: 20113808 kB
[user@dev01-w-s-f-fdr28 rpmbuild]$
So, uhm, what am I missing ? Is it a known issue with mem balooning or what is that memory balancing called? could I be missing something or doing it wrong?(although to me is seems unlikely at the moment)
Marek Marczykowski-Górecki
Hash: SHA256
On Wed, Feb 06, 2019 at 07:38:19AM -0800, Marcus Linsner wrote:
> I have an AppVM with
> Initial Memory: 2400 MB
> Max memory 24000 MB
> [v] Include in memory balancing
>
> has no swap enabled (swapon reports nothing) and uses an official kernel(eg. not compiled by me):
more RAM (*) and this is the reason why VMs in Qubes have swap by
default enabled.
If you need a lot of RAM allocated quickly (for example it applies for
Android build), better assign it initially and disable memory balancing
for this VM.
(*) Actually there are two parts of this: first detecting that qube
needs more RAM - it is reported with 10Hz frequency, as a trade-off
between allocation speed and system load caused by qmemman itself.
Secondly, it needs some time to actually get this memory, potentially
reclaiming it from other qubes first.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlxbAVgACgkQ24/THMrX
1ywz+Af+Ptpbd2wooVH9Knxd+DAzIAanX/wLwenKDwXc+WNIB5DdY8V4jMBgj2Fg
UDxJGQYgUT4HAnXGbtXT7QF7hp8BDfVVNf+ORARvSiU+O6EqCKMvlJXy/5/XdG5V
eNJMPVcCVIrQKMgumjBKZqDNjqf0vdq9ee9wJKoc+RsxCUMkXprSANJyzew5q/xj
hpjv9YMthLjrIBOVcSY/+IrRfG8FzJpbrv7uDMwJoQm5/LjB6FKz/+sMaleLpHKv
tBOt1Uf1auR1qVdDHZDqfRamTn0Y92VNfbfZVmOkj/fohtlcs0ecJjEDwMgmS6Mb
6jPxYZAeHt96ygN32CVPljW23tVBdw==
=bZbJ
-----END PGP SIGNATURE-----
Marcus Linsner
<marm...@invisiblethingslab.com> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Wed, Feb 06, 2019 at 07:38:19AM -0800, Marcus Linsner wrote:
> > I have an AppVM with
> > Initial Memory: 2400 MB
> > Max memory 24000 MB
> > [v] Include in memory balancing
> >
> > has no swap enabled (swapon reports nothing) and uses an official kernel(eg. not compiled by me):
>
> Lack of swap may be an issue here. Qmemman needs some time to assign
> more RAM (*) and this is the reason why VMs in Qubes have swap by
> default enabled.
> If you need a lot of RAM allocated quickly (for example it applies for
> Android build), better assign it initially and disable memory balancing
> for this VM.
>
> (*) Actually there are two parts of this: first detecting that qube
> needs more RAM - it is reported with 10Hz frequency, as a trade-off
> between allocation speed and system load caused by qmemman itself.
> Secondly, it needs some time to actually get this memory, potentially
> reclaiming it from other qubes first.
>
I'm now able to always reproduce the disk thrashing(disk reading only,
no writing) that happens before OOM-killer even triggers (in the case
with no swap), with:
[user@dev01-w-s-f-fdr28 ~]$ timeout=10s threads=1 alloc="$(awk
"Will alloc: $alloc kB for each of the $threads concurrent threads.";
echo "MemTotal before: $(awk '/MemTotal/{printf "%d kB\n", $2;}' <
/proc/meminfo)";time stress --vm-bytes "${alloc}k" --vm-keep -m
$threads --timeout $timeout ; echo "exit code: $?" ; awk
'/MemTotal/{printf "MemTotal afterwards: %d kB\n", $2;}' <
/proc/meminfo
MemTotal before: 14002196 kB
stress: info: [2191] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: info: [2191] successful run completed in 10s
real 0m10.268s
user 0m7.528s
sys 0m2.664s
exit code: 0
MemTotal afterwards: 14002196 kB
of course this is a kernel issue due to "Active(file)" (seen in
/proc/meminfo) being evicted and re-read for continuing the execution
of the active processes that are still running on the system...
I'm going to attempt to find a way(ie. kernel patch) to keep a minimum
"Active(file)" in order to avoid disk thrashing(and thus allow
OOM-Killer to trigger), as suggested to me by Mikko Rantalainen in the
comment of https://stackoverflow.com/q/52067753
but wish me luck, since I r no programmah :) ie. any hints are more than welcome
Cheers!
Dupéron Georges
On a non-Qubes system, I used to press "alt+sysrq f" to manually invoke the OOM killer when the disk would start thrashing.
Worked like a charm, but the OOM picks a random process so it can kill e.g. Xorg if you're unlucky.
This would work for dom0 in Qubes, but I don't know how you would trigger it in a VM.
Cheers,
Georges Dupéron
Marek Marczykowski-Górecki
Hash: SHA256
> This would work for dom0 in Qubes, but I don't know how you would trigger
> it in a VM.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
1yxTSAf/TV+9Z0rYgU87IuVBvabhOnV+heGjMCJNOJBH1okuW+R8jKSjJeZu5z6q
pwQxHtNK3BwTIsXh7iMdAvubFyBE2TqWl6W96IQEIxTx055qSOJE6KrCFZEonTm/
5N4pSRXly94CNsVkoUFcsXZfP9D0g5q9l6ss6tYDrQFFG/rX2+ymUR+PoswmkdxL
/CRDxvp0rdk00XGC9do+CFZNEM7Ge+2bUW9IBE8ZCxJnKTR5ZF8lHMxsDp1ZNACw
J0fz1P+xNW2HWU3A+x16w85VP0+5J5juyJPbTTOw8Y+24898a9dkgScwGXp/6nVG
LkI9rRp2+keX6J7hU10Hk0PuRlcd2g==
=ZiXm
-----END PGP SIGNATURE-----
Chris Laprise
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On Wed, Feb 06, 2019 at 06:29:22PM +0100, Dupéron Georges wrote:
>> This would work for dom0 in Qubes, but I don't know how you would trigger
>> it in a VM.
>
> There is `xl sysrq` which you can use to inject sysrq to a VM.
currently active window with 'xdotool getwindowfocus', for example.
> (*) Actually there are two parts of this: first detecting that qube
> needs more RAM - it is reported with 10Hz frequency, as a trade-off
> between allocation speed and system load caused by qmemman itself.
> Secondly, it needs some time to actually get this memory, potentially
> reclaiming it from other qubes first.
Basically, we have a race condition where the delays incurred by swap
are used to buy time for qmemman.
If I were more familiar with the subject, I might propose a memory
allocation method that is synchronous and therefore more deterministic.
--
Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886
Marek Marczykowski-Górecki
Hash: SHA256
> On 2/6/19 1:07 PM, Marek Marczykowski-Górecki wrote:
> > (*) Actually there are two parts of this: first detecting that qube
> > needs more RAM - it is reported with 10Hz frequency, as a trade-off
> > between allocation speed and system load caused by qmemman itself.
> > Secondly, it needs some time to actually get this memory, potentially
> > reclaiming it from other qubes first.
>
> There seems to be a fundamental flaw in this asynchronous method. Basically,
> we have a race condition where the delays incurred by swap are used to buy
> time for qmemman.
>
> If I were more familiar with the subject, I might propose a memory
> allocation method that is synchronous and therefore more deterministic.
allocations), I think it will overall reduce performance in most common
cases (not so rapid memory allocations). Adding synchronization to
memory allocation, would make it slower, including common use cases.
Qmemman, also try to maintain 30% margin of free memory inside each VM
(configurable in /etc/qubes/qmemman.conf), so it isn't only swap used
to buy some time.
Also note that VM have no guarantee it will get that memory (especially
if it's already at its maxmem), so need to be prepared to deal with such
cases too.
Linux memory subsystem generally works better if there is _some_ swap.
It is used for example to keep memory pages that process do not use for
a while (especially useful if application have some memory leaks, which
is unfortunately not that rare...).
Anyway, if you have some process rapidly requiring a lot of RAM, it may
be better to disable memory balancing for that qube, regardless if it's
fast enough or not. Constantly reassigning large amount of memory will
take some CPU time, that could be spent on the process itself otherwise.
For example we recommend disabling it for building Qubes OS ISO, here:
https://www.qubes-os.org/doc/qubes-r3-building/
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
1yx+nAf3Yy/CCjmpWs5gqcMGkMxrlDQp5lxQb32y+zAqGZ2L7e+in5KouXqXyP0z
CT4k4vb1c+f/poNaeooyEzazUca4Q+Ck6WRb5Hli1h54XbogNR3dbmX9xlXBI/cH
i+SGWPbAt+Wm8LWCn7k25Z45jYnSBD04t0m/EG0UmKG+YfdH9T/8aSxbAeSimhed
9Fj/6dyyHjRZLWucEwQ2vQVTD5/fJOHIAnfAXxUXjE+EjX1q++GX+jyVINwpHWBA
toBw77b4chHmF7G+auQgvTUQeDeOaTxARY2S6Obmpr+DXjP08gYl8d8teCosYlYt
ifykmxhleUqaT8VI6kanAMXWzrHC
=QoEb
-----END PGP SIGNATURE-----