Salt out of memory
230 views
Skip to first unread message
kindule
Jun 5, 2013, 9:05:13 AM6/5/13
to salt-...@googlegroups.com
Hi,
Recently, we use salt as our configure center. But facing some problem. It complained "out of memory".
Platfrom: Xen Guest
OS: Linux saltserver 3.8.0-19-generic #30-Ubuntu SMP Wed May 1 16:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
And the dmesg error log
[285454.714934] Mem-Info:
[285454.714935] Node 0 DMA per-cpu:
[285454.714938] CPU 0: hi: 0, btch: 1 usd: 0
[285454.714939] Node 0 DMA32 per-cpu:
[285454.714941] CPU 0: hi: 186, btch: 31 usd: 101
[285454.714944] active_anon:223351 inactive_anon:56 isolated_anon:0
[285454.714944] active_file:9 inactive_file:52 isolated_file:0
[285454.714944] unevictable:0 dirty:0 writeback:0 unstable:0
[285454.714944] free:12218 slab_reclaimable:2378 slab_unreclaimable:2915
[285454.714944] mapped:30 shmem:64 pagetables:1584 bounce:0
[285454.714944] free_cma:0
[285454.714948] Node 0 DMA free:4540kB min:704kB low:880kB high:1056kB active_anon:11156kB inactive_anon:0kB active_file:0kB inactive_file:108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31 all_unreclaimable? yes
[285454.714952] lowmem_reserve[]: 0 960 960 960
[285454.714955] Node 0 DMA32 free:44332kB min:44348kB low:55432kB high:66520kB active_anon:882248kB inactive_anon:224kB active_file:36kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:983808kB managed:949784kB mlocked:0kB dirty:0kB writeback:0kB mapped:104kB shmem:256kB slab_reclaimable:9500kB slab_unreclaimable:11660kB kernel_stack:1776kB pagetables:6320kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:394 all_unreclaimable? yes
[285454.714959] lowmem_reserve[]: 0 0 0 0
[285454.714961] Node 0 DMA: 5*4kB (U) 5*8kB (UEM) 4*16kB (UEM) 6*32kB (UM) 2*64kB (U) 2*128kB (UM) 3*256kB (UM) 2*512kB (UM) 0*1024kB 1*2048kB (R) 0*4096kB = 4540kB
[285454.714972] Node 0 DMA32: 197*4kB (UEM) 125*8kB (UEM) 143*16kB (UEM) 98*32kB (UE) 82*64kB (UEM) 57*128kB (UEM) 34*256kB (UEM) 17*512kB (UEM) 3*1024kB (UE) 0*2048kB 1*4096kB (R) = 44332kB
[285454.714981] 133 total pagecache pages
[285454.714982] 0 pages in swap cache
[285454.714984] Swap cache stats: add 0, delete 0, find 0/0
[285454.714985] Free swap = 0kB
[285454.714985] Total swap = 0kB
[285454.717953] 253936 pages RAM
[285454.717956] 7758 pages reserved
[285454.717957] 281157 pages shared
[285454.717958] 230061 pages non-shared
[285454.717959] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[285454.717969] [ 253] 0 253 3817 51 12 0 0 upstart-file-br
[285454.717972] [ 326] 0 326 4329 71 13 0 0 upstart-udev-br
[285454.717974] [ 336] 0 336 5388 130 16 0 -1000 udevd
[285454.717977] [ 373] 101 373 61863 97 23 0 0 rsyslogd
[285454.717979] [ 476] 0 476 5387 121 15 0 -1000 udevd
[285454.717981] [ 477] 0 477 5387 131 15 0 -1000 udevd
[285454.717983] [ 505] 102 505 5996 95 16 0 0 dbus-daemon
[285454.717986] [ 667] 0 667 3814 51 12 0 0 upstart-socket-
[285454.717989] [ 855] 0 855 13063 151 32 0 -1000 sshd
[285454.717991] [ 1042] 0 1042 3637 42 12 0 0 getty
[285454.717993] [ 1065] 0 1065 3637 41 12 0 0 getty
[285454.717995] [ 1075] 0 1075 3637 42 12 0 0 getty
[285454.717997] [ 1076] 0 1076 3637 40 12 0 0 getty
[285454.717999] [ 1080] 0 1080 3637 42 12 0 0 getty
[285454.718001] [ 1084] 0 1084 1094 34 8 0 0 acpid
[285454.718004] [ 1133] 0 1133 5331 53 16 0 0 cron
[285454.718006] [ 1134] 0 1134 4781 44 12 0 0 atd
[285454.718009] [ 1212] 103 1212 83794 305 62 0 0 whoopsie
[285454.718011] [ 1232] 0 1232 3199 37 12 0 0 getty
[285454.718013] [ 1234] 0 1234 3637 41 11 0 0 getty
[285454.718015] [ 1235] 0 1235 3199 37 12 0 0 getty
[285454.718017] [ 1241] 0 1241 123751 3978 68 0 0 salt-master
[285454.718019] [ 1242] 0 1242 37340 5224 75 0 0 salt-master
[285454.718021] [ 1249] 0 1249 58215 3726 64 0 0 salt-master
[285454.718023] [ 1250] 0 1250 58281 3932 64 0 0 salt-master
[285454.718025] [ 1251] 0 1251 178616 44392 169 0 0 salt-master
[285454.718027] [ 1252] 0 1252 178619 44373 169 0 0 salt-master
[285454.718029] [ 1253] 0 1253 178648 44401 169 0 0 salt-master
[285454.718031] [ 1254] 0 1254 178654 44399 169 0 0 salt-master
[285454.718033] [ 1255] 0 1255 178618 44391 169 0 0 salt-master
[285454.718035] [ 1548] 0 1548 6300 93 16 0 0 ntpd
[285454.718038] [ 1549] 108 1549 5243 91 15 0 0 ntpd
[285454.718040] [ 1558] 0 1558 146107 278 37 0 0 console-kit-dae
[285454.718042] [ 1625] 0 1625 48267 177 34 0 0 polkitd
[285454.718045] Out of memory: Kill process 1253 (salt-master) score 151 or sacrifice child
[285454.718112] Killed process 1253 (salt-master) total-vm:714592kB, anon-rss:177604kB, file-rss:0kB
Recently, we use salt as our configure center. But facing some problem. It complained "out of memory".
Platfrom: Xen Guest
OS: Linux saltserver 3.8.0-19-generic #30-Ubuntu SMP Wed May 1 16:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
And the dmesg error log
[285454.714934] Mem-Info:
[285454.714935] Node 0 DMA per-cpu:
[285454.714938] CPU 0: hi: 0, btch: 1 usd: 0
[285454.714939] Node 0 DMA32 per-cpu:
[285454.714941] CPU 0: hi: 186, btch: 31 usd: 101
[285454.714944] active_anon:223351 inactive_anon:56 isolated_anon:0
[285454.714944] active_file:9 inactive_file:52 isolated_file:0
[285454.714944] unevictable:0 dirty:0 writeback:0 unstable:0
[285454.714944] free:12218 slab_reclaimable:2378 slab_unreclaimable:2915
[285454.714944] mapped:30 shmem:64 pagetables:1584 bounce:0
[285454.714944] free_cma:0
[285454.714948] Node 0 DMA free:4540kB min:704kB low:880kB high:1056kB active_anon:11156kB inactive_anon:0kB active_file:0kB inactive_file:108kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15648kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:16kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31 all_unreclaimable? yes
[285454.714952] lowmem_reserve[]: 0 960 960 960
[285454.714955] Node 0 DMA32 free:44332kB min:44348kB low:55432kB high:66520kB active_anon:882248kB inactive_anon:224kB active_file:36kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:983808kB managed:949784kB mlocked:0kB dirty:0kB writeback:0kB mapped:104kB shmem:256kB slab_reclaimable:9500kB slab_unreclaimable:11660kB kernel_stack:1776kB pagetables:6320kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:394 all_unreclaimable? yes
[285454.714959] lowmem_reserve[]: 0 0 0 0
[285454.714961] Node 0 DMA: 5*4kB (U) 5*8kB (UEM) 4*16kB (UEM) 6*32kB (UM) 2*64kB (U) 2*128kB (UM) 3*256kB (UM) 2*512kB (UM) 0*1024kB 1*2048kB (R) 0*4096kB = 4540kB
[285454.714972] Node 0 DMA32: 197*4kB (UEM) 125*8kB (UEM) 143*16kB (UEM) 98*32kB (UE) 82*64kB (UEM) 57*128kB (UEM) 34*256kB (UEM) 17*512kB (UEM) 3*1024kB (UE) 0*2048kB 1*4096kB (R) = 44332kB
[285454.714981] 133 total pagecache pages
[285454.714982] 0 pages in swap cache
[285454.714984] Swap cache stats: add 0, delete 0, find 0/0
[285454.714985] Free swap = 0kB
[285454.714985] Total swap = 0kB
[285454.717953] 253936 pages RAM
[285454.717956] 7758 pages reserved
[285454.717957] 281157 pages shared
[285454.717958] 230061 pages non-shared
[285454.717959] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[285454.717969] [ 253] 0 253 3817 51 12 0 0 upstart-file-br
[285454.717972] [ 326] 0 326 4329 71 13 0 0 upstart-udev-br
[285454.717974] [ 336] 0 336 5388 130 16 0 -1000 udevd
[285454.717977] [ 373] 101 373 61863 97 23 0 0 rsyslogd
[285454.717979] [ 476] 0 476 5387 121 15 0 -1000 udevd
[285454.717981] [ 477] 0 477 5387 131 15 0 -1000 udevd
[285454.717983] [ 505] 102 505 5996 95 16 0 0 dbus-daemon
[285454.717986] [ 667] 0 667 3814 51 12 0 0 upstart-socket-
[285454.717989] [ 855] 0 855 13063 151 32 0 -1000 sshd
[285454.717991] [ 1042] 0 1042 3637 42 12 0 0 getty
[285454.717993] [ 1065] 0 1065 3637 41 12 0 0 getty
[285454.717995] [ 1075] 0 1075 3637 42 12 0 0 getty
[285454.717997] [ 1076] 0 1076 3637 40 12 0 0 getty
[285454.717999] [ 1080] 0 1080 3637 42 12 0 0 getty
[285454.718001] [ 1084] 0 1084 1094 34 8 0 0 acpid
[285454.718004] [ 1133] 0 1133 5331 53 16 0 0 cron
[285454.718006] [ 1134] 0 1134 4781 44 12 0 0 atd
[285454.718009] [ 1212] 103 1212 83794 305 62 0 0 whoopsie
[285454.718011] [ 1232] 0 1232 3199 37 12 0 0 getty
[285454.718013] [ 1234] 0 1234 3637 41 11 0 0 getty
[285454.718015] [ 1235] 0 1235 3199 37 12 0 0 getty
[285454.718017] [ 1241] 0 1241 123751 3978 68 0 0 salt-master
[285454.718019] [ 1242] 0 1242 37340 5224 75 0 0 salt-master
[285454.718021] [ 1249] 0 1249 58215 3726 64 0 0 salt-master
[285454.718023] [ 1250] 0 1250 58281 3932 64 0 0 salt-master
[285454.718025] [ 1251] 0 1251 178616 44392 169 0 0 salt-master
[285454.718027] [ 1252] 0 1252 178619 44373 169 0 0 salt-master
[285454.718029] [ 1253] 0 1253 178648 44401 169 0 0 salt-master
[285454.718031] [ 1254] 0 1254 178654 44399 169 0 0 salt-master
[285454.718033] [ 1255] 0 1255 178618 44391 169 0 0 salt-master
[285454.718035] [ 1548] 0 1548 6300 93 16 0 0 ntpd
[285454.718038] [ 1549] 108 1549 5243 91 15 0 0 ntpd
[285454.718040] [ 1558] 0 1558 146107 278 37 0 0 console-kit-dae
[285454.718042] [ 1625] 0 1625 48267 177 34 0 0 polkitd
[285454.718045] Out of memory: Kill process 1253 (salt-master) score 151 or sacrifice child
[285454.718112] Killed process 1253 (salt-master) total-vm:714592kB, anon-rss:177604kB, file-rss:0kB
Mina Naguib
Jun 5, 2013, 9:10:41 AM6/5/13
to salt-...@googlegroups.com
178MB RSS for the salt master is high, but not unreasonable. I just checked my medium-sized installation and each salt master is at ~50MB RSS
The linux OOM killer is doing it's job - when the box is out of memory it kills the process that it thinks will best solve that situation.
How much RAM does the VM have ? Is it running other services ? What are the characteristics of the salt installation ?
> You received this message because you are subscribed to the Google Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
Ollie Walsh
Jun 5, 2013, 12:10:22 PM6/5/13
to salt-...@googlegroups.com
[285454.714985] Free swap = 0kB
[285454.714985] Total swap = 0kBIf you run without swap you get OOM errors.
Cheers,
Ollie
Lonni J Friedman
Jun 5, 2013, 12:14:20 PM6/5/13
to salt-...@googlegroups.com
That's not quite a fair statement. Even if you have swap, if your
processes demand more memory than RAM+swap can provide, you'll hit an
OOM condition. If salt needs to start digging into swap just to stay
alive then something is still very broken somewhere, and performance
will be awful. Adding swap is merely a band-aid for whatever is
causing so much memory to be required.
processes demand more memory than RAM+swap can provide, you'll hit an
OOM condition. If salt needs to start digging into swap just to stay
alive then something is still very broken somewhere, and performance
will be awful. Adding swap is merely a band-aid for whatever is
causing so much memory to be required.
Ollie Walsh
Jun 8, 2013, 5:38:38 AM6/8/13
to salt-...@googlegroups.com
That's not how it works.
Lots of gory detail here: http://www.win.tue.nl/~aeb/linux/lk/lk-9.html
In a nutshell: The linux kernel is optimistic and allows more virtual memory than RAM+swap to be allocated.When/if a page of this virtual memory is used the kernel take it from the free RAM. If there is no free RAM it must swap out the least recently used pages. If there is no free swap the OOM killer runs.
Running with no swap forces a first-come-first served memory memory management scheme instead of allowing the kernel to optimizing how it is used given the current load.
Cheers,
Ollie
Reply all
Reply to author
Forward
0 new messages