Salt Master Memory Usage

[Charles Baker]

Looks to me like the salt-master just keeps using more and more memory until it nearly exhausts all available memory. Also, this same server has had zombie salt master processes. Anyone seen this?

[ec2-user@i-fed5e648 ~]$ ps aux | grep salt-master

root      3826  0.0  5.1 1255836 210252 ?      SNl  19:49   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      3829  0.0  5.3 1265308 218508 ?      SNl  19:49   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      3830  0.0  5.1 1255768 210152 ?      SNl  19:49   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      3841  0.0  5.3 1265760 215484 ?      SNl  19:49   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      3842  0.0  5.2 1255768 212260 ?      SNl  19:49   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5247  0.0  5.2 1255520 213336 ?      SNl  20:00   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5898  0.0  0.5 298988 23856 ?        SN   10:48   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5900  0.0  0.6 385012 24568 ?        SNl  10:48   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5901  0.0  0.8 385044 33840 ?        SNl  10:48   0:10 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5904  0.0  0.5 299120 24244 ?        SN   10:48   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root      5952  0.0  1.2 1478608 49476 ?       SNl  10:48   0:16 /usr/bin/python2.6 /usr/bin/salt-master -d

root      8511  0.0  0.8 1078436 35364 ?       SNl  20:19   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      8512  0.0  0.8 1077748 34516 ?       SNl  20:19   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      8527  0.0  0.8 1078460 33768 ?       SNl  20:19   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      8529  0.0  0.8 1078432 32624 ?       SNl  20:19   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

root      8579  0.0  0.8 1078440 34916 ?       SNl  20:19   0:00 /usr/bin/python2.6 /usr/bin/salt-master -d

ec2-user 10605  0.0  0.0 103312   856 pts/1    S+   20:39   0:00 grep salt-master

root     31054  0.0  9.7 1443416 397184 ?      SNl  19:19   0:02 /usr/bin/python2.6 /usr/bin/salt-master -d

root     31266  0.0  9.6 1439396 391820 ?      SNl  19:19   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root     31268  0.0  9.7 1439120 393756 ?      SNl  19:19   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root     31274  0.0  9.7 1439156 393688 ?      SNl  19:19   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

root     31673  0.0  9.8 1443100 398720 ?      SNl  19:20   0:01 /usr/bin/python2.6 /usr/bin/salt-master -d

[ec2-user@i-fed5e648 ~]$ free -m

             total       used       free     shared    buffers     cached

Mem:          3958       3815        142          0          2         17

-/+ buffers/cache:       3795        162

Swap:            0          0          0

[ec2-user@i-fed5e648 ~]$ sudo service salt-master restart

Stopping salt-master daemon:                               [  OK  ]

Starting salt-master daemon:                               [  OK  ]

[ec2-user@i-fed5e648 ~]$ free -m

             total       used       free     shared    buffers     cached

Mem:          3958       1046       2911          0         17        108

-/+ buffers/cache:        921       3036

Swap:            0          0          0

[ec2-user@i-fed5e648 ~]$ date

Wed Mar 23 20:40:43 UTC 2016

[ec2-user@i-fed5e648 ~]$ uptime

 20:41:05 up 86 days,  8:38,  1 user,  load average: 3.71, 2.82, 2.48

[ec2-user@i-fed5e648 ~]$ salt-call --versions-report

           Salt: 2015.5.8

         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)

         Jinja2: 2.2.1

       M2Crypto: 0.20.2

 msgpack-python: 0.4.6

   msgpack-pure: Not Installed

       pycrypto: 2.6.1

        libnacl: Not Installed

         PyYAML: 3.10

          ioflo: Not Installed

          PyZMQ: 14.5.0

           RAET: Not Installed

            ZMQ: 4.0.5

           Mako: Not Installed

        Tornado: 4.2.1

        timelib: Not Installed

       dateutil: 2.4.2

[ec2-user@i-fed5e648 ~]$

A sample error message in /var/log/messages:

Mar 23 20:19:05 ip-10-21-0-239 kernel: [ 8529]     0  8529    96563     6028   1       0             0 salt-master

Mar 23 20:19:05 ip-10-21-0-239 kernel: [ 8534]     0  8534    34497     1594   1       0             0 repoquery

Mar 23 20:19:05 ip-10-21-0-239 kernel: Out of memory: Kill process 31267 (salt-master) score 65 or sacrifice child

Mar 23 20:19:05 ip-10-21-0-239 kernel: Killed process 31267, UID 0, (salt-master) total-vm:1100276kB, anon-rss:386788kB, file-rss:12kB

--

Charles H. Baker
864.990.1297

Knowing is not enough; we must apply. Willing is not enough; we must do. Bruce Lee

[Dmitry Kuzmenko]

Hello,

It looks like a known bug: https://github.com/saltstack/salt/issues/31454
I'm working on it.

--
Best regards,
Dmitry

Dmitry Kuzmenko | Senior Software Engineer | DSR Corporation | e-mail: dmitry....@dsr-company.com | skype: dmitry.a.kuzmenko

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.