Reducing Salt Minion idle memory usage

[Peter Baumgartner]

I am using salt-minion on LXC containers and the minion processes chew up a non-trivial amount of RAM on the host:

$ sudo lxc-ls -1 | wc -l    # number of containers

44

$ ps -C salt-minion --no-headers | wc -l    # number of salt-minion processes (including host)

135

$ ps -C salt-minion --no-headers -o rss | awk '{s+=$1} END {print s}'    # RSS (in kb) used by salt-minion

5381244

That's over 5GB for 44 (+1) minions.

Are there any configuration values I can tune to reduce this?

Would it be possible to use something like systemd's socket activation to stop the minion services if they are idle?

[C. R. Oldham]

Hi Peter,

Don't you want to subtract out the memory that is shared by those processes so that it isn't counted more than once?  Or use /proc/{pid}/smaps to add up all the PSS values for all the minions?

Or do shared libraries not get shared as much across LXC containers?

--cro

C. R. Oldham

Senior Platform Engineer, SaltStack

c...@saltstack.com / 801-564-4673

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/6f26f041-bdea-4563-9d93-5e571163361c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Baumgartner]

Adding up PSS results in only a slightly smaller size, about 4GB

On Thu, Jun 22, 2017 at 6:22 PM, C. R. Oldham <c...@saltstack.com> wrote:
> Hi Peter,
>
> Don't you want to subtract out the memory that is shared by those processes
> so that it isn't counted more than once?

Good point, I didn't realize summing RSS would double count some memory.

> Or use /proc/{pid}/smaps to add up
> all the PSS values for all the minions?

I end up with a slightly smaller number, but still quite large at 4GB.

>
> Or do shared libraries not get shared as much across LXC containers?

According to this, the answer appears to be that they are isolated
https://github.com/moby/moby/issues/7950

>
> --cro
> C. R. Oldham
> Senior Platform Engineer, SaltStack
> c...@saltstack.com / 801-564-4673
>
> On Thu, Jun 22, 2017 at 4:53 PM, Peter Baumgartner <pe...@lincolnloop.com>
> wrote:
>>
>> I am using salt-minion on LXC containers and the minion processes chew up
>> a non-trivial amount of RAM on the host:
>>
>> $ sudo lxc-ls -1 | wc -l # number of containers
>> 44
>> $ ps -C salt-minion --no-headers | wc -l # number of salt-minion
>> processes (including host)
>> 135
>> $ ps -C salt-minion --no-headers -o rss | awk '{s+=$1} END {print s}' #
>> RSS (in kb) used by salt-minion
>> 5381244
>>
>> That's over 5GB for 44 (+1) minions.
>>
>> Are there any configuration values I can tune to reduce this?
>> Would it be possible to use something like systemd's socket activation to
>> stop the minion services if they are idle?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Salt-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to salt-users+...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/salt-users/6f26f041-bdea-4563-9d93-5e571163361c%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Salt-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/salt-users/hvcKV3Qo_t0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> salt-users+...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/salt-users/CACigG6UnQFefwuEG%2BkysZu%2BqA_k-Nqk0fH60nw4Vs7SE9mURpg%40mail.gmail.com.

[Peter Baumgartner]

Hi! Any further thoughts on this? I'm attaching a visualization of the
memory usage as well.

On Thu, Jun 22, 2017 at 10:10 PM, Peter Baumgartner