Salt Master fails after installing lots of minions

[Roger Criddle]

I had a functioning salt master.    After installing minions to several thousand servers, the master is no longer responding.      For example, when running a simple test.ping I get the following error:

[root]# salt "test.ldschurch.org" test.ping
[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate, is this user permitted to execute commands?

I have tried increasing the worker threads, and the max_open_files.    The only lines in my master file are:
interface: my_ip_address
max_open_files: 15000
worker_threads: 20

What settings need to be configured on my master to allow management of 5000 servers?    Or is that test.ping error indicative of some other problem?

[Volker]

Hi Roger,

> I have tried increasing the worker threads, and the max_open_files.
> The only lines in my master file are: interface: my_ip_address
> max_open_files: 15000 worker_threads: 20
>

You should add a few more on the minions end. Have a look at the
minion config and the following settings:

random_reauth_delay
recon_default
recon_max
recon_randomize

These settings basically stagger the minion reconnects and the way the
zeromq socket (re)connect to the master. Without tweaking those, you
will have 5000 minions trying to (re)connect to the master at once.

To prevent 5000 minions trying to answer at once, you should have a
look at the batch mode. Otherwise your master will simply explode,
even with a test.ping. I assume that is what you're already seeing.

> What settings need to be configured on my master to allow
> management of 5000 servers? Or is that test.ping error
> indicative of some other problem?

Without a lot of tweaking you will have trouble running that many
minions on a single master for various reason.

1. 5000 zmq sockets connecting to the master lead to a syn flood on
many many linux distros. Im not aware of any setting (sysctl, etc.)
that will prevent synfloods from happening with that many hosts trying
to connect at once.

2. asking 5000 minions to return their data will also flood the
master. while 1. is on port 4506, this will flood on port 4505.

3. you will have A LOT of small/random i/o on the master if you dont
disable the job-cache.

i wrote details regarding this earlier, if you want to know more, let
me know.

But finally:
you CAN run that many minions on a master. Neither salt nor ZeroMQ
will limit you here. Just dont expect the environment to be as quick
as with maybe a 100 minions. For example have the minions run with
random_reauth_delay, set the recon_* settings i mentiond above, if you
use the scheduler on the minions use the splay settings, disable the
job-cache on the master and use salt-eventsd
(https://github.com/felskrone/salt-eventsd) instead.

Here are my values from the configs:

master:
max_open_files: 65536
worker_thread: 20

minions:
random_reauth_delay: 270
recon_default: 1000
recon_max: 199000
recon_randomize: True

minion scheduler:
schedule:
list_upgrades:
function: wputil.report_pkgs
seconds: 600
maxrunning: 1
splay:
start: 1
end: 120


Read the comments in the config and you can do the math yourself how
long it takes for all minions to reconnect to the master if it was
restarted or how often the scheduled job runs :-) If you want to run
that many minions and a single master, its essential to understand
whata happening.

Btw, great to finally here from someone who is also running a big
environment. I felt pretty lonely... :-)

- felskrone

[Roger Criddle]

felskrone,
   Thanks for the info.   I believe you described exactly what I am seeing.   I will try those settings you gave.    I would like more detail on the small I/O from the job cache, and details on how to disable it.     How many minions are you running from one master?   How many minions total?

Roger 

[David Boucha]

Hi Roger,

updating those changes like felskrone mentioned should take care of your problem.  Salt can handle 5000 minions just fine. 

What version of Salt are you using?

Dave

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Roger Criddle]

I am using version 2014.1.3-1.

On Friday, May 23, 2014 12:01:46 PM UTC-6, Roger Criddle wrote:

[David Boucha]

did adding those config options to your minions fix your problem?

--

[Roger Criddle]

Also, are the options mentioned by felskrone available for Windows minions?    I don't see them in the default Windows minion file.

On Friday, May 23, 2014 12:01:46 PM UTC-6, Roger Criddle wrote:

[David Boucha]

Ah, you're running Windows minions?   Yeah, those options should work there, too.

--

[Roger Criddle]

@Dave,
     I have a mix of Red Hat and Windows minions.    I updated all the Red Hat servers with those options.   I stopped the minion service on all my Windows servers, while I work on a way to update them all.    But I am still getting the same error.    Am I missing something else?

Roger   

On Friday, May 23, 2014 12:01:46 PM UTC-6, Roger Criddle wrote:

[David Boucha]

Hm.  try running the salt-master in debug mode in the cli.  First stop the salt-master service, then from the terminal run as root:

salt-master -l debug

When you do that do you get any stacktraces?

--

[Volker]

Hey Roger,

> I would like more detail on the small I/O from the job cache, and
> details on how to disable it.

The saltmaster caches all published jobs and their replys locally in
its job-cache. Thats quite useful if you for example publish a job and
a minion (not the minions, just one of many that was targeted) does
not answer in time (meaning the salt-cli will timeout waiting for
returns before all minions have returned).

Even though you dont see it on the cli, the saltmaster still receives
the delayed minions return and writes the reply into its job-cache.
You can then query the job-cache with a specific job-id (jid) to find
out, which minions have returned (jobs.lookup_jid i believe).

Now, if a minion does not reply in time, the saltmaster wants to know,
if the job might be still executing on that minion and publishes a job
asking the minions if job (jid) xxxxxxxxxxxxxxxxx is still running
(saltutil.find_job).

That job is also put into the job-cache.

If you look at the job-cache (/var/cache/salt/master/jobs/), its just
some directories with files in it with each file representing a
minions return to a specific jid.

Now imagine publishing jobs for 5000 minions with 5000 returns,
writing 5000 small files, even more files if saltutil.find_job has to
be used (which is very likely with many minions). Over time you will
find many many files in the cache which take quite a while to query if
you're looking for something specific.

Depending on your environment, how many jobs you plan to publish
hourly/daily/weekly and how long you want/have to keep the job-history
(thats what the job-cache is essentially), the cache can become a
burden for the salt-master with millions of files in it.

For example in my environment we have about 14.000 minions with about
200.000 customers which can configure their webspace through a gui.
The gui publishes jobs via https through salt-api. I have to keep a
history for the last 6 months, in case some customer fucks up his
webspace configuration so we can prove, he did it himself.

There are ways around this dillema, for example disabling the
job-cache and using returners, but with that you lose salts encryption.

Another alternative might be salt-eventsd (sorry dave, dont mean to
advertise, but the question seems to come up more frequently lately :-) ):

https://github.com/felskrone/salt-eventsd

which is (in its default setup) a mysql-job cache for salt with the
possibility to push the returned data anywhere you want (postgres,
graphite, etc.). . I'll happily answer questions on how to use it but
in a different thread please.

> How many minions are you running from one master?

Currently we're in the process of setting up around 4500 minions per
salt-master (4 masters in total) on DELL Poweredge 610, 2.1ghz dual
hexacore, 16GB Ram. The limit here is the number of connections per
second our master can handle without being syn-flooded.

Another factor for us is, how many minions we can have report data
within a 10 minute time frame without overloading the master. Which
turned out to be around 4500 minions using the scheduler with the
aforementioned settings.

I've not setup any monitoring yet, but usually the limit is cpu, not ram.
Now that i mention it, CPU power is what the salt-master likes to use
a lot. Read this: https://github.com/saltstack/salt/pull/9235

You should consider using a 2048 bit keysize.

I think there has been commmits concerning the caching of keys on the
master, but im not sure. Anyone know?

> How many minions total?
We have a total of around 15.000 minion, with 14.000 being virtual
servers.

Hm, i maybe i should write a documentation section for larger setups
or maybe even a concrete example how we do it.

What you think, dave, something you like? :-)

- felskrone

[David Boucha]

I would LOVE to see a documentation of how you work with large environments!

[Roger Criddle]

I ran salt-master -l debug.   After all the startup messages, the only thing that comes up are the following 3 lines, that keep repeating:

[DEBUG   ] Updating fileserver cache
[DEBUG   ] diff_mtime_map: the maps are the same
[DEBUG   ] This salt-master instance has accepted 3641 minion keys.

Still get the same error when trying to run a salt test.ping.    But nothing shows up in the debug when I run the command.

[root tmp]# salt -v "l5513*" test.ping

[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate, is this user permitted to execute commands?

[Volker]

On 5/28/14 4:25 PM, Roger Criddle wrote:
> I ran salt-master -l debug. After all the startup messages, the only
> thing that comes up are the following 3 lines, that keep repeating:
>
> [DEBUG ] Updating fileserver cache
> [DEBUG ] diff_mtime_map: the maps are the same
> [DEBUG ] This salt-master instance has accepted 3641 minion keys.

What does

$ salt-run manage.up

output after starting up the salt-master in debug mode?

- felskrone

[Roger Criddle]

I get the same output from the salt-run manage.up command.   But still nothing different in the debug output, just those 3 repeating lines.     Did the master get messed up somehow?  The master was working great, until I started rolling out all the minions, then this behavior started.

[root tmp]# salt-run manage.up

[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate, is this user permitted to execute commands?

BTW, Felskrone, I would love to see a document on rolling salt out to a large environment.

[Volker]

On 5/28/14 5:11 PM, Roger Criddle wrote:
> I get the same output from the salt-run manage.up command. But still
> nothing different in the debug output, just those 3 repeating lines.
> Did the master get messed up somehow? The master was working great,
> until I started rolling out all the minions, then this behavior started.
>

It means your cli cannot talk to the salt-master (locally).

Try deleting all sockets in /var/run/salt/master and try again.

If it still does not work, well... no idea really. Maybe try deleting:

/var/cache/salt
/var/run/salt*

and try again.

If it still does not work, try starting from scratch (purge and
reinstall, but save your /etc/salt/master and /etc/salt/pki).

>
> BTW, Felskrone, I would love to see a document on rolling salt out to a
> large environment.
>

I try finding the time to do so. Not sure if its actually something that
belongs into the core documentation.

-felskrone

[Thomas Jackson]

We have this issue after we updated to 2014, the issue was opened here (https://github.com/saltstack/salt/issues/11865), the default for all those minion back offs were changed. We run around 10k minions a master and didn't have problems before, and don't after applying those minion settings. As for the job cache (as mentioned above) that won't be a real bottleneck unless you run lots and lots of jobs, as the volume there is tied to number of jobs run. In the next release of salt the job cache is modular on the master so you can use something else to store the return data in, but I added the feature for multi master support, not because the job cache was too slow.

[Volker]

On 5/28/14 5:54 PM, Thomas Jackson wrote:
> We have this issue after we updated to 2014, the issue was opened here (https://github.com/saltstack/salt/issues/11865), the default for all those minion back offs were changed. We run around 10k minions a master and didn't have problems before, and don't after applying those minion settings. As for the job cache (as mentioned above) that won't be a real bottleneck unless you run lots and lots of jobs, as the volume there is tied to number of jobs run. In the next release of salt the job cache is modular on the master so you can use something else to store the return data in, but I added the feature for multi master support, not because the job cache was too slow.
>

Hm, are you certain thats a 2014.x issue? Because i've seen this kind of
behaviour as early as july 2014. Thats why i commited all the recon_*
and random_reauth_delay configuration settings back then. Must have been
saltstack 0.10.x or something like that.

https://github.com/saltstack/salt/issues/5948

I guess this bug has been and still is in there somewhere despite all
the possible fixes and all the recon_* and random_reauth_* settings just
hide it.

-felskrone

[David Boucha]

Hm. Can you provide the output of   "salt-master --versions-report"  and "salt -G 'os:RedHat' test.versions_report"  and "salt -G 'os:Windows' test.versions_report"

Also, can you provide a sanitized version of your master config and minion config?

[David Boucha]

Also, what are the hardware specs on your Salt Master?

[Roger Criddle]

Dave,
      I did a complete purge and re-install of my master.   I saved the master file and the pki directory and restored them.    Had the same problem.   So I did a complete purge and re-install again, this time I did not restore the pki directory but instead this time I just deleted the minion_master.pub on the minions and restarted the service.   I accepted a few of the minion requests on the master and all was fine.   So then I tried to accept all the minion requests ( ~1900) and now I am back to the same problem where salt commands fail.

Here is some output you requested:
Master file:
interface: my_ip_address
max_open_files: 65536
worker_threads: 20

Minion file for RHEL servers:  (Windows is same except no startup state defined)
master: salt-itf

random_reauth_delay: 270
recon_default: 1000
recon_max: 199000
recon_randomize: True

startup_states: 'sls'
sls_list:
  - rhelpatch.rhelpatch

# salt-master --versions-report
Salt: 2014.1.4
Python: 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
Jinja2: 2.2.1
M2Crypto: 0.20.2
msgpack-python: 0.1.13
msgpack-pure: Not Installed
pycrypto: 2.0.1
PyYAML: 3.10
PyZMQ: 2.2.0.1
ZMQ: 3.2.4

]# salt -G 'os:RedHat' test.versions_report

[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate, is this user permitted to execute commands?

My salt master is a vmware guest, running RHEL6, with 4 CPUs and 8GB RAM.   My minions are a mix of RHEL6, RHEL5, Windows 2003/2008/2012.  Right now I have the minion service on my Windows servers disabled on all but about 200 of them.    So I have about 1900 RHEL minions that are active and about 200 Windows minions.

Roger

[David Boucha]

in the output of "top" is the salt-master maxing out a cpu?

[Roger Criddle]

I did another purge and this time I am trying to accept the keys in smaller chunks this time.   But when I did a whole bunch at a time, yes it would max out all 4 of the CPUs.

[Volker]

On 5/29/14 9:52 PM, Roger Criddle wrote:
> I did another purge and this time I am trying to accept the keys in
> smaller chunks this time. But when I did a whole bunch at a time, yes
> it would max out all 4 of the CPUs.

So you're waiting till all minions have sent their public keys while not
accepting any?

Then you accept maybe 10, another 20, etc. and then you do salt-key -A
to accept the rest?

Please describe exactly what you're doing.
How much is 'a bunch'?
How big are 'smaller chunks'?

Why is that important?
Because if you do a salt-key -A with maybe 2000 unaccepted minions and
you have not changed acceptance_wait_time from its default of 10, you
have the very same situation as described before.

With 2000 minions that would be approx. 200 trying to get authenticated
per second which might by triggering the described problem.

Working settings also seem to be hardware dependent. I've read in the
issues somehwere that someone was able to accept 10000 minions within 90
seconds (roughly 100 minions a second).

Depending on your masters ressources, you just cant accept too many
minions at a time nor have them reconnect to fast.

My hardware is roughly twice your vm (16 2.1ghz cores, 16gb) and my load
still goes through the roof when i restart my master and 15-25 minions
log in per second.

Adjustments:
- have a look at acceptance_wait_time
- write a skript thats accepts a few keys at a time from the list of not
yet authenticated keys and then sleeps a few seconds
- be patient with accepting keys, you only have to do it once, no need
to rush :-)

- felskrone

[Roger Criddle]

This is working now.   Thank you Dave and Felskrone for all the help.

The minion settings that seem to work for me are:

random_reauth_delay: 270
recon_default: 1000
recon_max: 30000
recon_randomize: True
auth_timeout: 60

Also note another cause of the error I listed above was that I was running out of inode space on /var.    I had a small /var partition and typically watch disk space utilization but do not typically watch inodes because I generally don't have an issue.    But I will be watching it now on the salt-master.

Thanks again,

Roger      

On Friday, May 23, 2014 12:01:46 PM UTC-6, Roger Criddle wrote: