salt-minion can't authenticate to master, even though key has been accepted

[jdelic]

Hey,

I need some help with this. Since salt 2014.1 my vagrant network setup with pre-seeded keys fails with

"""

Minion failed to authenticate with the master, has the minion key been accepted?

[WARNING ] SaltReqTimeoutError: Waited 3 seconds

"""

In this case already the master box can't authenticate against itself. I'm running this on Debian wheezy with the salt distribution from http://debian.saltstack.com/debian. The interesting part is that the key has actually been accepted. I.e:

"""

root@saltmaster:/etc/salt# salt-key

Accepted Keys:

saltmaster.test

Unaccepted Keys:

Rejected Keys:

"""

and

"""

root@saltmaster:/etc/salt# openssl rsa -pubin -noout -modulus -in /etc/salt/pki/master/minions/saltmaster.test | openssl md5

(stdin)= 5a59e647d449606efebc235f498ddaa8

root@saltmaster:/etc/salt# openssl rsa -noout -modulus -in /etc/salt/pki/minion/minion.pem | openssl md5

(stdin)= 5a59e647d449606efebc235f498ddaa8

"""

when I run salt-call manually, I get:

"""

[DEBUG   ] Reading configuration from /etc/salt/minion

[INFO    ] Using cached minion ID from /etc/salt/minion_id: saltmaster.test

[DEBUG   ] Configuration file path: /etc/salt/minion

[DEBUG   ] Reading configuration from /etc/salt/minion

[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

[WARNING ] SaltReqTimeoutError: Waited 3 seconds

Minion failed to authenticate with the master, has the minion key been accepted?

"""

The /etc/hosts on this box contains:

"""

192.168.56.88       saltmaster.test

"""

and the interface is up and pingable:

"""

root@saltmaster:/etc/salt# ping saltmaster.test

PING saltmaster.test (127.0.1.1) 56(84) bytes of data.

64 bytes from saltmaster.test (127.0.1.1): icmp_req=1 ttl=64 time=0.022 ms

"""

Does anyone have an idea what can be wrong with this setup? or what I can check next?

Best regards,

jdelic

[bruno binet]

It seems that you're trying to ping you salt master from itself (127.0.1.1).

So can you try to ping saltmaster.test from your minion instead?

Can you ensure the salt master server ports 4505 and 4506 are open?

Maybe also watch logs from the salt master to see if there is any valuable information?

 

Does anyone have an idea what can be wrong with this setup? or what I can check next?

Best regards,

jdelic

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Hollingsworth]

I see the same behavior intermittently, except I am running on AWS. I have a script that 

- Brings up an instance

- Installs salt master and minion

- Preseeds keys per instructions at http://docs.saltstack.com/en/latest/topics/tutorials/preseed_key.html
- Calls salt-call test.ping

This works about 80% of the time. The other 20%, it fails with

[WARNING ] SaltReqTimeoutError: Waited 60 seconds

Minion failed to authenticate with the master, has the minion key been accepted?

Any hints as to what could be going on?

-Matt

[Colton Myers]

How many minions are we talking about?  Are you bringing up one at a time or many?

--

Colton Myers

[Matt Hollingsworth]

Only one, which is both a master and a minion (we add the other minions later, this is happening during a cluster bootstrapping process).

What I did to get around this problem is:

service salt-master restart

sleep 5

service salt-minion restart

sleep 5

salt-call state.sls.highstate

If salt-call exits non-zero, I do all of that again. I presently have it set to retry 5 times. It usually works by the third or fourth try, but that's the only way that it's been 100% reliable.

-Matt

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/zEA224dbXvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Colton Myers]

Anything unique about your network setup?  Is there a lot of latency or anything?  I assume the case of the filename of the minion key on the master matches the minion ID exactly?

--

Colton Myers

[Matt Hollingsworth]

Nope, it pretty much can't be network related, since it is connecting via loopback (the minion is on the same machine as the master it's authenticating to), and iptables are completely open. Yep, it the minion ID matches, and it works if I go through enough of these restart cycles, it just doesn't work right off. 

-Matt

[Colton Myers]

Well, that sounds like something we should look into.  Could you please file an issue on Github?

--

Colton Myers

[Stephen Wood]

I just ran into this issue myself. I'm running an old version of salt (2014.1.0), but I'm updating this email thread in case others get hit with it. 

After running a saltutil.sync_all, my master node was effectively DDoS'ed. I couldn't run commands on any minions and the CPU was pegged for each worker process.

I tried various things (restarting the minions, clearing out the cache on master and minions), but the only thing that got the server responsive again was to delete all keys and slowly add them again.

I added the following things to my minion config and so far things are much more stable. I'm also using --batch-size 25% flags when I run across the entire fleet.

recon_randomize: True

random_reauth_delay: 120

Stephen Wood

www.heystephenwood.com