master does not see minion (with detailed debug info)

[Volker S.]

Hi,

i have a master with about 2300 minions running successfully.

After migrating a server to a new hardware, i am trying to reconnect this (new) minion to the master. Here's what i do:

Starting the minion in debug-mode:

###
server139:~# salt-minion -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
[INFO    ] Found minion id from getfqdn(): wp139.webpack.hosteurope.de
[INFO    ] Configuration file path: /etc/salt/minion
[INFO    ] Setting up the Salt Minion "wp139.webpack.hosteurope.de"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Chowned pidfile: /var/run/salt-minion.pid to user: root
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/pymodules/python2.6/salt/grains']
[DEBUG   ] Attempting to authenticate with the Salt Master at 10.30.8.10
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Waiting for minion key to be accepted by the master.
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Waiting for minion key to be accepted by the master.
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Waiting for minion key to be accepted by the master.
###

So far so good.

Since the old minions key was deleted prior to the migration, the newly generated minion-key should show up on the master.
But for some odd reason it does not, salt-key does not list any unaccepted minions.

Here's what i checked:
* the minion uses the exact same config that all the other minions use (md5 checked)
* i can delete and re-connect other minions just fine, just not this one (tested with 2 minions)
* i deleted all keys on the minion and had them re-generated, no success
* i reinstalled the minion completely after purging it
* i have the master running in debug-mode in screen, but i dont see any messages of the minion trying to connect
* i did not find any files related to the old or new minion in /var/cache/salt/master/minions
* i verified connectivity using telnet and tcpdump, seems works just fine (60sec dump here: http://pastebin.com/qL4u7ygc)

Im really out of ideas here.

I guess restarting the master might help, but its done that easily. Currently (im not sure if its since zeromq3) restarting the master makes my setup unusable due to some sort of flooding-problem that seems to occur occasionaly, especially on a restart.

###
Jun 17 12:57:32 salt kernel: [  209.743451] TCP: TCP: Possible SYN flooding on port 4506. Dropping request.  Check SNMP counters.
Jun 19 11:27:58 salt kernel: [167558.742069] TCP: TCP: Possible SYN flooding on port 4506. Dropping request.  Check SNMP counters.
###

This is reproducable and might be related to this: https://github.com/saltstack/salt/issues/4106

But thats a different problem.

- felskrone

[Volker S.]

Hi,

Since the old minions key was deleted prior to the migration, the newly generated minion-key should show up on the master.
But for some odd reason it does not, salt-key does not list any unaccepted minions.

i have the exact same problem with another minion now. This time its not a migration though, the minion just does not respond anymore (but it did before).

Starting it in debug-mode i can see that it trys to connect to the master and waits for the key get accepted:

###

[DEBUG   ] Attempting to authenticate with the Salt Master at 10.30.8.10
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

###

Funny thing is, that the key is actually still present for that minion. It seems like either the master does not accept the minion or the minion does not notice, that its accepted by the master. Either way communication between minion and master is broken.

I tried deleting the key from the master and reconnect the minion but that does not change anything. The master does not notice the connecting minion, the key does not show up in salt-key and the minion cant connect anymore.

Any hints on how to debug this situation?

- v

[Lonni J Friedman]

Sounds like issue 5599:
https://github.com/saltstack/salt/issues/5599

[Colton Myers]

I agree, this looks a lot like #5599.  We don't have much to go on yet, but I'll bug Tom to look into it more proactively, now that we have more than one infrastructure showing the problem.

--

Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.