Minion Won't "pair" with Master

[Charlie Heselton]

Hello all,

Apologies, in advance, if I picked the wrong list for this topic.  Please feel free to redirect, as appropriate.

I have a rather small saltstack implementation.  I have a couple minions that I am trying to associate with my single master, and no matter what I try, they seem to be failing.  The master simply isn't seeing the keys from the minions.  I've gone through a number of troubleshooting steps, including running both the master and the minion in the foreground with debug logging enabled.  Nothing really seems to indicate what the problem might be.

There is a firewall between the master and the problem minions, but I've ensured ports 4505 and 4506 are open.

What else can I do to get these guys talking?

Thanks.

-Charlie

[Clint Dilks]

Hi,

As a first question are you specifically setting the name of your master in /etc/salt/minion ?  If not are you aware that it will try and talking to a master called salt by default?

Have you testing that DNS, Ping and etc seem to working correctly between both master and minion

Does  telnet <master> 4505 and telnet <master> 4506 seem to be working from a minion?

Thanks

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charlie Heselton]

See inline comments below:

On Wednesday, June 22, 2016 at 6:04:33 PM UTC-7, Clint Dilks wrote:

Hi,

As a first question are you specifically setting the name of your master in /etc/salt/minion ?  If not are you aware that it will try and talking to a master called salt by default?

The minions have been configured with the IP address of the master.  DNS is not configured for the master. 

Have you testing that DNS, Ping and etc seem to working correctly between both master and minion

Does  telnet <master> 4505 and telnet <master> 4506 seem to be working from a minion?

There is a firewall separating these minions and the master, so ICMP (ping) is not allowed.  However, tcptraceroute and netcat can connect to the master, so the ports are open and allowed through the firewall. 

Thanks

I've done some packet captures, among the other troubleshooting things I've tried.  The packet capture shows that the TCP handshake is established; some data is exchanged, but then the master sends a RST packet, followed by a flurry of TCP retransmits from the minion.  I can't find any evidence in the debug logs as to why the connection is reset by the master.  I thought perhaps there was a subnet specific setting that I overlooked in the master config or something, but couldn't find anything obvious upon re-examination.  This also doesn't make sense because there are minions on other segments that can connect fine, and have successfully exchanged keys with the master.

[C. R. Oldham]

I've done some packet captures, among the other troubleshooting things I've tried.  The packet capture shows that the TCP handshake is established; some data is exchanged, but then the master sends a RST packet, followed by a flurry of TCP retransmits from the minion.

Is the RST packet really coming from the master or is the firewall forging it for some reason?

The behavior you describe matches situations we've seen in the field with misconfigured/broken/overzealous firewalls, especially those that like to clean out their internal state tables frequently.

If it is a DPI firewall you might try briefly switching to the TCP transport from ZMQ to see if something in the packets ZMQ generates is tickling the DPI. 

--

--cro

C. R. Oldham, Platform Engineer, SaltStack

c...@saltstack.com / 801-564-4673

[Charlie Heselton]

Looks like the firewall rules hadn't been implemented yet.  Strange that it would complete the TCP handshake before the connection was closed, but maybe it was a layer-7 thing.

Thanks for the views and replies.