Connectivity issue between master and minion

[Nirlay Kundu]

I have a salt-master running with an IP 10.106.223.190. I configured minion file to have :

it own ID id: minion-202

and master : master: 10.106.223.190

Then I restarted the salt-master :  service salt-master start on .190

But I am not able to see the minion.

[root@saltstack salt]# salt '*' test.ping

1stminion:

    True

1stminion:

    True

1stminion.:

    True

[root@saltstack salt]# salt-key -L

Accepted Keys:

1stminion

1stminion.

Unaccepted Keys:

Rejected Keys:

[Colton Myers]

The first thing I would check is firewall rules -- make sure the minion can call out and the master has its ports open.  (I assume that the master is OK since you have other minions connected)  I would just have the master bind to all interfaces (which is the default, I think) rather than specifically binding to the IP address you listed.

In any case, could you please run the minion in debug mode (salt-minion -d -l debug) and give its startup logs?

--

Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nirlay Kundu]

I have the master running on one VM, in the same VM there are some minions which are connected. I am having problems in connecting a minion which is in a different VM, but in the same subnet, Ip connectivity works fine. How would I configure the master to bind to all interfaces ? In the minion where I am having trouble, I specified the master's IP address and gave itself an id - nothing else is changed.

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# salt-minion -d -l debug

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# service salt-minion start

Starting salt-minion daemon:

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]#

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# ps -aux |grep salt

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ

root      5466  0.1  4.1 243736 15976 ?        Sl   15:16   0:00 /usr/bin/python2.6 /usr/bin/salt-minion -d

root      5478  0.2  4.1 243744 16048 ?        Sl   15:17   0:00 /usr/bin/python2.6 /usr/bin/salt-minion -d -l debug

root      5496  0.0  0.1  61184   748 pts/2    S+   15:17   0:00 grep salt

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]#

[Colton Myers]

So have you checked the firewall rules (iptables or whatever) on your master VM?  IP connectivity doesn't mean you have the ports open.  4505 and 4506 are the ports you need to open for the master.

To bind to all interfaces, either set the master to bind to 0.0.0.0 or just remove the ip address configuration line.

--

Colton Myers

[Nirlay Kundu]

I will check on the ports.

To set the master bind to 0.0.0.0, I uncommented the interface line and it is giving me 

[root@saltstack salt]# service salt-master stop

Stopping salt-master daemon:                               [  OK  ]

[root@saltstack salt]# service salt-master start

Error parsing configuration file: /etc/salt/master - while parsing a block mapping

  in "<string>", line 14, column 1:

    interface: 0.0.0.0

    ^

expected <block end>, but found '<block mapping start>'

  in "<string>", line 241, column 2:

     file_roots:

     ^

[Nirlay Kundu]

sorry, found the sysntax error, it was a space issue. But I will check the ports now as still I do not see my minion when I do salt-key -L

[Nirlay Kundu]

Yes, I checked all ports are open. But minion is not listening on 4505 and 4506.

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# lsof -i

COMMAND    PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

dhclient  2230    root    4u  IPv4  13227      0t0  UDP *:bootpc

portmap   2380     rpc    3u  IPv4  13646      0t0  UDP *:sunrpc

portmap   2380     rpc    4u  IPv4  13647      0t0  TCP *:sunrpc (LISTEN)

hpiod     2676    root    0u  IPv4  14695      0t0  TCP localhost.localdomain:2208 (LISTEN)

hpssd.py  2681    root    4u  IPv4  14713      0t0  TCP localhost.localdomain:2207 (LISTEN)

sendmail  2734    root    4u  IPv4  14943      0t0  TCP localhost.localdomain:smtp (LISTEN)

avahi-dae 2834   avahi   13u  IPv4  15418      0t0  UDP *:mdns

avahi-dae 2834   avahi   14u  IPv6  15419      0t0  UDP *:mdns

avahi-dae 2834   avahi   15u  IPv4  15420      0t0  UDP *:50607

avahi-dae 2834   avahi   16u  IPv6  15421      0t0  UDP *:54941

sshd      3106    root    3u  IPv6  22056      0t0  TCP *:ssh (LISTEN)

sshd      3106    root    4u  IPv4  22058      0t0  TCP *:ssh (LISTEN)

sshd      4704    root    3u  IPv4 128393      0t0  TCP 9410dddc-4219-4851-953d-f5bff2b3017d.root.public.test.example.com:ssh->67.20.183.235:52592 (ESTABLISHED)

yum-updat 5631    root    7u  IPv4 140773      0t0  TCP 9410dddc-4219-4851-953d-f5bff2b3017d.root.public.test.example.com:33039->216.92.2.158:http (SYN_SENT)

rpc.statd 6064 rpcuser    3u  IPv4  66171      0t0  UDP *:netviewdm3

rpc.statd 6064 rpcuser    6u  IPv4  66162      0t0  UDP *:728

rpc.statd 6064 rpcuser    7u  IPv4  66178      0t0  TCP *:734 (LISTEN)

cupsd     6130    root    4u  IPv4  66947      0t0  TCP localhost.localdomain:ipp (LISTEN)

cupsd     6130    root    6u  IPv4  66950      0t0  UDP *:ipp

[Mike Chesnut]

From the minion, can you telnet to 4506 on the master?

[Colton Myers]

Yes, the minion doesn't have to have *any* ports open -- it connects to the master.  Telnet is a good way to check if the ports are open on the master.

--

Colton Myers

[Nirlay Kundu]

Nope. Does this mean that the ports on the master are not open ?

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# telnet 10.106.223.190 4506

Trying 10.106.223.190...

telnet: connect to address 10.106.223.190: No route to host

telnet: Unable to connect to remote host: No route to host

[Colton Myers]

Correct, they are not open.  You'll have to change your iptables config to allow ports 4505 and 4506 on the master.

--

Colton Myers

[Mike Chesnut]

Is this in AWS? My guess would be you haven't allowed the ports via security groups.

[Nirlay Kundu]

also, the minion has to listen on ports 4505 and 4506, correct ?

[Nirlay Kundu]

Nope. We are using Nimbula and it is in the same vlan.

[Colton Myers]

No, the minion can be firewalled into oblivion.  As long as it can connect out, it doesn't have to have any ports open to the outside.  One of the great things about Salt.  =)

--

Colton Myers

[Nirlay Kundu]

OK. We enabled all inbound connectivity to that VM. Any other way to check ?

[Colton Myers]

So you've added the exceptions to iptables on your master?

--

Colton Myers

[Nirlay Kundu]

I changed a few things,. now I am getting an error 

 WARNING: Master hostname: salt not found. Retrying in 30 seconds

where is that hostname picke from ? I do not have any hostname in minion file, I gave an IP

[Colton Myers]

Right, you should replace the commented hostname line in the minion config with the IP.

It sounds like you might have a typo in your minion config, because it's going back to the default 'salt' hostname for master.

--

Colton Myers

[Nirlay Kundu]

OK, solved that issue. Still not sure of the connectivity. all VMs are in the same seclist. No iptables.

[Nirlay Kundu]

default CentOS6 iptables setting are there. Pls see attachd. but itis all anywhere to anywhere. Will this cause a problem in connectivity ?

[Colton Myers]

It *looks* like it should work, based on everything we see in your screenshot.

However, if it's still not connecting, can you give us the output of `iptables -v -n -L` please?  Just a copy of the output would be fine, we don't need an actual screenshot.

If you're not seeing any errors on the minion's end, are you sure the minion has not yet shown up in your `salt-key` list?

--

Colton Myers

[Romeo Theriault]

Maybe this has all been covered, but this is what I'd try:

* Disable iptables completely: service iptables stop

* Can you ping the master's ip address?

* If no, you'll have to check something with the way your virtual env. handles the networking.

* If yes, can you telnet to the master's port?

* If yes, you *should* be able to connect to the master.....

If not. I don't know :)

Romeo

--

Colton Myers

--

Colton Myers

--

Colton Myers

telnet: connect to address MailScanner warning: numerical links are often malicious: 10.106.223.190: No route to host

telnet: Unable to connect to remote host: No route to host

On Friday, May 10, 2013 12:38:46 PM UTC-4, basepi wrote:

Yes, the minion doesn't have to have *any* ports open -- it connects to the master.  Telnet is a good way to check if the ports are open on the master.

--

Colton Myers

On Fri, May 10, 2013 at 10:36 AM, Mike Chesnut <mche...@gmail.com> wrote:

From the minion, can you telnet to 4506 on the master?

On Fri, May 10, 2013 at 9:27 AM, Nirlay Kundu <nir...@gmail.com> wrote:

Yes, I checked all ports are open. But minion is not listening on 4505 and 4506.

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# lsof -i

COMMAND    PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

dhclient  2230    root    4u  IPv4  13227      0t0  UDP *:bootpc

portmap   2380     rpc    3u  IPv4  13646      0t0  UDP *:sunrpc

portmap   2380     rpc    4u  IPv4  13647      0t0  TCP *:sunrpc (LISTEN)

hpiod     2676    root    0u  IPv4  14695      0t0  TCP localhost.localdomain:2208 (LISTEN)

hpssd.py  2681    root    4u  IPv4  14713      0t0  TCP localhost.localdomain:2207 (LISTEN)

sendmail  2734    root    4u  IPv4  14943      0t0  TCP localhost.localdomain:smtp (LISTEN)

avahi-dae 2834   avahi   13u  IPv4  15418      0t0  UDP *:mdns

avahi-dae 2834   avahi   14u  IPv6  15419      0t0  UDP *:mdns

avahi-dae 2834   avahi   15u  IPv4  15420      0t0  UDP *:50607

avahi-dae 2834   avahi   16u  IPv6  15421      0t0  UDP *:54941

sshd      3106    root    3u  IPv6  22056      0t0  TCP *:ssh (LISTEN)

sshd      3106    root    4u  IPv4  22058      0t0  TCP *:ssh (LISTEN)

sshd      4704    root    3u  IPv4 128393      0t0  TCP 9410dddc-4219-4851-953d-f5bff2b3017d.root.public.test.example.com:ssh->MailScanner warning: numerical links are often malicious: 67.20.183.235:52592 (ESTABLISHED)

--

Romeo

[Nirlay Kundu]

My MASTER iptables are

[root@saltstack salt]# iptables -v -n -L

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination

 696K  616M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED

    2   168 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0

72665 4360K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0

  109  5244 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22

12736  764K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination

    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 642K packets, 58M bytes)

************************************************

I stopped iptables per Romeo and the minion showed up. 

THANKS A TON, EVERYBODY !!

[Colton Myers]

I got Joseph to look at your iptables output -- if you want to use iptables, find your iptables config file (for RedHat, it will be at /etc/sysconfig/iptables), find the line that looks like

    -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

and duplicate it twice, changing the port (22) to 4505 and 4506.  That should open those ports for your master.

--

Colton Myers

[Nirlay Kundu]

Yes, thanks. I used this to bring up iptables and specifying the ports and it works

iptables -I INPUT 1 -p tcp -m tcp --dport 4506 -j ACCEPT

An unrelated question : After I add a minion id to the minion file, I need to restart the minion to show up in the master's salt-key lists. is this per design ?

[Colton Myers]

Any change to the master or minion configs requires a restart of whichever one was edited (master or minion) before the changes take effect.

Not ideal, but changes to the minion or master configs tend to be infrequent, so it's not too bad.

--

Colton Myers

[Nirlay Kundu]

I modified the minion file in a minion and added a minion id, restarted minion and then from the master did salt-key -L; As expected the minion id showed up in the unaccepted list. Then I deleted the minion id and restarted the minion; I was expecting the minion to disappear from the master's salt-key's list. It did not. Then I accepted the keys of the deleted the minion, it shows up in the accepted key-list.   Is this how it should work ?

thanks

Nirlay

[Nirlay Kundu]

Another issue : I accepted a minion key on a different VM. It shows up in my master's list of accepted keys. The minion id is active (uncommented) on the minion. However, test.ping does not return True for that minion, although it returns True for a minion on the master itself. Restarted the minion few times.

What am I doing wrong ?

[Colton Myers]

Run the minion in debug mode, and then look at the log for the minion.  See if it's actually successfully connecting to the master.

--

Colton Myers

[Nirlay Kundu]

where is the log file generated ?

[Colton Myers]

By default it should be in /var/log/salt/minion I think.  You could also run the minion in non-daemonized mode with the command

    salt-minion -l debug

Either works.

--

Colton Myers

[Nirlay Kundu]

My minion currently has

id: minion-202

id: minion-202-1

id: minion-202-2

My master's output from salt-key is

[root@saltstack salt]# salt-key -L

Accepted Keys:

1st salt minion

1stsaltminion

2ndsaltminion

minion-202

minion-202-3

Unaccepted Keys:

minion-202-1

minion-202-2

Rejected Keys:

I was expecting that test.ping should return True for minion-202.

[root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# salt-minion -l debug

[INFO    ] Configuration file path: /etc/salt/minion

[INFO    ] Setting up the Salt Minion "minion-202-2"

[DEBUG   ] Created pidfile: /var/run/salt-minion.pid

[DEBUG   ] Chowned pidfile: /var/run/salt-minion.pid to user: root

[DEBUG   ] Reading configuration from /etc/salt/minion

[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/python2.6/site-packages/salt/grains']

[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory

[DEBUG   ] Attempting to authenticate with the Salt Master at 10.106.223.198

[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem

[ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

[INFO    ] Waiting for minion key to be accepted by the master.

[Nirlay Kundu]

I think I got my issue. I am getting the follwong :

[ERROR   ] The master key has changed, the salt master could have been subverted, verify salt master's public key

[CRITICAL] The Salt Master server's public key did not authenticate!

The master may need to be updated if it is a version of Salt lower than 0.14.1, or

If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.

The master public key can be found at:

/etc/salt/pki/minion/minion_master.pub

Is there anyway other than deleting the master public key ?

[ERROR   ] The master key has changed, the salt master could have been subverted, verify salt master's public key

[CRITICAL] The Salt Master server's public key did not authenticate!

The master may need to be updated if it is a version of Salt lower than 0.14.1, or

If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.

The master public key can be found at:

/etc/salt/pki/minion/minion_master.pub

[Jeffrey Ollie]

On Sun, May 12, 2013 at 7:05 PM, Nirlay Kundu <nir...@gmail.com> wrote:
>
> My minion currently has
>
> id: minion-202
> id: minion-202-1
> id: minion-202-2

You may have these lines in your config file, but only one takes effect.

> My master's output from salt-key is
>
> [root@saltstack salt]# salt-key -L
> Accepted Keys:
> 1st salt minion
> 1stsaltminion
> 2ndsaltminion
> minion-202
> minion-202-3
> Unaccepted Keys:
> minion-202-1
> minion-202-2
> Rejected Keys:

The server has not accepted the key that the minion is using.

> I was expecting that test.ping should return True for minion-202.

There is no minion actively using that id.

> [root@9410dddc-4219-4851-953d-f5bff2b3017d salt]# salt-minion -l debug
> [INFO    ] Configuration file path: /etc/salt/minion
> [INFO    ] Setting up the Salt Minion "minion-202-2"

This is the ID that the minion is currently using.

> [DEBUG   ] Created pidfile: /var/run/salt-minion.pid
> [DEBUG   ] Chowned pidfile: /var/run/salt-minion.pid to user: root
> [DEBUG   ] Reading configuration from /etc/salt/minion
> [DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains',
> '/usr/lib/python2.6/site-packages/salt/grains']
> [DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a
> directory
> [DEBUG   ] Attempting to authenticate with the Salt Master at 10.106.223.198
> [DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
> [ERROR   ] The Salt Master has cached the public key for this node, this
> salt minion will wait for 10 seconds before attempting to re-authenticate

> [INFO    ] Waiting for minion key to be accepted by the master.t in     out     source              

This is the minion telling you that the master has not accepted the key using the id that the minion is using.

I think that your best bet is to start from scratch:

1) Shut down all the minions.

2) Reconfigure the minions so that they have one id listed in the configuration.

3) Delete all the keys from the master using "salt-key -D" 

4) Start up all of the minions.

5) Accept all of the keys on the master using "salt-key -A"

--
Jeff Ollie

[Jeffrey Ollie]

On Sun, May 12, 2013 at 7:46 PM, Nirlay Kundu <nir...@gmail.com> wrote:

I think I got my issue. I am getting the follwong :

[ERROR   ] The master key has changed, the salt master could have been subverted, verify salt master's public key

[CRITICAL] The Salt Master server's public key did not authenticate!

The master may need to be updated if it is a version of Salt lower than 0.14.1, or

If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.

The master public key can be found at:

/etc/salt/pki/minion/minion_master.pub

Is there anyway other than deleting the master public key ?

Actually, it looks like you've made things a bit worse from before by regenerating the master key.  Yes, now you'll need to delete /etc/salt/pki/minion/minion_master.pub which is a cached copy of the master key that minions use to authenticate the master. 

 

Also, if everyone could please trim your replies I'd appreciate it!

--
Jeff Ollie

[Nirlay Kundu]

Thanks. Deleting the minion_master.pub and restarting the minion works.

How does one regenerate the master key ?

[Jeffrey Ollie]

On Sun, May 12, 2013 at 8:07 PM, Nirlay Kundu <nir...@gmail.com> wrote:

Thanks. Deleting the minion_master.pub and restarting the minion works.

How does one regenerate the master key ?

By deleting the key from the master, but you shouldn't need to do that.

 

--
Jeff Ollie