Unreachable minions after salt-master re-install

[Fredrik Averpil]

Hi,

I re-installed the salt-master completely. I’m accepting they keys for my minions, which are now listed under “Accepted keys” when running salt-key. All good.

What’s weird now is that I cannot get a response from any of my minions when e.g. running salt '*' test.ping. These are all Windows minions.

It doesn’t help to delete the keys for the minion using salt-key -D.

When I run salt-run state.event pretty=True, I can see all minions repeatedly attempting a salt/auth:

salt/auth    {
    "_stamp": "2016-02-23T14:01:10.104476",
    "act": "accept",
    "id": "WS87GEN04",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA02ZQdFv6tZTjDSiv/UIg\n8GVzOF/zILLfR+QvsZDAdSNyIU88BY7mDpSYGA46iF/6GKgWMNjHmKFlm/QMY1K9\nl0JSgh6ZQ/YurulVdlLS7KcTqCvy4QKu/HowOM0lYblZs2r7ltnxb2K5nHjgp+S5\nnjawQi4Tc81Xzdcc+NsJWwSxeJ98a87JJmGVJbSvAzTVNVHbfDp34m1Tjy2MHjVj\nP0jMSIacajZ3DpYGzOQZ0nit6pUQL8duy6ZH8OXTzuYGAiF45KKbYvQc7V7Ch/Dl\nzFdNkDxRWisr99FhWWoo6PR5KS8iLwfJ1vpps7nwSwyOCyf6gSNymHhyyUcXqrvZ\nWwIDAQAB\n-----END PUBLIC KEY-----",
    "result": true
}

What should I do?

Versions report:

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.2
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-123.el7.x86_64
         system: CentOS Linux 7.2.1511 Core

Regards,
Fredrik

​

[RabidCicada]

What's the version of the minions?

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fredrik Averpil]

Ah … I now see this in the minion log:

The master public key can be found at:
c:\salt\conf\pki\minion\minion_master.pub
[INFO ] The salt minion is shut down
Invalid master key

What can I do to fix this?
I’m guessing the only choice I have is to walk around to each machine and stop the minion, delete the minion_master.pub and restart the service?

Could I have prepared for this somehow, prior to reinstalling the salt-master?
Could I have deleted C:\salt\conf\pki\minion\minion_master.pub – or could I have saved the master key?

Regards,
Fredrik

​

[Fredrik Averpil]

Both salt-master and minions are 2015.8.7.

Regards,

Fredrik

[Colton Myers]

Could I have prepared for this somehow, prior to reinstalling the salt-master?

Could I have deleted C:\salt\conf\pki\minion\minion_master.pub – or could I have saved the master key?

Either of these solutions would have worked. You could also transfer the keys from the old master to the new master, which is probably the easiest.

--

Colton Myers

[Fredrik Averpil]

Hi again Colton,

I actually had some issues with this the second time around and I haven't found any documentation or guide on this. Is there anything on this in the docs?

I saved the master.pem and the master.pub and re-installed saltstack. I then created the pki folder and placed the pem/pub in it prior to starting up the salt-master. This actually caused the salt-minion service on the minions (Windows-based) to stop. The solution, again, was to remove the key on each minion ...ugh, not so fun without Salt...

What would this process ideally look like?

Should I back up the entire pki folder with all of its contents, not just the pub/pem?

Regards,

Fredrik

--

[Colton Myers]

Hmmm, retaining the master pub/pem keys should be sufficient. I wonder if you ran into some other bug which caused your windows minion to stop? Or perhaps you didn't have the permissions right on the keys or something? I'm grasping at straw.....

--

Colton Myers

@basepi