Salt Key Management

[ed.lane]

Almost ready for production release and then I encountered this issue....

According to http://docs.saltstack.org/en/latest/topics/tutorials/preseed_key.html

preseeding keys requires that the minion_id to be preregistered on the master.

In our production environment we cannot anticipate the name or minion_id that will be given to the minion when it is first spun-up in the cloud.

This is done by an entirely different system outside of Salt.

The current Salt key exchange method described here makes a turnkey solution problematic.

I asked a "master chef" how they dealt with this issue and he pointed me here:

http://wiki.opscode.com/display/chef/Nodes#Nodes-Authentication

Is this a "batteries not included" or am I missing something.

-ed

[Ryan Schneider]

I'm actually actively working on a patch to add x509 certificate-based auto-acceptance to Salt.  At a high-level it works like this:

- Master (when configured to do so) loads a Certificate Authority Cert and optional CRLs.

- Minions (when configured to do so) send their client certificate with each auth request.

-- this is additive, the existing public/private key pairs are still used, I wanted to keep my changes minimal

- Master (when configured to do so) auto-accepts the Minion's public key if:

-- the client certificate in the auth request is valid (issued by your CA, and not in CRL)

-- the subject DN matches an optional regex

-- the issuer DN matches an optional regex

While I wasn't planning on using it this way, this support could be used much like Chef's validation.pem.  We already have an x509 infrastructure in place, I was planning on having each minion use it's existing cert in the above workflow, but I guess if you used the same cert for all clients it would roughly mimic what Chef is doing (if I'm reading their docs correctly).

Anyways, I hope to have my preliminary x509 support in my github fork tomorrow, and will then look into issuing a Pull Request or opening an Issue once I'm happy with it.

Thanks,

Ryan

[Joseph Hall]

One can only imagine what the reasons are why the minion id is unpredictable.

Salt Cloud, which is meant to run on the master, generates the minion
key on the master (immediately approving them) and then lays them down
for you on the minion when it spins it up. To ensure that the minion
doesn't try to connect to the wrong master, a master_finger option can
be used as described here:

https://github.com/saltstack/salt-cloud/issues/21#issuecomment-9928867

If your system runs from the salt master, I'm sure a version of this
step could be approximated. Outside of salt-cloud, I don't know that
we have anything else built-in at the moment.

As I was typing this, a message from Ryan Schneider came in. It looks
like he's ready to send in a pull that might help as well.

[Clint Savage]

Ryan,

This may be completely off-topic, but do you have any interest in the
tls module in salt to do this? I've not added the CRL part, as I
probably should, but if you wanted to hack that, it would totally rock
and make doing the components below easier and completable within salt
itself.

https://github.com/saltstack/salt/blob/develop/salt/modules/tls.py

Cheers,

Clint

[Ryan Schneider]

Hi Clint,

After your mail I took a quick look at the tls module, thanks I didn't know it existed.

From my googling around today, my understanding is that pyOpenSSL doesn't currently expose all the required OpenSSL bits to handle CRLs properly, so that might make it difficult.

As I said, for my personal need we already have a PKI in place, which is why I'm trying to (lightly) integrate it with Salt.

While I think it's currently outside of my scope, using the tls module to integrate a full lightweight PKI into the Salt Master does sound fascinating (and would be a cool bootstrapping solution for people like the OP that need a way to manage minion life cycles).

Currently I'm struggling with deciding how to handle the case where a minion does have its cert revoked. I want to move his Salt public key to rejected, but don't want to be constantly rechecking minion certs and I'm currently not keeping any cert-to-minion mapping data on the master. I saw the other thread about maintaining metadata on the minion keys, if that gets implemented I might piggy back on that.

Anyways thanks, I'll take a closer look at the tls module tomorrow.

Ryan

[ed.lane]

Joseph,

On Tuesday, October 30, 2012 8:15:28 PM UTC-6, Joseph wrote:

One can only imagine what the reasons are why the minion id is unpredictable.

There seems to be a tendency in this group to consider only the single use case.  One inwhich

Salt is responsible for the entire VM life cycle.  In many established installations salt-cloud will not be used

at all and salt must coexist with other provisioning tools.  In those environments salt must

be nimble in addition to being agile :^)

I actually appreciate the constant "Why would you do it that way?" I receive here but please consider that other

perfectly legitimate use cases also exist for salt.

-ed

[Joseph Hall]

On Wed, Oct 31, 2012 at 9:34 AM, ed.lane <ed.l...@gmail.com> wrote:
> I actually appreciate the constant "Why would you do it that way?" I receive
> here but please consider that other
> perfectly legitimate use cases also exist for salt.

It was meant more as a prompt stating that "I don't know what you're
dealing with, and I'm assuming you have a good reason for not going
into detail, but perhaps you might take the opportunity to do so." It
seems it was poorly worded.

One of the nice things about the flexibility of Salt is that it
considers that there is no such thing as a one-size-fits-all solution.
I have no doubt that your use case is legitimate, and I hope that a
solution is available. I don't know what that solution is, but
hopefully further discussion will help bring an appropriate one to
light.

--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)

[Jeff Bauer]

Ed,

There's nothing preventing you from provisioning your minions with
pre-seeded keys from the salt master. With salt-cloud,
salty-vagrant, and Shaker we make an assumption that you'll know
the minion id (which can be independent of the hostname). But
you could use something like Fabric from the salt master to
create and copy the keys to your minion at a later stage, after
the servers have been instantiated.

Jeff Bauer
Rubicon, Inc.

[ed.lane]

On Wednesday, October 31, 2012 9:49:17 AM UTC-6, Joseph wrote:

On Wed, Oct 31, 2012 at 9:34 AM, ed.lane <ed.l...@gmail.com> wrote:
> I actually appreciate the constant "Why would you do it that way?" I receive
> here but please consider that other
> perfectly legitimate use cases also exist for salt.

It was meant more as a prompt stating that "I don't know what you're
dealing with, and I'm assuming you have a good reason for not going
into detail, but perhaps you might take the opportunity to do so." It
seems it was poorly worded.

I thought I could avoid a lengthy description/justification of the problem by just

providing a link for how Chef solves it.

-ed

"communication is hard, it will always be hard, but it is worth it" -me

[Sean Channel]

Ed,

One hackish solution here is to enable auto_accept on the master and
limit the 4505:4506 ports in the firewall.

There is another solution I need to research for you here (help,
anyone.. please), which is to use the event system to detect when a
minion is registering and re-name pre-generated keys. They don't have
to be created and seeded with the minion id, but they do need to be
named that way on the master once the minion id is set.

Unless anyone else can please look into that I'll try to get back to it
later. In the meantime I also suggest getting onto IRC and seeing what
bright ideas come up in #salt (on freenode network).

_S

On 10/31/2012 09:27 AM, ed.lane wrote:
>
>
> On Wednesday, October 31, 2012 9:49:17 AM UTC-6, Joseph wrote:
>>

>> On Wed, Oct 31, 2012 at 9:34 AM, ed.lane <ed.l...@gmail.com <javascript:>>

[Webmail EDLane]

Sean,

I am really embarrassed I missed the "auto_accept" feature in the master config file

entirely. I thought I read this file thoroughly in the past but I must have missed it.

This approach should work as you describe with the proper firewall rules.

An "auto_subscribe" feature like Chef or the one described by Ryan would be ideal but this

will probably suffice for now.

Thanks to everyone for the help.

-ed

[Dan Garthwaite]

Some ideas:

• What if the master could be put into 'auto accept' mode with either a unix signal, And then either reverted with another sigal, first minion, or timed out?
  
• A watch directory where a file could be touched for acceptable ID's.
• A yaml file describing conditions to accept new minion keys?
• ip address, name regex
• Acceptence by grain shared secret.  This will allow most any provision system to create the /etc/salt/minion file with a shared secret in the grains section.

[Ryan Schneider]

FWIW, my pull request with my first pass on x509 support is here:

https://github.com/saltstack/salt/pull/2416

I'll address Thomas' questions when I get to the office. 

Thanks,

Ryan

[Thomas S Hatch]

There is a yaml file that can be set to auto accept keys that match expressions

[ed.lane]

There seems to be part of this thread missing but I am intrigued...

Who do I have to buy lunch to get this feature???

-ed

[Dan Garthwaite]

On Thu, Nov 1, 2012 at 1:15 PM, Thomas S Hatch <that...@gmail.com> wrote:

There is a yaml file that can be set to auto accept keys that match expressions

Well, *high five* then.  Awesome. 

[Thomas S Hatch]

http://docs.saltstack.org/en/latest/ref/configuration/master.html#autosign-file

there it is :)