Best practises for managing secret keys with puppet?
Daniel Pittman
One of the current problems we face with our puppet network, and about which
I would like to solicit opinions, is the distribution of secret keys for
(mostly) SSL secured services.
The most pressing example of this is that for reasons of availability want to
process SSL on multiple machines for a single certificate. The same issue
applies to a wide range of internal and public services, though, where we need
to distribute key material.[1]
The prospect of putting the secret key into our revision control system has
... well, little appeal is probably being fair: we could certainly do it, but
it suddenly means that a whole bunch of extra data has to be treated as high
security rather than low security.[2]
On the other hand, it would be good if a trusted agent like puppet can be
configured to automatically bring up nodes that run secure services from
scratch with no human intervention involved. That helps with availability,
as well as security issues like key rollover.
On the gripping hand, I mistrust human involvement: if I have to give *people*
(including me) access to the key material, rather than software, I have a
whole set of less trustworthy and reliable agents handling it.
So, on the whole my feeling is that an automatic "key distribution service"
that was accessible to puppet but (mostly) not to people would be ideal.
I also feel that it would be ideal to keep that service distinct from our
standard puppet Subversion repository, to keep the need for security
boundaries small.
I am interested here in other views on my assessment of the problem, and in
pointers to practical tools used to solve the problem.
Daniel
Oh, and at the moment I am considering the puppet client/server protocol
trustworthy enough to enforce access control to the key material. :)
Footnotes:
[1] Generally, key material where we don't require a passphrase to unlock
before use, because we have made the availability/security decision based
on our business needs.
[2] ...well, technically "medium grade" security, since at present reading
our puppet manifests is worth little to an attacker, but writing to them
is valuable. So, we protect writes heavily but don't worry too much
about, for example, a backup of the VCS repository they live in.
--
✣ Daniel Pittman ✉ dan...@rimspace.net ☎ +61 401 155 707
♽ made with 100 percent post-consumer electrons
Michael DeHaan
G'day.
One of the current problems we face with our puppet network, and about which
I would like to solicit opinions, is the distribution of secret keys for
(mostly) SSL secured services.
The most pressing example of this is that for reasons of availability want to
process SSL on multiple machines for a single certificate. The same issue
applies to a wide range of internal and public services, though, where we need
to distribute key material.[1]
The prospect of putting the secret key into our revision control system has
... well, little appeal is probably being fair: we could certainly do it, but
it suddenly means that a whole bunch of extra data has to be treated as high
security rather than low security.[2]
One way to handle this would be by keeping confidential information in a seperate version control repository (not public), rather than in your main one. Puppet has a system of module paths so you could keep your confidential info seperate from the content you would want to give to everyone who would normally be working with Puppet, and check *both* of these out on the Puppet server. For development systems/testing, you could just check out a copy of a different repo, with testing/stage credentials in the modules instead.
How is everyone else handling this?
--Michael
Nigel Kersten
> G'day.
>
> One of the current problems we face with our puppet network, and about which
> I would like to solicit opinions, is the distribution of secret keys for
> (mostly) SSL secured services.
>
> The most pressing example of this is that for reasons of availability want to
> process SSL on multiple machines for a single certificate. The same issue
> applies to a wide range of internal and public services, though, where we need
> to distribute key material.[1]
>
>
> The prospect of putting the secret key into our revision control system has
> ... well, little appeal is probably being fair: we could certainly do it, but
> it suddenly means that a whole bunch of extra data has to be treated as high
> security rather than low security.[2]
What about keeping encrypted versions of the key in your revision
control system, and then writing a function for the puppet servers
that decrypts them using a key that you need to distribute out of band
to your puppet servers?
You'd still be sending the unencrypted data across, but this should be
a secured connection, and it will end up in your client catalogs, but
they should be protected just as much as the data on the client disk
would be.
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
--
nigel
Alan Barrett
> The prospect of putting the secret key into our revision control
> system has ... well, little appeal is probably being fair: we could
> certainly do it, but it suddenly means that a whole bunch of extra
> data has to be treated as high security rather than low security.[2]
I configure puppet to print an error message that explains the
situation:
# util::manually_copied_file -- set permissions on a manually-copied
# file, and print an error message if the file is missing.
#
# usage:
# util::manually_copied_file { "/dirname/filename":
# message => "where to copy it from, or why it's not in puppet",
# owner => root,
# group => bin,
# mode => 0400,
# }
#
define util::manually_copied_file($message, $owner, $group, $mode)
{
# If the file exists and has a size > 0, then do nothing.
# Otherwise, print an error message and fail.
exec { "util::manually_copied_file check $name":
unless => "/bin/test -s $name",
command => "/bin/cat <<'EOF'; /bin/false
Please copy ${name} manually - ${message}
EOF
",
logoutput => true,
before => File[$name],
require => [],
}
# Set the ownership and permissions, but do not modify the content
file { $name:
ensure => file,
replace => false,
owner => $owner,
group => $group,
mode => $mode,
}
}
> So, on the whole my feeling is that an automatic "key distribution
> service" that was accessible to puppet but (mostly) not to people
> would be ideal.
That would be nice.
--apb (Alan Barrett)
Thomas Bellman
> One way to handle this would be by keeping confidential information in a
> seperate version control repository (not public), rather than in your
> main one. Puppet has a system of module paths so you could keep your
> confidential info seperate from the content you would want to give to
> everyone who would normally be working with Puppet, and check *both* of
> these out on the Puppet server. For development systems/testing, you
> could just check out a copy of a different repo, with testing/stage
> credentials in the modules instead.
>
> You could also use a custom function to pull this information from other
> sources for accessing a keystore server side, though I'd be curious to
> what those other services might be.
>
> How is everyone else handling this?
We have a filerserving module named "private" that is defined like
[private]
path /config/private/%h
allow *
in fileserver.conf. Each client gets its own subdirectory
"/config/private/client1", "/config/private/client2", and so on.
In there we stuff things like SSH host keys, X.509 host certificates,
license keys, and other secret data, and let Puppet install them via
file {
"/etc/ssh/ssh_host_rsa_key":
source => "puppet:///private/ssh_host_rsa_key", ...;
}
We currently don't have /config/private under version control, but
if we did it would definitely be in a separate repository from
our manifests.
/Bellman
Joe McDonagh
> G'day.
Hey Daniel, your puppet SSL keys can be used for other services as well.
I successfully used them as authentication for Splunk's SSL receiver
when I was piloting the software. IDK if this helps you, but I feel like
this tidbit may get overlooked sometimes. How do you currently manage
your puppet keys?
--
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L'ennui est contre-révolutionnaire
Daniel Pittman
> Daniel Pittman wrote:
>
> Hey Daniel, your puppet SSL keys can be used for other services as well.
*nod* Sadly, we need a whole bunch of different public SSL services,
including SSL services on host names and domains that we operate on the behalf
of clients.
It was, in fact, mostly services like that which I was considering when
I asked the initial question. So, while reuse of the puppet keys would be
occasionally convenient for low value stuff, it doesn't solve my bigger
problem.
[...]
> How do you currently manage your puppet keys?
Entirely manually, using the puppet CA, because it just works, and because we
don't have any significant cross-over between the internal CA we use and the
places that we run puppet.
Daniel