how to associate certificates created with keywhiz

[Lakshmi Narasimhan Sundararajan]

Team,
I am a new user with keywhiz.

Frankly, it was a breeze understanding vault. Having understood vault, I wanted to check keywhiz.
Two items are favorable with keywhiz, because of its fusefs integration for secrets and way for automating everything.
vault concept of seal/unseal is a showstopper here. But ease of setup and use is definitely for vault so far.

With that brief context out of the way, I have a 2 node cluster. Master and Minion.
I used both certstrap and openssl directly to create keys and certifications.
But I do not understand how to make the keywhiz server/cli and keywhiz-fs use these.

How do I do this?

Given that my nodes are not running local, I had to use SAN to setup DNS and IP for server.
I was able to provide these inputs to certstrap too. Would appreciate if this got covered in the writeup too.

Below are some items I see as implicitly assumed to be known to everyone.
1) What are configuration files for keywhiz (server/cli and keywhiz-fs)?
2) Where do they reside?
3) Server and Client running in the same node is assumed everywhere, even with cert creation (so only localhost or 127.0.0.1 is your keywhiz server).
This is not really the case.
4) How to associate certificates at both the server and client. I have never worked with/or installed certificates. I had to read up a lot to understand the whole concept. It is assumed here everyone knows how to work/create/install certificates. This is not a breeze so far.
5) keywhiz has concept of groups that contains secrets and clients within it. But the keywhiz-fs identifies as user. So how is the user here mapped to the client?
6) With secrets there is additionally a metadata concept. What is the idea/usage behind it?
7) Additionally, I do not come from java background, so keytool that stores certificates for java was not obvious to me.

Taking a simple 2 node cluster and walking us with a documentation would really be appreciated.

With warm regards and wishes for a very bright and happy new year 2016, I look forward to your replies.
LN

[Matthew McPherrin]

Yes, our docs on how to go from a test or development deploy on a single machine are lacking.  One of my tasks for the near future is to split our documentation into three sections (test deployment, developing keywhiz, and production deployment).

1 & 2) The keywhiz server and cli configuration file is passed in on the command line.  In our deployments, we put it in the same folder as the .jar file on our server, but you can put it wherever you like.  A sample development config is ./server/src/main/resources/keywhiz-development.yaml

keywhiz-fs has no configuration file.  All parameters are passed in on the command line.

3) Indeed, localhost is how we develop and test, but not use keywhiz in production.  It should just be a matter of supplying the correct hostnames instead of 127.0.0.1...

4) Yeah, our docs don't have a full guide on how to run a Certificate Authority.  A CA has to issue certs for the clients, and you need to set that as the trust store in the server.  I think this is a fairly niche thing to know, so we can definitely include better documentation for that.

5) keywhiz-fs clients are put into a group.  Secrets are put into a group. The server looks at the client certificate to figure out what the name of the client is, and presents secrets in its groups.

6) Metadata is used to make any changes to the unix permissions needed for a secret.  Some other programs want particular permissions on their secrets (eg, ssh can be picky about this), so we provide a way to tighten the unix permissions.

I will try to incorporate some of these points into our documentation in the coming year.  I appreciate you taking the time to point out what can be confusing in our infrastructure.

--
You received this message because you are subscribed to the Google Groups "keywhiz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keywhiz-user...@googlegroups.com.
To post to this group, send email to keywhi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/062823d9-b977-40ee-924f-5c163666c213%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lakshmi Narasimhan Sundararajan]

Thanks for your pointers.  But I will have to tell they are still at 10000 feet level.

As a first step to associate certificates and keys I did the following, especially for my setup with a master and a minion.
a) After reading about ssl I created a custom script that does the following.
--> create a root authority
--> create a intermediate authority
---> create certificates and keys for all nodes in setup (defined by nodes.yaml file)

I patched the openssl.cnf file to have req_extensions to have SANs included. SAN definition, is part of nodes.yaml

dir view:
---> contains openssl.cnf.ca /* patched for SAN */
---> nodes.yaml /* defines cluster */
---> certutil.py /* a python script that does everything, it is not fully automated though */
1) Creates a Root CA Authority (give CN=RootAuth)
2) Creates an Intermediate CA Authority and a CA Chain (give CN=InterAuth)
3) Creates keys and certificates for all nodes in the cluster signed by the intermediate authority.

< Invoke as python certutil.py>

After this, I created truststore and keystore files as below.

keytool -import -file 'intermediate/cert/ca-chain.cert.pem' -alias ca \
 -storetype pkcs12 -storepass ponies -keystore dev_and_test_truststore.p12

~ openssl pkcs12 -export -in intermediate/cert/master.cert.pem \
      -inkey intermediate/private/master.key.pem -out dev_and_test_keystore.p12

Copied server/src/main/resources/keywhiz-development.yaml.h2 as keywhiz-development.yaml
And modified truststore and keystore paths. Left the others as default.

The app crashes with certificate invalid. I had the certs validated through openssl, they seem fine.
Any pointers to get through setting up certificates for this cluster environment?

I have attached the files for your reference here. Your inputs much appreciated.

certutil.py:
=================================================
import yaml
import os
import shutil
import sys
import tempfile
import stat

# needed inputs
CONF="openssl.cnf.ca" # Root CA configuration
INTER="1nt3rm3d1@t3" # Intermediate Authority password
SECRET="s3cr3t" # Root AUthority password
TESTBED='nodes.yaml' # this is  yaml file that describes the testbed


def setup_dir(workd, newconf):
  shutil.copy(CONF, newconf)
  for i in ["certs", "crl", "newcerts", "private", "csr"]:
    os.mkdir("%s/%s" %(workd, i))

  open("%s/index.txt" % workd, 'w').close()
  #chmod 700 private
  os.chmod("%s/private" % workd, stat.S_IRWXU)
  for i in ["serial", "crlnumber"]:
    with open("%s/%s" %(workd, i), "w") as myfile:
      myfile.write("1000")

  #touch index.txt
  #echo 1000 > serial
  #echo 1000 > crlnumber
  #sed -i "s#^dir\(.*\)=.*#dir\1= $dir#" $CONF
  os.system("sed -i.bak \"s#^dir.*=.*#dir  = %s#\" %s" % (workd, newconf))
  #popd

def create_secure_key(passwd, out):
  cmd="openssl genrsa -aes256 -passout pass:%s -out %s 4096" % (passwd, out)
  os.system(cmd)

def create_key(out):
  cmd="openssl genrsa -out %s 2048" % out
  os.system(cmd)

def create_root_cert(conffile, key, keypass, outcert):
  cmd="openssl req -config %s -key %s -passin pass:%s -new -x509 -days 7300 -sha256 -extensions v3_ca -out %s" % \
               (conffile, key, keypass, outcert)
  os.system( cmd)
  os.chmod(outcert, stat.S_IREAD)


def patch_intermediate_config_properties():
  #sed -i "s#^dir\(.*\)=.*#dir\1= $int#" $int/$CONF
  os.system("sed -i.bak \"s#^dir.*=.*#dir = %s#\" %s" %(inter, INTERCACONF))
  #sed -i "s#^private_key\(.*\)=.*#private_key\1= \$dir/private/intermediate.key.pem#" $int/$CONF
  os.system("sed -i \"s#^private_key.*=.*#private_key = \$dir/private/intermediate.key.pem#\" %s" % INTERCACONF)
  #sed -i "s#^certificate\(.*\)=.*#certificate\1= \$dir/certs/intermediate.cert.pem#" $int/$CONF
  os.system("sed -i \"s#^certificate.*=.*#certificate = \$dir/certs/intermediate.cert.pem#\" %s" % INTERCACONF)
  #sed -i "s#^crl\(.*\)=.*#crl\1= \$dir/crl/intermediate.crl.pem#" $int/$CONF
  os.system("sed -i \"s#^crl.*=.*#crl = \$dir/crl/intermediate.crl.pem#\" %s" % INTERCACONF)
  #sed -i "s/^policy\(.*\)=.*/policy\1= policy_loose/" $int/$CONF
  os.system("sed -i \"s/^policy.*=.*/policy = policy_loose/\" %s" % INTERCACONF)

def create_csr_with_secure_key(conffile, key, keypass, outcsr):
  # openssl req -config $CONF -key private/intermediate.key.pem -passin pass:$INTER -new -sha256 -out csr/intermediate.csr.pem
  os.system("openssl req -config %s -key %s -passin pass:%s -new -sha256 -out %s" % (conffile, key, keypass, outcsr))

def create_csr(conffile, key, outcsr):
  # openssl req -config $CONF -key private/node.key.pem -new -sha256 -out csr/node.csr.pem
  os.system("openssl req -config %s -key %s -new -sha256 -out %s" % (conffile, key, outcsr))

#$(cd "$dir" && openssl ca -config $CONF -passin pass:$SECRET -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem)
def create_inter_cert(caconf, capass, csr, outcert):
  os.system("openssl ca -config %s -passin pass:%s -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in %s -out %s" % \
                 (caconf, capass, csr, outcert))

def create_server_cert(caconf, capass, csr, outcert):
  os.system("openssl ca -config %s -passin pass:%s -extensions server_cert -extensions v3_req -days 375 -notext -md sha256 -in %s -out %s" % (caconf, capass, csr, outcert))


def create_client_cert(caconf, capass, csr, outcert):
  os.system("openssl ca -config %s -passin pass:%s -extensions usr_cert -extensions v3_req -days 375 -notext -md sha256 -in %s -out %s" % (caconf, capass, csr, outcert))

print "1. Setup top directory"
top=tempfile.mkdtemp(prefix="CERT", dir=os.getcwd())
inter="%s/intermediate" % top
os.mkdir(inter)

shutil.copy(TESTBED, top)

print "initializing top directory.. %s" %(top)
CACONF="%s/%s" % (top,CONF)
CAKEY="%s/private/ca.key.pem" % (top)
CACERT="%s/certs/ca.cert.pem" % (top)
setup_dir(top, CACONF)

INTERCACONF="%s/%s" % (inter, CONF)
INTERCAKEY="%s/private/intermediate.key.pem" % (inter)
INTERCACERT="%s/certs/intermediate.cert.pem" % (inter)
INTERCACSR="%s/csr/intermediate.csr.pem" % (inter)
setup_dir(inter, INTERCACONF)

CACHAIN="%s/certs/ca-chain.cert.pem" % (inter)

print "2. Create root key"
#openssl genrsa -aes256 -passout pass:$SECRET -out $top/private/ca.key.pem 4096
create_secure_key(SECRET, CAKEY)

print "3. create root certificate"
create_root_cert(CACONF, CAKEY, SECRET, CACERT)
#$(cd "$dir" && openssl req -config $CONF -key private/ca.key.pem -passin pass:$SECRET -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem)

print "4. Prepare intermediatary certificate"
patch_intermediate_config_properties()

print "5. create intermediate key"
create_secure_key(INTER, INTERCAKEY)

print "6. create intermediate request"
#$(cd "$int" && openssl req -config $CONF -key private/intermediate.key.pem -passin pass:$INTER -new -sha256 -out csr/intermediate.csr.pem)
create_csr_with_secure_key(INTERCACONF, INTERCAKEY, INTER, INTERCACSR)

print "7. Sign intermediate request to create a certificate"
#$(cd "$dir" && openssl ca -config $CONF -passin pass:$SECRET -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem)
create_inter_cert(CACONF, SECRET, INTERCACSR, INTERCACERT)

print "8. Install root certificate in all clients"
print "9. create ca trust chain, to enable clients without knowing root to validate successfully."
#cat $int/certs/intermediate.cert.pem $dir/certs/ca.cert.pem  > $int/certs/ca-chain.cert.pem
shutil.copy(INTERCACERT, CACHAIN)
with open(CACHAIN, "ab") as f:
  with open(CACERT, "rb") as s:
      f.write(s.read())

print "10. creating certs for nodes in TESTED[%s] based on conf file %s" % (TESTBED, INTERCACONF)

s=open(TESTBED,'r')
d=yaml.safe_load(s)

for node,info in d.items():
  print "Node: ", node
  print "Server: ", info.get('server', False)
  print "IP: ", info.get('ip', "127.0.0.1")
  print "DNS: ", info.get('dns', "localhost")

  # create and patch conf file for each node specific to SAN
  nconf="%s/%s-%s" %(inter, node,CONF)
  nkey="%s/private/%s.key.pem" % (inter, node)
  ncsr="%s/csr/%s.csr.pem" % (inter, node)
  ncert="%s/certs/%s.cert.pem" % (inter, node)

  shutil.copy(INTERCACONF, nconf)
  os.system('sed -i.bak \"/\[ alt_names \]/,+3d\" %s' % nconf)

  with open(nconf, "a") as myfile:
    myfile.write("\n")
    myfile.write("[ alt_names ]\n")
    for index, val in enumerate(info.get('ip', "127.0.0.1")):
        myfile.write("IP.%d=%s\n" %(index, val))
    for index, name in enumerate(info.get('dns', "localhost")):
        myfile.write("DNS.%d=%s\n" % (index, name))

  # omit -aes256 to drop passwords for key
  # openssl genrsa -aes256 -passout pass:$INTER -out intermediate/private/key 2048
  create_key(nkey)

  # create csr
  # config file is updated with SAN (ip/dns)
  create_csr(nconf, nkey, ncsr)

  # create cert signed with intermediate certificate auth
  if info.get('server', False):
    create_server_cert(nconf, INTER, ncsr, ncert)
  else:
    create_client_cert(nconf, INTER, ncsr, ncert)

print "All done"
=================================================


The testbed definition...
nodes.yaml:
==================================================
master:
  server: true
  ip:
    - 127.0.0.1
    - 10.134.193.180
  dns:
    - localhost
    - master

minion:
  server: false
  ip:
    - 10.134.194.187
  dns:
    - minion

==================================================

3. openssl cnf file
===========================================
# OpenSSL root CA configuration file.
# Copy to `/root/ca/openssl.cnf`.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /home/cephadm/certs/root/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_loose

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only
req_extensions      = v3_req

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = US
stateOrProvinceName_default     = California
localityName_default            = Fremont
0.organizationName_default      = DemoStore
organizationalUnitName_default  = Certs
emailAddress_default            = de...@demostore.org

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning

# below has to be the last few lines (script assumes so)
[ alt_names ]
DNS.1=localhost
IP.1=127.0.0.1
===========================================

[Lakshmi Narasimhan Sundararajan]

Team, the certificate setting up still does not work for me. Would you please help me understand what am I missing.
This time, I followed the wiki (https://github.com/square/keywhiz/wiki/Development-and-test-key-material) exactly with no changes at all.
Whereever password was asked, I gave as ponies.

I get same error as earlier. Below I paste the history cmds.
 1308  ./certstrap init --key-bits 2048 --years 30 --common-name "Keywhiz Test CA"
 1309  keytool -import -file 'out/Keywhiz_Test_CA.crt -alias ca -storetype pkcs12 -storepass ponies -keystore out/Keywhiz_Test_CA.p12
 1310  keytool -import -file out/Keywhiz_Test_CA.crt -alias ca -storetype pkcs12 -storepass ponies -keystore out/Keywhiz_Test_CA.p12
 1311  history
 1312  cp out/Keywhiz_Test_CA.p12 ../keywhiz/dev_and_test_truststore.p12
 1313  ./certstrap request-cert --domain localhost --ip 127.0.0.1 --organizational-unit server
 1314  ./certstrap sign --years 30 --CA "Keywhiz Test CA" localhost
 1315  openssl pkcs12 -export -in out/localhost.crt -inkey out/localhost.key -out out/localhost.p12
 1316  cp out/localhost.p12 ../keywhiz/dev_and_test_keystore.p12
 1317  cd ../keywhiz
 1318  java -jar server/target/keywhiz-server-*-SNAPSHOT-shaded.jar server keywhiz-development.yaml | tee logs

Error logs:
INFO  [2016-01-12 09:00:19,486] org.eclipse.jetty.setuid.SetUIDListener: Opened admin@44b194fe{HTTP/1.1}{localhost:8085}
INFO  [2016-01-12 09:00:19,491] org.eclipse.jetty.server.Server: jetty-9.2.z-SNAPSHOT
WARN  [2016-01-12 09:00:19,608] org.eclipse.jetty.util.component.AbstractLifeCycle: FAILED SslContextFactory@72f8ae0c(dev_and_test_keyst
ore.p12,dev_and_test_truststore.p12): java.security.cert.CertificateException: Unable to validate certificate: unable to find valid cert
ification path to requested target
! sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
! at sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source) ~[na:1.8.0_66]

What am I doing wrong? Pointers appreciated.
LN

[Lakshmi Narasimhan Sundararajan]

And my only updates were to pick my build cert instead of the defaults in the spec. And I copied the config file to local dir to update it.
The kw server was invoked with this file as argument.

[user@machine keywhiz]$ diff keywhiz-development.yaml ./server/src/main/resources/keywhiz-development.yaml.h2
14c14
<       keyStorePath: dev_and_test_keystore.p12
---
>       keyStorePath: server/src/main/resources/dev_and_test_keystore.p12
17c17
<       trustStorePath: dev_and_test_truststore.p12
---
>       trustStorePath: server/src/main/resources/dev_and_test_truststore.p12
[cephadm@lns-master-0 keywhiz]$

LN

[Matthew McPherrin]

What you did looks right to me.  I'll try to replicate today and see what works.

What OS and Java version are you using?

--

You received this message because you are subscribed to the Google Groups "keywhiz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keywhiz-user...@googlegroups.com.
To post to this group, send email to keywhi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/3543bd8e-1104-4da4-9d33-84627edbda6c%40googlegroups.com.

[Lakshmi Narasimhan Sundararajan]

==> uname -a
Linux lns-master-0 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
===> cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

===> java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

Looking forward to your inputs to get this up and running.

LN

[Matthew McPherrin]

I have gone through setting up a CA infrastructure for Keywhiz as per the wiki page, and haven't found any errors in the instructions that would lead to what you're seeing.

Could you send me the dev_and_test_keystore.p12 and dev_and_test_truststore.p12 you've generated, and perhaps I can find the problem by examining them?  I don't know if google groups allows attachments: please email them to m...@squareup.com.

I should have some more time tomorrow to help debug this problem further.

To view this discussion on the web visit https://groups.google.com/d/msgid/keywhiz-users/fa09a077-e99b-4d9c-8d75-1d30a302cac9%40googlegroups.com.