Deploying Simian Client: invalid PEM key format: %s' % str(e)

[Thomas Maerz]

I am trying to run the preflight check /usr/local/munki/preflight --debug and getting an invalid PEM key format error:

DEBUG:root:Certname from facter: "None"
WARNING:root:Certname was not found in facter!
DEBUG:root:_GetPuppetSslDetails(None.pem)
DEBUG:root:_ValidatePuppetSslCert: required_issuer <OBFUSCATED, BUT THEY MATCH>
DEBUG:root:_ValidatePuppetSslCert: default_required_issuer <OBFUSCATED, BUT THEY MATCH>
DEBUG:root:_ValidatePuppetSslCert: /etc/simian/ssl/certs/None.pem
DEBUG:root:Skipped cert None.pem, IO Error [Errno 2] No such file or directory: '/etc/simian/ssl/certs/None.pem'
ERROR:root:Failed to harvest Puppet SSL cert facter specified.
DEBUG:root:_GetNewestPuppetSslCert found certs QZ120011F5W.pem
DEBUG:root:_ValidatePuppetSslCert: required_issuer <OBFUSCATED, BUT THEY MATCH>
DEBUG:root:_ValidatePuppetSslCert: default_required_issuer <OBFUSCATED, BUT THEY MATCH>
DEBUG:root:_ValidatePuppetSslCert: /etc/simian/ssl/certs/QZ120011F5W.pem
DEBUG:root:Looking at issuer <OBFUSCATED, BUT THEY MATCH>
DEBUG:root:_GetPuppetSslDetails found cert QZ120011F5W.pem with timestamp 1446495123.0
DEBUG:root:_GetPuppetSslDetails priv should be /etc/simian/ssl/private_keys/QZ120011F5W.pem
DEBUG:root:_Get(ca_id)
DEBUG:root:_GetExternalValue(ca_id)
DEBUG:root:_GetExternalConfiguration(settings)
DEBUG:root:_Get(ca_public_cert_pem)
DEBUG:root:_GetExternalPem(ca_public_cert_pem)
DEBUG:root:_Get(server_public_cert_pem)
DEBUG:root:_GetExternalPem(server_public_cert_pem)
DEBUG:root:_Get(required_issuer)
DEBUG:root:_GetExternalValue(required_issuer)
Traceback (most recent call last):
  File "/usr/local/munki/simian_client.py", line 87, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/munki/simian_client.py", line 73, in main
    preflight.RunPreflight(runtype, server_url=server_url)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/preflight.py", line 365, in RunPreflight
    secure_config, client_id, user_settings, client_exit)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/preflight.py", line 139, in LoginToServer
    token = client.GetAuthToken()
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1828, in GetAuthToken
    self.DoSimianAuth()
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1346, in DoSimianAuth
    self._InitializeAuthClass(interactive_user)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1300, in _InitializeAuthClass
    auth1.LoadSelfKey(o['priv_key'])
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/auth/base.py", line 779, in LoadSelfKey
    key = self._LoadKey(keystr)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/auth/base.py", line 753, in _LoadKey
    raise ValueError('invalid PEM key format: %s' % str(e))
ValueError: invalid PEM key format: None

My private key is generated as per https://github.com/google/simian/wiki/SimianAndCertificates and is located in /etc/simian/ssl/private_keys and looks like this:

-----BEGIN PRIVATE KEY-----
BLAH BLAH BLAH
BLAH BLACH BLAH
-----END PRIVATE KEY-----

I also have my certificate name properly in /etc/simian/settings.cfg:

# Simian Configuration File.

#

# This file must live on client machines at /etc/simian/settings.cfg

#

# For more information, please visit the following Wiki page:

#   http://code.google.com/p/simian/wiki/Configuration

#

[settings]

# This should match your App Engine Application ID.

# For example, if your app is http://example.appspot.com this is "example"

subdomain = <obfuscated>

# Do not change unless you're using a Google Apps Domain with App Engine.

domain = appspot.com

# The full DN of the CA that Simian certificates are signed by.

required_issuer = <OBFUSCATED, BUT THEY MATCH>

# Root CA Cert chain PEM path; default is provided with Simian.

root_ca_cert_chain_pem = /usr/local/munki/simian/roots.pem

# Path to directory of client certificates (client public cert and private key).

# If you're using Puppet for configuration management, this can be set to:

#   /etc/puppet/ssl/

#

# This directory should contain two subdirectories:

# ./certs/<certname below or from facter>.pem

# ./private_keys/<private key of cert above>.pem

client_ssl_path = /etc/simian/ssl/

# Domain for simianadmin uauth; set to your Google Apps Domain if not gmail.

auth_domain = gmail.com

# Enable Apple SUS integration.

applesus = true

# Variables to supplement Puppet's facter.

# Any variables not returned by facter will be populated from the values here.

# For deployments lacking Puppet/facter, these will be used exhaustively.

certname = QZ120011F5W.pem

primary_user = foouser

hostname = tenten

configtrack = stable

simiantrack = stable

site = MTV

location = US-MTV-FOO

Any ideas? I've tried regenerating my certificates already.

[Justin McWilliams]

Are you running as root, or with sudo?  This is required for preflight.

Interesting log output:

DEBUG:root:Certname from facter: "None"

WARNING:root:Certname was not found in facter!

...

DEBUG:root:_ValidatePuppetSslCert: /etc/simian/ssl/certs/None.pem

DEBUG:root:Skipped cert None.pem, IO Error [Errno 2] No such file or directory: '/etc/simian/ssl/certs/None.pem'

...

It seems the client went on to find and use QZ120011F5W.pem, but it's be nice to rule out the above issue.  Does "sudo /usr/local/bin/simianfacter" output the certname specified in settings.cfg, or an empty value (or worse, string "None")?

For posterity, here's the exception (but it's unclear if SyntaxError or AttributeError is being caught, as str(e) is evaling to "None): https://github.com/google/simian/blob/master/src/simian/auth/base.py#L753

--
You received this message because you are subscribed to the Google Groups "Simian Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simian-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Maerz]

Yes, I am running preflight as root on the client machine.

/usr/local/bin/simianfacter outputs:

tenten:test root# /usr/local/bin/simianfacter

/usr/local/munki/simian/lib/python2.6/site-packages/tlslite-0.3.8-py2.6.egg/tlslite/utils/cryptomath.py:9: DeprecationWarning: the sha module is deprecated; use the hashlib module instead

certname => QZ120011F5W

primary_user => foouser

sp_local_host_name => tenten

configtrack => stable

simiantrack => stable

site => MTV

location => US-MTV-FOO

applesus => True

I've also tried running it with the certname set to both QZ120011F5W.pem and QZ120011F5W with no file extension.

Thomas

[Justin McWilliams]

If you're comfortable editing the source by hand (feel free to say you're not), can you add a logging line to output the value of the "keystr" var?  This will at least show you the value of the PEM attempting to be validated, to confirm if it's empty, malformed, None, or something else unexpected.  Otherwise, I can take a crack at adding some more validation with logging and output there...

https://github.com/google/simian/blob/master/src/simian/auth/base.py#L753

[Thomas Maerz]

I outputted the value of keystr right before the error being thrown:

-----BEGIN PRIVATE KEY-----

Matching private key contents with .pem file

-----END PRIVATE KEY-----

Is that supposed to be a filename or the contents of the file?

Thomas

[Justin McWilliams]

It's supposed to be the full contents.  And I think we handle erroneous leading/trailing whitespace and newlines, but nothing else.

Can you also remove the "except (SyntaxError, AttributeError), e:" and subsequent logging line?  I'd like to see what exception is being thrown within tlslite....

[Thomas Maerz]

Alright. I'm no developer, but I gave it a shot. It won't complete when I comment out the two lines you mentioned so I improvised:

    try:

      print keystr

      key = tlslite.utils.keyfactory.parsePEMKey(keystr)

    except (SyntaxError, AttributeError), e:

       print "Printing e"

       print e

       raise

#      raise ValueError('invalid PEM key format: %s' % str(e))

    return key

Output:

Printing e

None

Traceback (most recent call last):

  File "/usr/local/munki/simian_client.py", line 87, in <module>

    sys.exit(main(sys.argv[1:]))

  File "/usr/local/munki/simian_client.py", line 73, in main

    preflight.RunPreflight(runtype, server_url=server_url)

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/preflight.py", line 365, in RunPreflight

    secure_config, client_id, user_settings, client_exit)

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/preflight.py", line 139, in LoginToServer

    token = client.GetAuthToken()

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1828, in GetAuthToken

    self.DoSimianAuth()

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1346, in DoSimianAuth

    self._InitializeAuthClass(interactive_user)

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1300, in _InitializeAuthClass

    auth1.LoadSelfKey(o['priv_key'])

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/auth/base.py", line 783, in LoadSelfKey

    key = self._LoadKey(keystr)

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/auth/base.py", line 752, in _LoadKey

    key = tlslite.utils.keyfactory.parsePEMKey(keystr)

  File "build/bdist.macosx-10.10-intel/egg/tlslite/utils/keyfactory.py", line 146, in parsePEMKey

  File "build/bdist.macosx-10.10-intel/egg/tlslite/utils/OpenSSL_RSAKey.py", line 141, in parse

SyntaxError: None

[Justin McWilliams]

Well, that's quite unhelpful exception output, thanks tlslite :(   https://github.com/trevp/tlslite/blob/master/tlslite/utils/openssl_rsakey.py#L100

I see raise SyntaxError() in a number of cases.  Your cert clearly started with "----BEGIN PRIVATE KEY-----", so it must be one of the subsequent issues.

Does this work (i.e. "RSA key ok"), or fail with something enlightening?

  openssl rsa -check -noout -in /etc/simian/ssl/private_keys/QZ120011F5W.pem

[Thomas Maerz]

tenten:~ root# openssl rsa -check -noout -in /etc/simian/ssl/private_keys/QZ120011F5W.pem

RSA key ok

[Thomas Maerz]

I suppose unless anyone has any other ideas, the next option I can think of to continue moving forward with this is to go ahead and deploy puppet as well since that is how Google does it, it is probably more likely to work with Puppet and Facter on the system.

Thomas

[Thomas Maerz]

I deployed puppet in my test environment  and all is well now.

...