Puppet SSL Certificates not working

[Michael Holt]

I'm trying to get Simian working with Puppet SSL Certs and am having some issues.

Here's what I've done:

1) Retrieved ca_crt.pem (/lib/ssl/ca/ca_crt.pem) from Puppet Server and imported it to ca_public_cert_pem in Simian

2) Ran puppet cert generate mysimianinstance.appspot.com to create certificates for Simian

3) Imported Private Key & Public Cert that I just created into Simian

4) Added the ca_cert.pem and Public Cert created in step 2 into PWD/etc/simian/ssl per https://github.com/google/simian/wiki/PackagingDeployableClient

5) changed settings file to my environment

6) Changed client_ssl_path to the Puppet SSL Directory

7) Built the DMG and installed on test client.

This is what I am getting at this point from Munki:

 

Starting...
    Performing preflight tasks...
    preflight stderr: /usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/gae_client.zip/google/appengine/tools/dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead
/usr/local/munki/simian/lib/python2.6/site-packages/tlslite-0.3.8-py2.6.egg/tlslite/utils/cryptomath.py:9: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
WARNING:root:Root CA Cert Chain was EMPTY!
ERROR:root:Failed to harvest Puppet SSL cert facter specified.
WARNING:root:Root CA Cert Chain was EMPTY!
ERROR:root:MunkiDownloadError getting Apple SUS catalog.
Traceback (most recent call last):
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/flight_common.py", line 597, in GetAppleSUSCatalog
    '%s/applesus/' % url, APPLE_SUS_CATALOG)
  File "/usr/local/munki/munkilib/updatecheck.py", line 3478, in getResourceIfChangedAtomically
    verify=verify)
  File "/usr/local/munki/munkilib/fetch.py", line 277, in getResourceIfChangedAtomically
    message=message, resume=resume, follow_redirects=follow_redirects)
  File "/usr/local/munki/munkilib/fetch.py", line 390, in getHTTPfileIfChangedAtomically
    raise GurlDownloadError(err)
GurlDownloadError: HTTP result 403: forbidden
Checking for available updates...
    Retrieving list of software for this machine...
ERROR: Could not retrieve manifest W8042353AGZ from the server: HTTP result 403: forbidden
ERROR: Could not retrieve managed install primary manifest.
Checking Apple Software Update catalog...
    Skipping Apple Software Update check because sucatalog is unchanged, installed Apple packages are unchanged and we recently did a full check.
Finishing...
    Performing postflight tasks...
    postflight return code: 1
    postflight stderr: /usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/gae_client.zip/google/appengine/tools/dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead
/usr/local/munki/simian/lib/python2.6/site-packages/tlslite-0.3.8-py2.6.egg/tlslite/utils/cryptomath.py:9: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
WARNING:root:Root CA Cert Chain was EMPTY!
ERROR:root:GetAuth1Token(): AdditionalHttpHeaders lacks a token.
Traceback (most recent call last):
  File "/usr/local/munki/simian_client.py", line 87, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/munki/simian_client.py", line 75, in main
    postflight.RunPostflight(runtype)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/postflight.py", line 87, in RunPostflight
    client.PostReport('postflight', params)
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1774, in PostReport
    return self._SimianRequest('POST', '/reports', str(body))
  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/client/client.py", line 1467, in _SimianRequest
    raise SimianServerError(response.status, response.reason, response.body)
simian.client.client.SimianServerError: (403, 'Forbidden', '')
Done.

[Justin McWilliams]

This is the error of note:

   ERROR:root:Failed to harvest Puppet SSL cert facter specified.

Which means the certname specified in settings.cfg could not be found in the client_ssl_path path specified.

You can try "sudo /usr/local/munki/preflight --debug" for increased verbosity, which may give more insight to how the process of harvesting the cert was unsuccessful.  Logs lines in this method call: https://github.com/google/simian/blob/master/src/simian/client/client.py#L1113

Note, even if the specified certname is not found, Simian should fallback to the newest cert in the specified path: https://github.com/google/simian/blob/master/src/simian/client/client.py#L1148

If no certs are found, then it should hard fail, halting further Munki execution: https://github.com/google/simian/blob/master/src/simian/client/client.py#L1296

So I think it found a cert, but that cert may not be valid/match the server (hence 403)?  I thought we had better logs around such a failure, though; it's odd the preflight exited with "success", while Munki cannot connect, and postflight similary cannot confirm with confirmation that "AdditionalHttpHeaders lacks a token."

If debug output doesn't help you, please do share.

- Justin

--
You received this message because you are subscribed to the Google Groups "Simian Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simian-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Holt]

This is what i'm getting:

DEBUG:root:SSL configuring with context

DEBUG:root:SSL connect(('myapp.appspot.com', 443))

DEBUG:root:IsValidCert() ok=0 cert=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA, returning 0

DEBUG:root:SSL connected ('myapp.appspot.com', 443)

DEBUG:root:Requesting PUT /uploadfile/log/install.log

DEBUG:root:Waiting for response

DEBUG:root:Response status 200

DEBUG:root:UploadFile uploading file: /tmp/munki_ps_ef_output_ts_Khs/ps_ef_output

DEBUG:root:Do(PUT, /uploadfile/log/ps_ef_output) try #1

DEBUG:root:Connecting to https://myapp.appspot.com:None

DEBUG:root:Loaded 323284 bytes of CA cert chain and configured ctx

DEBUG:root:SSL configuring with context

DEBUG:root:SSL connect(('myapp.appspot.com', 443))

DEBUG:root:IsValidCert() ok=0 cert=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA, returning 0

DEBUG:root:SSL connected ('myapp.appspot.com', 443)

DEBUG:root:Requesting PUT /uploadfile/log/ps_ef_output

DEBUG:root:Waiting for response

DEBUG:root:Response status 200

WARNING:root:Sending -SIGKILL to installd

ERROR:root:Could not kill installd!

INFO:root:Reinstalling Munki client....

INFO:root:Fetching repair client from: https://myapp.appspot.com/repair

INFO:root:DEBUG2: Options: {'logging_function': <function display_debug2 at 0x105720d70>, 'additional_headers': {}, 'file': '/tmp/munki_repair_dmg_Zp_uvO/munkiclient.dmg.download', 'cache_data': None, 'url': u'https://myapp.appspot.com/repair', 'follow_redirects': False, 'download_only_if_changed': False, 'can_resume': False}

INFO:root:DEBUG2: connection_willSendRequestForAuthenticationChallenge_

INFO:root:DEBUG2: Authentication challenge for Host: myapp.appspot.com Realm: None AuthMethod: NSURLAuthenticationMethodServerTrust

INFO:root:DEBUG2: Allowing OS to handle authentication request

INFO:root:DEBUG1: Status: 403

INFO:root:DEBUG1: Headers: {u'Alternate-Protocol': u'443:quic,p=1', u'Content-Length': u'0', u'Server': u'Google Frontend', u'Cache-Control': u'no-cache', u'Date': u'Mon, 20 Jul 2015 21:52:44 GMT', u'Content-Type': u'text/html; charset=utf-8'}

DEBUG:root:RepairClientError: MunkiDownloadError getting Munki client: HTTP result 403: forbidden

Traceback (most recent call last):

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/preflight.py", line 447, in RunPreflight

    flight_common.RepairClient()

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/flight_common.py", line 947, in RepairClient

    'MunkiDownloadError getting Munki client: %s' % str(e))

RepairClientError: MunkiDownloadError getting Munki client: HTTP result 403: forbidden

INFO:root:DEBUG2: Options: {'logging_function': <function display_debug2 at 0x105720d70>, 'additional_headers': {}, 'file': '/Library/Managed Installs/applesus.sucatalog.download', 'cache_data': None, 'url': u'https://myapp.appspot.com/applesus/', 'follow_redirects': False, 'download_only_if_changed': False, 'can_resume': False}

INFO:root:DEBUG1: Status: 403

INFO:root:DEBUG1: Headers: {u'Alternate-Protocol': u'443:quic,p=1', u'Content-Length': u'0', u'Server': u'Google Frontend', u'Cache-Control': u'no-cache', u'Date': u'Mon, 20 Jul 2015 21:52:44 GMT', u'Content-Type': u'text/html; charset=utf-8'}

DEBUG:root:MunkiDownloadError getting Apple SUS catalog.

Traceback (most recent call last):

  File "/usr/local/munki/simian/lib/python2.6/site-packages/simian-2.4-py2.6.egg/simian/mac/client/flight_common.py", line 597, in GetAppleSUSCatalog

    '%s/applesus/' % url, APPLE_SUS_CATALOG)

  File "/usr/local/munki/munkilib/updatecheck.py", line 3478, in getResourceIfChangedAtomically

    verify=verify)

  File "/usr/local/munki/munkilib/fetch.py", line 277, in getResourceIfChangedAtomically

    message=message, resume=resume, follow_redirects=follow_redirects)

  File "/usr/local/munki/munkilib/fetch.py", line 390, in getHTTPfileIfChangedAtomically

    raise GurlDownloadError(err)

GurlDownloadError: HTTP result 403: forbidden

DEBUG:root:Preflight completed successfully.

[Justin McWilliams]

Ok, so preflight is actually successfully authenticating to the server... attempting to repair (and use Munki's gurl to download the repair client) but failing due to auth, then attempting to download the Apple Update catalog (also using Munki's gurl) but also failing due to auth... both of these failures are not "hard", so it completes "successfully", Munki then goes on to do it's thing and fails auth similarly, then postflight later complains there's no auth token in AdditionalHttpHeaders.

I'd wager this is not working, or the file is not readable by Munki afterwards: https://github.com/google/simian/blob/master/src/simian/mac/client/preflight.py#L157

Can you verify if any headers are set in /private/var/root/Library/Preferences/ManagedInstalls.plist after running preflight?

You're definitely running as root or with sudo, right?

- Justin

[Michael Holt]

I am running with Sudo.  I just checked the path you suggested and there is no ManagedInstalls.plist file inside

Michael Holt / IT Administrator
(661) 705-4711/ mh...@reallifechurch.org

Real Life Church Office: (661) 775-7401 
23841 Newhall Ranch Rd
Valencia, CA 91355
http://www.reallifechurch.org

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

You received this message because you are subscribed to a topic in the Google Groups "Simian Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/simian-discuss/PF9pw1-US1k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to simian-discus...@googlegroups.com.

[Justin McWilliams]

Hrmm... this should get created by Munki (see this and the lines immediately below):

   https://github.com/munki/munki/blob/master/code/client/munkilib/munkicommon.py#L85

When Simian calls munkicommon.SecureManagedInstallsPreferences():

  https://github.com/google/simian/blob/master/src/simian/mac/client/preflight.py#L408

I have to run (can try to look deeper tomorrow) but please try on another machine, and confirm that directory (/private/...) doesn't have non-standard permissions...

- Justin

[Michael Holt]

I wasn't able to test today on another machine but will give it a try tomorrow morning when I get to the office

...

[Michael Holt]

I just tried it on a new laptop and ran into the same issue

...

[Michael Holt]

I'm not sure but this could be the issue... not sure what's causing it.  Was looking through the debugging and noticed an error in the json return at the DEBUG2: Options: line

{

'logging_function': <function display_debug2 at 0x10faad140>, 

'additional_headers': {}, 

'file': '/tmp/munki_repair_dmg_k2ibtn/munkiclient.dmg.download',
'cache_data': None,
'url': u'https://myapp.appspot.com/repair', <----- there is a u before the URL

[Justin McWilliams]

The "u" before the URL is because the value is unicode encoded string.

The problem here is still that the ManagedInstalls.plist in /private does not exist.  Perhaps try touching that, or using (as root) defaults to create it?

--

[Michael Holt]

I touched it as sudo and rebooted but am still having the same errors

[Michael Holt]

Just to make sure it was built correctly I rebuilt Simian on a fresh OS X Install to a new appengine instance and built and deployed a new package but am having the same result

...

[Michael Holt]

i'm at a loss, tried on 3 different machines and am not getting anywhere with it

...

[Justin McWilliams]

You could try Munki, standalone/not with Munki, to see if the plist is created.  Then, if so, install Simian.

I don't have any additional ideas offhand, other than trying on a vanilla OS X install, in case this is an issue with your image.

--

[Michael Holt]

It's working now.  Turned out to be one of the puppet configuration settings that I had copied... guess that's what I get for copying someone else's code without digging into what it does hahaha

[Justin McWilliams]

For my own benefit of helping future customers experiencing the same issue, what puppet config was at fault?

[Michael Holt]

I was using Graham Gilbert's Mac_Admin Puppet Configuration (https://github.com/grahamgilbert/puppet-mac_admin)

I'm going to spend some time looking through it and see what it was changing that was causing an issue.  I also hope to write a How-To when i'm all said and done on how to using Simian + Puppet to manage macs

[Michael Holt]

so just a note for anyone that stumbles across this.  If using Graham's mac_admin puppet config avoid running mac_admin::munki and macadmin::munki::munkitools as this is what caused my issues.  I've actually removed all of the files related to these two from my cloned repo

...