Proposal for how Puppet should directly utilize with Munki

[Allister Banks]

Hey long-timers and lurkers,

I've been working on incorporating Puppet into planned infrastructure,
with the major benefit that it eases the process of leveraging Simian
as Munki's server. Since Puppet currently lacks as good a mechanism
for dealing with packages as Munki provides, the folks in ##osx-server
and I have been debating how best approach making a Munki 'package
provider' (mechanism to leverage when installing packages) for Puppet.
I have mocked things up in bash for Puppet to work with Munki's
existing functionality, but Munki could be made more flexible to
enable the use case and workflow that Puppet would be safest
performing. Out of the proposal I've written to present the Puppet
folks with, here are the parts describing the 'purpose' of doing all
this, and the specific 'implementation' I'm envisioning:


Purpose: When Mac workstations are being prepared or repurposed, and
ongoing during maintenance periods for servers, Munki should be relied
upon as Puppets package provider. While Munki already offers packages
to clients and enables some level of enforcement, this will allow
Puppet to have a better method of deploying/ensuring/deactivating
services, and ensure Puppet is aware of the most critical packages
when in simulation mode.

Implementation: Puppet would maintain two files:
ManagedInstalls.plist(Munki's primary config file), and a
'LocalMunkiManifest' file that Munki would operate with. The admin
would specify the name of the package, and Puppet would ensure that
setting is present in the LocalMunkiManifest file and ask Munki to
operate with that configuration. When Puppet is restricted from
causing changes on the system it's managing, Munki should still be
allowed to operate with its GUI client, and this facet would
effectively be dormant unless deemed necessary(i.e. if orchestration
is requested via MCollective calling Puppet).


If people could give feedback however they see fit (to this list, me
directly, or on Twitter @sacrilicious) I'd greatly appreciate it.
Thanks, Allister

[Gary Larizza]

Hey There,

Great to hear about Puppet/Munki integration again!  Let me comment in-line...

So it looks like the solution is to create a local manifest, pipe that to Munki from the command line, let it run, and then scrape the exit status - correct?  I've not used Munki in quite some time, so please excuse any issues about its operation.  Having said that, do you still get the opportunity to select a 'forced install' versus an install that doesn't occur if an application is running?  Or would these all be Forced installs?  Would this batch a number of package installations all at once, or create a local manifest for EACH package in a manifest (Say you wanted to install Acrobat, Office, and a Safari plugin - does it create a manifest for Munki with ALL packages in it, or one for each and pipe it through Munki via the command line)?  If it batches it, is there a way to tell which packages succeeded and which failed?  

Those are a couple of the first questions off the top of my head.  I really like the idea and really want to see this happen, so I'm definitely willing to help as I'm able!

[Greg Neagle]

Responses somewhere down below...

-Greg

My understanding of the proposed functionality might make a Puppeteer's head explode, but I think it's the best approach for an initial take at integrating Puppet and Munki.

Here's my understanding:

When Puppet is asked to ensure a package is present, it adds the package to the managed_installs section of the local manifest. (and makes sure the same package is not in the unmanaged_installs)

When Puppet is asked to ensure a package is absent, it adds the package to the unmanaged_installs, removing it from managed_installs if needed.

That's it. That is _all_.

Puppet does not run managedsoftwareupdate or check its return status. It trusts Munki to do its job. Puppet manages Munki, and trusts Munki to do its part, not micromanaging it.

This is the only way currently to integrate Puppet with Munki without stomping all over the interface niceities of Munki - things like notifying users of updates, not installing updates that require a logout or restart without the user's consent, and not updating applications the user is currently using.

One refinement of this might be to have Puppet run managedsoftwareupdate --auto if Puppet has made changes to the local manifest. In this case, if the machine is at the login window, the installs and removals will happen right away. If there is a user logged in, they _may_ get a notification of available updates, depending on notification settings and how long it's been since the last notification.

Now, experienced Puppeteers might be thinking "this isn't like existing package providers for Puppet at all!" and they'd be right. But Munki is not a package manager, so it can't be integrated like one.

I think this proposal is a good first step at integrating Puppet and Munki. More might be done in the future as we figure out how best to reconcile the very different approaches of Puppet and Munki.

[Gary Larizza]

<poof>

 

Puppet does not run managedsoftwareupdate or check its return status. It trusts Munki to do its job. Puppet manages Munki, and trusts Munki to do its part, not micromanaging it.

This is the only way currently to integrate Puppet with Munki without stomping all over the interface niceities of Munki - things like notifying users of updates, not installing updates that require a logout or restart without the user's consent, and not updating applications the user is currently using.

One refinement of this might be to have Puppet run managedsoftwareupdate --auto if Puppet has made changes to the local manifest. In this case, if the machine is at the login window, the installs and removals will happen right away. If there is a user logged in, they _may_ get a notification of available updates, depending on notification settings and how long it's been since the last notification.

Now, experienced Puppeteers might be thinking "this isn't like existing package providers for Puppet at all!" and they'd be right. But Munki is not a package manager, so it can't be integrated like one.

I think this proposal is a good first step at integrating Puppet and Munki. More might be done in the future as we figure out how best to reconcile the very different approaches of Puppet and Munki.

The main difference that Greg's referring to is that Puppet's job is to ensure an end-state.  If you tell it to install a package, it will check to see if it exists, install it (if it doesn't exist), and check AGAIN after installation to ensure that it has actually been installed.  If the final check fails, then it reports an error and it will try again at the next run (And all resources that DEPEND on that package will NOT get executed on this run).

To create a package provider in Puppet that is Ensurable (i.e. that can ensure a package absent and present), you need at least three methods: create(), exists?(), and destroy().  These methods will be used, respectively, to install the package, check for its presence on the system, and uninstall the package.  A package provider doesn't need to be ensurable (i.e. it can have a method to INSTALL but not to UNINSTALL - this is what the current pkgdmg provider does).  Basically, if you implement what Greg recommends then you would only be checking the local manifest shipped to Munki and not the status of the package itself.  Is it in the manifest? Then the exists?() method will return true.  And vice-versa.

I agree that this would be a good starting point, but there's also room to wreck Munki's interface niceities too (and treat it like a bastard-package-manager-stepchild)  ;)

[Greg Neagle]

On Mar 29, 2012, at 11:05 AM, Gary Larizza wrote:

I think this proposal is a good first step at integrating Puppet and Munki. More might be done in the future as we figure out how best to reconcile the very different approaches of Puppet and Munki.

The main difference that Greg's referring to is that Puppet's job is to ensure an end-state.  If you tell it to install a package, it will check to see if it exists, install it (if it doesn't exist), and check AGAIN after installation to ensure that it has actually been installed.  If the final check fails, then it reports an error and it will try again at the next run (And all resources that DEPEND on that package will NOT get executed on this run).

To create a package provider in Puppet that is Ensurable (i.e. that can ensure a package absent and present), you need at least three methods: create(), exists?(), and destroy().  These methods will be used, respectively, to install the package, check for its presence on the system, and uninstall the package.  A package provider doesn't need to be ensurable (i.e. it can have a method to INSTALL but not to UNINSTALL - this is what the current pkgdmg provider does).  Basically, if you implement what Greg recommends then you would only be checking the local manifest shipped to Munki and not the status of the package itself.  Is it in the manifest? Then the exists?() method will return true.  And vice-versa.

Exactly. Puppet is managing Munki, not packages. That's Munki's job.  This would be a Munki provider for Puppet, not a package provider that happens to use Munki as a back-end.

I know that's not your end goal, but you have to start with something.

I agree that this would be a good starting point, but there's also room to wreck Munki's interface niceities too (and treat it like a bastard-package-manager-stepchild)  ;)

But that will require someone that's not me to actually write code that essentially replaces /usr/bin/munki/managedsoftwareupdate with its own logic.

-Greg

[Greg Neagle]

I lied.

Here's some code to get someone started writing a better provider for Puppet that can use Munki's infrastructure. Tested with the current Git revision:

#!/usr/bin/python
# encoding: utf-8
#
# Copyright 2009-2011 Greg Neagle.
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
mean_munki
"""

import sys
import subprocess
from munkilib import updatecheck
from munkilib import installer

localmanifestpath = None
if len(sys.argv) == 2:
localmanifestpath = sys.argv[1]

updatecheckresult = updatecheck.check(localmanifestpath=localmanifestpath)
if updatecheckresult == 1:
need_to_restart = installer.run()
if need_to_restart:
unused_retcode = subprocess.call(['/sbin/shutdown', '-r', 'now'])

[Raul Cuza]

I wanted to interject that leveraging munki as a package installation
tool on OS X is something any configuration management (cfgtmgt)
system running on OS X would find useful.

I agree that munki is not a package manager which requires processes
for installing, upgrading, configuring, and removing software packages
[ http://en.wikipedia.org/wiki/Package_management_system ]. Munki does
do three out of the four however very well. One could choose to inject
configuration of the software as packages, which means they do not
need any other tool. I find this way of doing configuration too
cumbersome because it means even simple configs have to be put through
a packaging process before being deployed. Being able to change a
single file on hundreds of Macs by changing one file in a version
control system and pushing a button is a much more desirable work
flow.

I politely disagree with Greg N. that adding a package to
managed_installs or unmanaged_installs is the limit of what munki can
do for a cfgmgt system. By defining the receipts needed to install the
vast universe of install "packages" that munki can handle, it makes it
possible use tools instead of mouse clicks to manage software on a
Mac. The MSU.app is just a nice bonus compared to this inovation
(THANK YOU! THANK YOU!).

A replacement for managedsoftwareupdate could be written that let's
the cfgmgt tool issue the command to install a package and leverage
the pkgsinfo files to do the install and verification. The cfgmgt tool
could also use munki methods and the pkgsinfo files to get the status
of an install. And finally, the cfgmgt tool could use munki methods to
uninstall a package. This senario ignores the user of the machine.

A local manifest would be a way of making similar changes but not
ignoring the user. In this case verification would have to be a
function of time as installation would not necessarily happen
immediately. The verification method would need to know when the local
manifest changed, when the last munki run occurred, and other cases
that would allow it to decide how to report the state of the managed
machine (is it in compliance or not?).

<joke>I'm just wondering when will Apple make a hostel take over of
Munki and some cfgtmgt tool and incorporate it in to an Enterprise App
Store.</joke>

Raúl

[Greg Neagle]

On Mar 29, 2012, at 11:42 AM, Raul Cuza wrote:

I politely disagree with Greg N. that adding a package to
managed_installs or unmanaged_installs is the limit of what munki can
do for a cfgmgt system. By defining the receipts needed to install the
vast universe of install "packages" that munki can handle, it makes it
possible use tools instead of mouse clicks to manage software on a
Mac. The MSU.app is just a nice bonus compared to this inovation
(THANK YOU! THANK YOU!).

A replacement for managedsoftwareupdate could be written

So write it!

[daygl...@gmail.com]

Not trying to rain on the parade, but I am looking for some clarification.

On 2012-03-29, at 6:49 AM, Allister Banks wrote:
> Purpose: When Mac workstations are being prepared or repurposed, and
> ongoing during maintenance periods for servers, Munki should be relied
> upon as Puppets package provider. While Munki already offers packages
> to clients and enables some level of enforcement, this will allow
> Puppet to have a better method of deploying/ensuring/deactivating
> services, and ensure Puppet is aware of the most critical packages
> when in simulation mode.

I am getting ready to deploy Puppet for our Macs, but we have already been running Munki for a long time.

After reading this, I am still not clear what the advantage of a munki provider would be.

> Implementation: Puppet would maintain two files:
> ManagedInstalls.plist(Munki's primary config file)

Something like this is what Puppet is designed to do. Other than the fact ManagedInstall.plist will be a binary plist,
you should have no difficulties managing this file with Puppet using a number of techniques...

- wholesale (using the File type)
- by key (using Exec type)
- by mcx (using Mcx type)
- some combo of these

> and a 'LocalMunkiManifest' file that Munki would operate with.

So, the Puppet provider would manage this file?

How is this different from simply distributing said manifest as file using Puppet?

> The admin would specify the name of the package, and Puppet would ensure that
> setting is present in the LocalMunkiManifest file and ask Munki to
> operate with that configuration. When Puppet is restricted from
> causing changes on the system it's managing, Munki should still be
> allowed to operate with its GUI client, and this facet would
> effectively be dormant unless deemed necessary(i.e. if orchestration
> is requested via MCollective calling Puppet).

So, now you have two places where you can model package deployment?

Perhaps I misunderstand?

--
B.

[Allister Banks]

EXACTLY the type of folks I was hoping would chime in!

On Mar 29, 2:55 pm, daygloje...@gmail.com wrote:
> Not trying to rain on the parade, but I am looking for some clarification.
>

> Implementation: Puppet would maintain two files:
> > ManagedInstalls.plist(Munki's primary config file)
>
> Something like this is what Puppet is designed to do. Other than the fact ManagedInstall.plist will be a binary plist,
> you should have no difficulties managing this file with Puppet using a number of techniques...
>
> - wholesale (using the File type)
> - by key (using Exec type)
> - by mcx (using Mcx type)
> - some combo of these

PThe servers contents are not an apt/yum repo, and therefore Puppet
needs to affect some change, locally, and that change is binary; if
ManifestURL is present, Munki's normal operation model cannot continue
as we're used to. Preflight and postflight, as Simian uses, is of
course an option, but one way or the other, the ManagedInstalls.plist
currently needs to be altered for the Puppet operation I'm
envisioning.

>
> > and a 'LocalMunkiManifest' file that Munki would operate with.
>
> So, the Puppet provider would manage this file?

> So, now you have two places where you can model package deployment?

Puppet-savvy folk know this, of course, but Puppet strives to excel at
delivering 'the trinity' - package, config(file), and service. Munki
can kinda sorta 'enforce' anything that works in a package-based use
case, but resorts to techniques like scripts inserted into rebuilt
packages or pkginfo files - which disperses the documentation of the
IT infrastructure. And, more importantly, Puppet can't deliver the
trinity or guarantee much of the package part of a node's state
without a good amount of overheard, which Munki was purpose-built to
address.

>
> Perhaps I misunderstand?
>
> --
> B.

I'm saying let them both play to their strengths, while allowing
Puppet to rule the roost in certain cases: Anything compliance-
related, or worth documenting in Puppet code for disaster recovery on
Macs providing services, or just to ease in workstation preparation or
repurposing.

Allister

[Greg Neagle]

This is a proof-of-concept only:

./munki_do.py --catalog production --install Firefox
Checking for available updates...
Retreiving catalog "production"...
0..20..40..60..80..100
Downloading Firefox 10.0.3esr.dmg...
0..20..40..60..80..100
Verifying package integrity...
Downloading DA_FirefoxConfig-6.0.pkg.dmg...
0..20..40..60..80..100
Verifying package integrity...

The following items will be installed or upgraded:
+ Firefox-10.0.3
Web browser from Mozilla
+ DA_FirefoxConfig-6.0
Configures Firefox for use at Disney Animation. Installs Timmy search plugin.
Installing Mozilla Firefox (1 of 2)...
Mounting disk image Firefox 10.0.3esr.dmg...
Copying Firefox.app to /Applications...
The software was successfully installed.
Running postinstall_script for Firefox...
Installing Disney Animation Firefox Configuration (2 of 2)...
Mounting disk image DA_FirefoxConfig-6.0.pkg.dmg...
Preparing for installation…
Preparing the disk…
Preparing Disney Animation Firefox Configuration…
Waiting for other installations to complete…
Configuring the installation…
97.75 percent complete...
Finishing the Installation…
100.0 percent complete...
The software was successfully installed.

-------------------

#!/usr/bin/python
# encoding: utf-8
#
# Copyright 2009-2011 Greg Neagle.
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""

munki_do
"""
import optparse
import subprocess
import sys

from munkilib import FoundationPlist

from munkilib import updatecheck
from munkilib import installer

p = optparse.OptionParser()
p.add_option('--catalog', '-c', action="append",
help='Which catalog to consult. May be specified multiple times.')
p.add_option('--install', '-i', action="append",
help='An item to install. May be specified multiple times.')
p.add_option('--uninstall', '-u', action="append",
help='An item to uninstall. May be specified multiple times.')

options, arguments = p.parse_args()
manifest = {}
manifest['catalogs'] = options.catalog or ['production']
manifest['managed_installs'] = options.install or []
manifest['managed_uninstalls'] = options.uninstall or []
FoundationPlist.writePlist(manifest, '/tmp/localmanifest.plist')

updatecheckresult = updatecheck.check(
localmanifestpath='/tmp/localmanifest.plist')

if updatecheckresult == 1:
need_to_restart = installer.run()
if need_to_restart:

print "Please restart immediately!"

[Greg Neagle]

# ./munki_do.py --checkstate Firefox --checkstate FinalCutPro7
Firefox: installed
FinalCutPro7: not installed

...and with that, I think I've demonstrated all the essential functionality for a Munki Puppet package provider. Puppeteers: go forth and do good works.

-Greg

--------------------

#!/usr/bin/python
# encoding: utf-8
#

# Copyright 2009-2012 Greg Neagle.

#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
munki_do
"""
import optparse
import subprocess
import sys

from munkilib import FoundationPlist
from munkilib import updatecheck
from munkilib import installer

from munkilib import munkicommon

p = optparse.OptionParser()
p.add_option('--catalog', '-c', action="append",
help='Which catalog to consult. May be specified multiple times.')
p.add_option('--install', '-i', action="append",
help='An item to install. May be specified multiple times.')
p.add_option('--uninstall', '-u', action="append",
help='An item to uninstall. May be specified multiple times.')

p.add_option('--checkstate', action="append",
help='Check the state of an item. May be specified multiple times.')

options, arguments = p.parse_args()
cataloglist = options.catalog or ['production']

if options.checkstate:
updatecheck.MACHINE = munkicommon.getMachineFacts()
updatecheck.CONDITIONS = munkicommon.getConditions()
updatecheck.getCatalogs(cataloglist)
for check_item in options.checkstate:
installed_state = 'unknown'
item_pl = updatecheck.getItemDetail(check_item, cataloglist)
if item_pl:
if updatecheck.installedState(item_pl):
installed_state = "installed"
else:
installed_state = "not installed"
print "%s: %s" % (check_item, installed_state)

if not options.install and not options.uninstall:
exit()

manifest = {}
manifest['catalogs'] = cataloglist

[Gary Larizza]

Great!

Now I'll start you with a Puppet provider:

Puppet::Type.type(:package).provide(:munki) do

  desc "Manages Munki"

  # The magic goes here

end

There!  You just fill in the middle part!

Seriously, thanks for drafting that script!  I'l take that back and start poking at it.  I've got a week in Portland coming up, so maybe if I get left alone long enough I can push out something that will actually work ;)

[Greg Neagle]

A difference between my code and yours:

Mine actually does something. 

Sent from my iPhone

[Gary Larizza]

Also, might want to do this at line 49 (since Puppet will need to check the exit code for --checkstate to work properly)

if item_pl:

           if updatecheck.installedState(item_pl):

               installed_state = "installed"

               exit_code = 0

           else:

               installed_state = "not installed"

               exit_code = 1

       print "%s: %s" % (check_item, installed_state)

       exit(exit_code)

On Thu, Mar 29, 2012 at 5:44 PM, Greg Neagle <gregn...@mac.com> wrote:

[Greg Neagle]

'twas a proof of concept with demo-quality code. I have no idea what Puppet expects or is looking for. Someone more familiar with Puppet can take my example and go from there. 

Sent from my iPhone

[Gary Larizza]

On Thu, Mar 29, 2012 at 8:20 PM, Greg Neagle <gregn...@mac.com> wrote:

'twas a proof of concept with demo-quality code. I have no idea what Puppet expects or is looking for. Someone more familiar with Puppet can take my example and go from there. 

I follow.  Do you mind if I push this up somewhere so someone else (probably internally) can have a look and poke at it?  I know you're not keen on it falling onto a machine, so I thought I'd ask first.

I do actually have it 'interacting' with Puppet, but would obviously like to do a bit more with it until it's fleshed out.

--
Gary Larizza

Professional Services Engineer

Puppet Labs

悟

[Greg Neagle]

I don't mind. 

Sent from my iPhone

[Allister Banks]

Thank you both for coming together to break ground on this! I know I
have a ways to go to sell folks on the use case, but I truly believe
the payoff will be more overlap between two awesome communities.

Allister

[nate]

i'm torn on this one. while munki does a better job of dealing with pkg installs/uninstalls, etc., writing a puppet provider that just drives munki feels a little clunky.

using puppet just to manage a munki manifest seems like a waste. you can do that now without writing a separate provider by using a template or file resource.

would replicating in ruby munki's pkg handling behavior bee too much trouble than it's worth? i don't want to reinvent the wheel, but it seems like working on a more functional pkg provider for puppet would be the way to go.

[Greg Neagle]

I think my demo code for "munki_do" addresses most of the "clunkiness" issues. (And munki_do could be useful outside of Puppet integration, though it needs some serious fleshing out.)

-Greg

[Gary Larizza]

In the vein of  "Code we threw together to make things work", I've pushed a couple of things to Github here --> https://github.com/glarizza/puppet-munki

Now look - this is NOT how I want it to be.  I created a new custom type called 'munki_package' because I didn't want to modify the existing 'package' type in a module.  This is just a proof of concept that will 'work' if you have an existing Munki setup.  You'll need to install munki_do.py into /usr/local/munki/ and will need to have /Library/Preferences/ManagedInstalls pointing at a Munki server.

Like I said - THIS. IS. NOT. IDEAL.  But it works for me.  Right now.  All-sorts-of-patches accepted.  Refactors wanted.  Your mileage may vary.  A lot.

[sebbo]

Before putting in my 2 cents, I should say that we've been using Munki for a long time, we've been exploring Puppet and will be using it soon.

I for one embrace the distinction between the two systems. This is my intended use:

• Puppet: state/configuration management (including Munki config), admin essentials. Static and enforced stuff.
  
• Munki: package management. This is a much more dynamic service for us, making use of features such as optional_installs, blocking_applications, unattended_install, force_install_after_date, etc..
  

Puppet and Munki obviously overlap somewhat for packages but the paradigms are different enough not to mesh together cleanly.

As I said, we don't have Puppet in production yet, so I could be missing something or change my position later.

Thanks,

Seb.