Yaml support in Munki

[Gregory Neagle]

Now that Munki 7.1 is almost final (it will likely have an official release this week), I’m moving my attention for future changes to other things. One of these things is this PR:

https://github.com/munki/munki/pull/1261

The intent is to allow Munki admins to write and maintain manifests and pkginfo files in YAML instead of the XML plist format.

This is a large set of changes.

Some reservations I have:

1) It relies on a third-party YAML library: https://github.com/jpsim/yams This seems to be used in many other Swift projects, and seems to be stable and well-supported. Still it is a risk to depend on it.

2) The PR also (optionally) encodes _catalogs_ as YAML. I’m not convinced that’s necessary. The main impetus for YAML support is to improve human-readability of Munki’s repo files. Humans will read and write manifess and pkginfo files; humans generally will not read (and will never write) catalog files. _Not_ supporting YAML catalog files would reduce the scope of changes here a great deal.

3) I’m unlikely to switch to using YAML-format files in the near future. I use MunkiWebAdmin 2, which knows nothing about YAML-formated Munki files, and internal documentation and experience is all around traditional plist files. (I’ve used plists for 25+ years. I’ve never (yet) authored a YAML file from scratch). All this to say, I’m not likely to find and be super motivated to fix issues with YAML support; that will need to be the work of others.

My initial thought is to merge Rod’s PR into a new branch that people can build and test and play with. We can then work on figuring out what might need to happen to get to a Munki 7.2 or 7.3 release containing at least some YAML support.

I would appreciate other’s perspectives on adding YAML support to Munki.

-Greg

[Gregory Neagle]

4) There are a lot of examples in Munki’s wiki and elsewhere using plist format for manifests and pkginfo. Will people expect these all to be duplicated so every example is provided in both plist and YAML format? If so, that’s work for others to complete…

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/munki-dev/CE62C0C6-897A-4F09-B3B4-22EB8C775D6A%40mac.com.

[Per Olofsson]

I find YAML support in Munki interesting, but I have to admit that my immediate reaction is why is this a client change, and not server side tooling? This is likely colored by me spending the past winter migrating from a traditional Linux VM setup to a CI/CD pipeline and a cloud hosted repo.

To get a feel for what a YAML repo would look like I wrote a quick and dirty conversion tool today, gist and thread is on the MacAdmins Slack: https://macadmins.slack.com/archives/C04QVPFGU/p1776687667301289

It only converts manifests and pkgsinfo, as I don't see why catalogs would need to use YAML — as Greg notes they aren't edited by humans. Catalogs are also used by AutoPkg, and it seems like a good idea to decouple them and not require changes.

Right out of the gate it's clear to me that manifests and pkginfo files are a nicer to read and write in YAML, but also that the usual Norway style footguns will be a support burden — notably most version numbers will often need to be quoted. Input validation can quite easily catch these, which is one reason why I would prefer to do the YAML → plist conversion on the server side, e.g. during makecatalogs.

The other reason is that parsing structured data is a notoriously bug prone area, and the Munki client runs with elevated privileges. Since Apple doesn't ship a YAML parser this means adding a third party dependency with all that that entails. If I was the maintainer I would be very cautious here.

So, if YAML support is added to makecatalogs, makepkginfo, munkiimport, and manifestutil we get the benefit of nicer manifests and pkginfo files, but with no changes to the client code and no changes needed to AutoPkg (though you still might want to make it output YAML to not get mixed filetypes in the repo).

--
Per Olofsson, IT-service, University of Gothenburg

[Gregory Neagle]

It would be pretty simple/straightforward to handle YAML pkginfo entirely “server side”: Munki clients don’t actually request pkginfo files at all; they request catalogs. catalogs are constructed from pkginfo files using the `makecatalogs` tool. Admins know to run this tool after any pkginfo edit (and some tools do this automatically). So we could leave catalogs as formatted as plist, and `makecatalogs` could handle both plist and YAML pkginfo files as input, always outputting plist-formatted catalogs.

Manifest files are different here. Munki clients _do_ request these directly. If clients can’t properly read YAML-formatted manifest files, there must then be a new tool/step to translate YAML-formatted manifest files to plist format. This would be a new task that admins would have to remember to do; if they created/edited a manifest file in YAML format and then forgot to run the step that converts it to plist, any clients using the affected manifest would have failures.

That makes it seem that it’s unavoidable to have the client able to read/process YAML-formatted manifests at least, and once you do that…

-Greg

To view this discussion visit https://groups.google.com/d/msgid/munki-dev/DA27437A-8CCF-4DE5-8A87-8B6B634BA498%40gu.se.

[Allister Banks]

My story is similar where I don’t see our environment using YAML at all in the near/short term - how our server operates means we do not touch manifests directly ever, (a handful of catch-alls are defined in Terraform and anything custom is ‘attached’ to computers dynamically via tags,) and all of our example code touching pkginfo files (primarily for specifically tag-based sharding) is either facilitated by copy-pasting in as XML/plist and/or we use MunkiAdmin to validate stuff like escaping or any of the other thing it can help us with.
Autopkg and our ‘toil reducer’ promotion code means we barely care about modifying pkginfo files most days of the year just the same, and currently nobody in my team or organization has decided that YAML is a must-have, I see no benefit to converting that workflow to YAML, neither. I should be sensitive to the folks who equate the probably upwards of 3x line length or more of the XML version as similar to bash being ungainly versus what you could accomplish in other scripting languages, but when the plist format is functionally the same (and YAML requires learning its specific equivalent of escaping/care&handling) I don’t find it a downside to scan/scroll for a bit longer without any other apparent flaw.
All that being said I wouldn’t “yuck another’s yum” as they say, if it brings more folks into the tent or helps cross-team collaboration then I’m all for it.
Allister

[Per Olofsson]

So I guess the question is who here wants YAML manifests but doesn't have a CI/CD pipeline that would automatically do the conversion?

--
Per Olofsson, IT-service, University of Gothenburg

To view this discussion visit https://groups.google.com/d/msgid/munki-dev/7A9FC461-94A4-4EB1-8ABB-789AAF13393D%40mac.com.

[Nixtar]

In the past I’ve considered building a custom backend for munki that can dynamically generate catalogs/manifests based on server side business logic such as what network the request is coming from.

Our environment is quite complex (approx 1400 physical locations) and the desire to be able to cater for site based customisation is there, but not super high on the priorities. While this kind of thing could be done with client identifiers etc, at our scale that would be a lot to manage.

One issue holding me back from attempting this work is that we are a c#/.NET development team and there are no first party plist serialisation libraries and all the third party ones I have evaluated have some limitation that would prevent it from covering some corner case for munki. Off the top of my head, multi line strings for things like scripts comes to mind.

Yaml is interesting to me and it has the potential to solve that issue. However we would need it to be supported everywhere the client reads from the repo.

Regarding supporting both types at the same time.

I haven’t looked too deep at the PR, but does the client need to opt into YAML or is it automatically determined?

I’d prefer an opt in that also sets the http Accept header for requests from plist/xml (if it’s not already being set) to application/yaml then let the backend deal with it.

An admin could have 2 separate repos on your web server with both plist and Yaml and use the header to decide what file to return during a transition period over to 7.1 if you were so inclined.

Another pro for YAML that I haven’t seen mentioned is that we could also develop Yaml schemas for the various munki object types, that way you get autocomplete and validation/warnings in your IDE when editing them.