How to save time on download recipe runs with using "StopProcessingIf"

[Miq Viq]

Hi,

I just found out while I was checking some nice recipes that some download recipes are using "StopProcessingIf" to stop the recipe if there is no new items to download.

Like this:

        <dict>

            <key>Processor</key>

            <string>StopProcessingIf</string>

            <key>Arguments</key>

            <dict>

                <key>predicate</key>

                <string>download_changed == FALSE</string>

            </dict>

        </dict>

As this greatly reduces the amount of time (and bandwidth) used for recipe runs is there any reason why this should not be a highly recommended part of any download-recipe?

-MiqViq

[Timothy Sutton]

Download recipes which use URLDownloader (which is almost if not all of them) already have this kind of functionality.

URLDownloader does not re-download the file if it determines (based on ETag, Last-Modified headers or file size matches) that the file you've already downloaded matches the one you would be about to download on the server.

For example:

➜ ~ autopkg run -v AdobeReader.download
Processing AdobeReader.download...
AdobeReaderURLProvider
AdobeReaderURLProvider: Found URL http://ardownload.adobe.com/pub/adobe/reader/mac/11.x/11.0.10/en_US/AdbeRdr11010_en_US.dmg
URLDownloader
URLDownloader: Item at URL is unchanged.
URLDownloader: Using existing /Users/tsutton/Library/AutoPkg/Cache/com.github.autopkg.download.AdobeReader/downloads/AdobeReader.dmg

In this case the recipe is not _stopped_, it just doesn't re-download the file. If recipes were stopped, that would be bad, because recipes based on the download recipes would also stop.


Tim

> --
> You received this message because you are subscribed to the Google Groups "autopkg-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to autopkg-discu...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gregory Neagle]

Expanding on Tim's reply -- it's not always the right thing to do to Stop if there isn't a new download.

You might not have a new download, but perhaps you don't have a current package built from that download.

Perhaps you have a current package build from that download, but it hasn't been yet imported into your Munki repo.

If the recipe stopped/aborted because there was no new download, other important steps later might not be followed.

-Greg

[Anthony Reimer]

…and I will expand further on Greg & Tim's replies.

I do use StopProcessingIf, but _only_ when I want to make a copy of an already-processed package to move into my DeployStudio repo. If I did not use StopProcessingIf, the Copier processor I use in the recipe would copy the package over to the Packages folder on every single run, even if the parent recipe correctly recognized that there had been no update and did not download anything new. So I use StopProcessingIf to eliminate redundant copying, not redundant downloading.

The alternative would be to write a custom processor for it (e.g., ConditionalCopier). Since I don't know Python, I tried and failed.

Having said this, I always document my use of StopProcessingIf in the recipe Description (and all of its children), thusly:

This recipe suppresses unnecessary copying of the package using the "StopProcessingIf" processor. This means this recipe or any recipe that uses this as a parent recipe could stop execution before reaching the child. You may need to remove the cache for the recipe in question and start again if you manually delete or move the package that this recipe creates.


Anthony Reimer