How practice DRY with Packer

[Nasir Amin]

Hi,

I am using Packer version 1.4.1
What is the best practice to manage generic templates.
We have a few packer template files for different windows versions. The template's builders section is common and we use var files to pass in any user variables.

The reason we are having to keep multiple json template files is due to the Provisioner sections being different for different types of templates.

Question:
Is there a way we can use the same template file with just the builders section.
For provisioners, we can either put them in a separate file and then somehow inject it into the main tempalte OR
if we can pass in the Provisioners in the form of a json array through a user variable from a variables file?

Some of my provisioners are powershell scripts, ansible and windows restart. Is it possible to pass in provisioners from a variable file like this?

  "host_provisioners": [ 
    {
      "type": "powershell",
      "inline": [
                "cmd.exe /c powershell.exe Set-ExecutionPolicy Unrestricted -force",
                "a:\\scripts\\Configure-MinorSettings.ps1",                     
                "a:\\scripts\\win2019-setup-ssh.ps1",           
                "a:\\scripts\\Setup-Docker.ps1"
                ]
    },
    {
      "type": "windows-restart",
      "restart_check_command": "powershell -command \"& {Write-Output 'Guest Restarted Successfully.'}\""
    }
]
}

Then in the main template file where my builders are, I do this:

"provisioners": "{{user host_provisioners }}"

It seems like you have to put provisioners and builders in the same file. It would be good if we can do on one of the following and if it is already possible:

• Isolate builders and provisioners into separate files and then combine them in the packer build command with a flag like --provisioner-file=<path to file>
• OR Allow passing the provisioners from a variables file.

The first approach will probably be more flexible.

What is the best practice on structuring and managing multiple templates or making your templates generic? As things stand, it seems impossible to apply DRY principles to Packer.

I'll really appreciate any guidance on this please as our packer scripts and templates are becoming a bit hard to scale and maintain.

I see packer will eventually support HCL2 which should make some of the things easier e.g. conditional provisioners etc

https://github.com/hashicorp/packer/issues/1768

Regards,

Nas

[Rickard von Essen]

You can use jq to join several json files and thus keep snippets that you join together. See https://stackoverflow.com/questions/19529688/how-to-merge-2-json-file-using-jq

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/ea90623a-2da1-453e-969c-0e7c39a04969%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nasir Amin]

Hi Richard,

Thanks. So that means I'll have to generate files at runtime and then delete them once the packer build has completed?

Regards,

To unsubscribe from this group and stop receiving emails from it, send an email to packe...@googlegroups.com.

[Rickard von Essen]

or you can pipe it like this:

cat template.json | packer build -

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/317528bb-11d0-42f8-ad75-e6fcfa9411e0%40googlegroups.com.

[Nicholas Bayle]

So I'm trying to merge multiple packer json files together using the following json files:

- variables used by all platforms

- platform-specific requirements (usually a builder & sometimes a provisioner and/or post-processor)

- the provisioners used for all platforms

- the post-processors used by all platforms

So, generating the json and running packer to build a VM would look roughly like:

PACKER_LOG=1 jq -s 'reduce .[] as $item ({}; . * $item)' packer-variables.json packer-vm.json packer-provisioners.json packer-post-processors.json | packer build -force -

The json output at first glance looks ok, but on closer inspection I have an issue. If packer-vm.json contains this snippet (among other things):

  "provisioners": [
    {
      "type": "shell",
      "inline": ["echo 'vagrant' | sudo -S -- bash -c 'dd if=/dev/zero of=/EMPTY bs=1M; rm -f /EMPTY; sync'"]
    }
  ],

and packer-provsioners.json contains:

{
  "provisioners": [
    {
      "type": "shell",
      "inline": ["echo 'vagrant' | sudo -S bash -c 'echo { \\\"platform_type\\\": \\\"{{user `platform`}}\\\" } > /.platform_info' && echo 'vagrant' | sudo -S chmod 666 /.platform_info"]
    },
    {
      "type": "chef-solo",
      "cookbook_paths": ["cookbooks"],
      "run_list": ["things"],
      "version": "14.13",
      "chef_environment": "{{user `chef_env_type`}}",
      "environments_path": "./environments"
    },
    {
      "type": "shell",
      "inline": ["echo 'vagrant' | sudo -S apt-get --yes --purge autoremove chef"]
    },
    {
      "type": "shell",
      "inline": ["dpkg-query -f '${Package} ${Version}\n' -W > /tmp/manifest; sleep 2"]
    },
    {
      "type": "file",
      "source": "/tmp/manifest",
      "destination": "manifest",
      "direction": "download"
    },
    {
      "type": "shell",
      "inline": ["rm /tmp/manifest"]
    }
  ]
}

Then jq will just wipe out the provisioners from packer-vm.json (i.e. no dd command in this example). Does anyone know the magic jq sauce to preserve that during a merge?

Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CALz9Rt8sZk%2BEQGpFqmaWk_4Tc97n%2BBwwY_EvZ9Yu4LdXBaGvnw%40mail.gmail.com.

[Nicholas Bayle]

So everybody just does copy/paste and manages multiple files by hand? A nightmare indeed.

[Nicholas Bayle]

For posterity, here is how you do it:

jq -s '[.[] | to_entries] | flatten | reduce .[] as $dot ({}; .[$dot.key] += $dot.value)' file1.json file2.json fileN.json | packer build -

[Rickard von Essen]

If you are on that scale python or ruby would do json manipulation in all imaginary ways.

But frankly nowadays everyone uses Docker and the only need to build one base image to run it on (or none). ;-) 

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/CAGzi4%3D66Tn-KO6P3Tdj6qnV0JNj6Q-e4VDwcH583q8MKQL%3Dksg%40mail.gmail.com.