Parallel post-processing in Packer ???

[John C]

Currently I'm using Packer to build a VMWare template and then have configured the post-processor section of the JSON with multiple vsphere tasks, to copy the template to multiple remote datacenters once built, using OVFTOOL.   The copying itself works just fine, but it runs serially ... and because the template I'm copying is large, it takes a very long time to finish the process as Packer is waiting for each OVFTOOL copy to complete before starting the next.  

So my question is ... does Packer have the ability to start all of the OVFTOOL copies in parallel?  I've been looking around the HashiCorp website, the Internet, and this Google group, but so far haven't found anything.  And if not, what alternatives do I have to make this work?  Thanks.

[Rickard von Essen]

What's the structure of your post-processor section in your packer template? Is it a sequence? 

On Oct 25, 2017 18:20, "John C" <johnn...@gmail.com> wrote:

Currently I'm using Packer to build a VMWare template and then have configured the post-processor section of the JSON with multiple vsphere tasks, to copy the template to multiple remote datacenters once built, using OVFTOOL.   The copying itself works just fine, but it runs serially ... and because the template I'm copying is large, it takes a very long time to finish the process as Packer is waiting for each OVFTOOL copy to complete before starting the next.  

So my question is ... does Packer have the ability to start all of the OVFTOOL copies in parallel?  I've been looking around the HashiCorp website, the Internet, and this Google group, but so far haven't found anything.  And if not, what alternatives do I have to make this work?  Thanks.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/4693c230-517b-4284-a146-e48cb11758d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John C]

-- Basically I'm using the vsphere post-processor to upload the template to a remote destination, and then using a shell-local post-processor that converts the VM to a template on the remote destination

-- I'm doing this copy/convert to 4 different locations at this time

-- They all run successfully, but serially such that it waits until the task is done before moving on to the next one ... and depending on the remote destination, it can take hours to complete

-- The documentation talks about a 'sequence definition' that puts an additional set of brackets around tasks in the post-processors section, but doesn't indicate if that means parallel or not.  Also a previous post noted a user using these additional brackets to attempt running tasks in parallel, but it still didn't work that way ... and the reply was to remove them, which I'm doing and that doesn't seem to work either.

Code below (cleaned up for easier reading / security reasons) ...

====================================================

 "post-processors": [
    {
      "type": "vsphere",   # Copy to destination 1
       ...
    },
    {
      "type": "shell-local",   #Make VM a template in destination 1
      "inline": "bundle exec rake mark_as_template",
      "environment_vars": [
         ...
      ]
    },
    {
      "type": "vsphere",   # Copy to destination 2
       ...
    },
    {
      "type": "shell-local",   #Make VM a template in destination 2
      "inline": "bundle exec rake mark_as_template",
      "environment_vars": [
         ...
      ]
    },
    {
      "type": "vsphere",   # Copy to destination 3
       ...
    },
    {
      "type": "shell-local",   #Make VM a template in destination 3
      "inline": "bundle exec rake mark_as_template",
      "environment_vars": [
         ...
      ]
    },
    {
      "type": "vsphere",   # Copy to destination 4
       ...
    },
    {
      "type": "shell-local",   #Make VM a template in destination 4
      "inline": "bundle exec rake mark_as_template",
      "environment_vars": [
         ...
      ]
    }
]

 

[Alvaro Miranda Aguilera]

As is today the builders can be run in parallel, but provisioner and post-provisioner runs in sequence.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/d9865e5b-9381-411c-a1f6-db57557ba39e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alvaro