Packer+Terraform best practices?

[Max Englander]

Hello,

I saw that the last question related to Packer was in August 2014.

The poster had an example of running Packer+Terraform where the two tools are invoked separately.

While I recognize that it is good and sometimes necessary to combine tools with wrappers, I feel that it *ought* to be possible to have Terraform build images with Packer, in keeping with the "infrastructure with the push of a button" spirit of Terraform.

So my questions are:

• Is there any philosophical argument for *not* using Terraform to build images with Packer, and to instead coordinate the two with a separate tool?
• Are the best practices for using Terraform+Packer?
• Are there any integrations that have escaped my Googling, or that are planned between Terraform and Packer?

The only way I can currently think to get Terraform to invoke Packer is to define a template provider, and use a local-exec provisioner to invoke Packer within it. This is not really the right tool for the job (a "shell" or "packer" provider would be better), and is clunky for a couple of reasons. 

For instance, I want both Packer and Terraform to share the names of the images that Packer is producing. Packer provides `datetime` and `uuid`, and is therefore better suited to produce the names. To get the image names back to Terraform, however, I need to - in the Terraform template provider - awk the output and cat it into a file.

Max

[Panagiotis Moustafellos]

A packer provider sounds like a great addition to Terraform.

> --
> You received this message because you are subscribed to the Google Groups
> "Terraform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to terraform-too...@googlegroups.com.
> To post to this group, send email to terrafo...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/terraform-tool/24837bcf-7d1b-4933-9a5c-a7018240b4b9%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Panagiotis Moustafellos

[Luis Rojas]

Any news on this?

[peter.k...@pressly.com]

I'm curious too about this.. it would be nice to have a simple way to provision new systems and connect them (packet+terraform+consul) .. btw.. I am assuming Atlas uses all of these under-the-hood with a new web UI in front..?

[Dave Cunningham]

For the sake of discussion, I have done this in 2 different ways:

The first way was a simple wrapper like you described (actually a makefile) and is documented here:  (This example is GCE but other providers will be the same) http://google.github.io/jsonnet/doc/casestudy_fractal.1.html

The second (and far superior) way was to build a more sophisticated wrapper that integrated packer into Terraform instance & disk resources, but this I have only implemented for Google cloud.  This allows automatic management of images, cacheing and rebuilding only when necessary, automatically deleting stale images, etc.  This is not written up but is described on 2 slides starting here http://www.slideshare.net/davidcunningham739/easy-deployment-management-of-cloud-apps/15

I have not yet thought about how to do this with AWS but it's something i need to support.

A packer provider would sit somewhere between these two in terms of usabilty -- it would mean you don't have to run packer yourself with an external wrapper, but you would still have to manage a separate image resource instead of that being done automatically using the instance / disk configs.

On Thu, Aug 20, 2015 at 10:16 PM, <peter.k...@pressly.com> wrote:

I'm curious too about this.. it would be nice to have a simple way to provision new systems and connect them (packet+terraform+consul) .. btw.. I am assuming Atlas uses all of these under-the-hood with a new web UI in front..?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---

You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/9be85109-e414-407f-90e7-c9d9a64e096d%40googlegroups.com.

[Martin Atkins]

I wrote this up as an issue on github a while back:
https://github.com/hashicorp/terraform/issues/1586

Since then I prototyped a few things and my opinion evolved to favoring putting a new UI on the Terraform core to make a packer alternative that integrates better with Terraform:
https://github.com/hashicorp/terraform/issues/2789

I put some more exposition on my blog:
http://apparently.me.uk/padstone-terraform-for-deployment/

I am curious as to how many others are interested in such a solution. If it seems interesting I'd love to collaborate with others to turn it from a prototype into a real project. The bulk of the work here is implementing Terraform resources and provisioners that are more suited to builds, like AMI resources and a chef-solo provisioner. I already started some bits of this in PRs on the Terraform repo.

I am also looking at using this to better handle rollback by having Padstone spin up a fresh set of resources for a new version, then use normal Terraform to pivot permanent infrastructure over to the new resources and shut off the old version stuff only once the deploy is confirmed good. Not tried it in practice yet but it seems plausible in theory.

Let me know what you think! Still experimenting right now so very open to ideas from others.

[Jack Pearkes]

This is a great discussion, some really interesting ideas here and things I've heard talked about internally at HashiCorp. It seems like there are some different motivations for some of these features, but I wanted to specifically mention what part of these problems Atlas solves as it was talked about in passing earlier. (I'm in charge of Atlas, but have worked on all of our tools in some way). Don't want to hijack or detract from the larger discussion, though.

To put it simply and in a Terraform context, Atlas automates the build phase (with Packer) and stores artifacts so Terraform can reference them for deploy as they are changed. Because Atlas has this registry of artifacts and the ability to execute both Terraform and Packer, it essentially becomes a UI on top of the Packer build to deploy pipeline, and lets us pull switches for operators in a low-effort way.

Atlas does a lot, including integration with our other tools like Consul. There is a familiar use cases style section for the curious: https://atlas.hashicorp.com/help/intro/use-cases

Hope that clarifies how we're trying to solve a few of these problems with Atlas. There's so much exciting stuff to do across all of these tools...way too fun to work on it. XD

Best,

Jack

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---

You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/2767f8bb-2b7b-4716-b2bb-210596be3765%40googlegroups.com.

[Joe Barnes]

I am certainly interested in the Terraform+Packer integration story.  At the moment I'm tinkering with a project where the AMI needs the DB URL injected in it.  Hence I need to define the DB in terraform, get the IP/DNS name as input to packer, build the AMI, then roll out some EC2s.  For now I will try the wrapper approach mentioned earlier in this thread.  

Thanks,

Joe

[Matt Black]

@Joe an alternative approach to this problem is rather than baking the AMI with the DB hostname hard coded inside - why not build you application so it dynamically finds the DB on startup? This creates a much more flexible architecture.

Consul specifically has a k/v store for doing this kind of thing:

https://www.consul.io/docs/agent/http/kv.html

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/491fd8c8-6b21-4966-a6a6-c3ac7d7fa149%40googlegroups.com.

[Joe Barnes]

@Matt: For this particular project, it would be overkill to stand up other servers.  I just need something that stands up our Lift application server in a cluster.

But since you brought it up, in general I've always favored fully-baked AMIs.  When a scaling groups summons a new EC2 instance, I like knowing we've minimized the number of things that can fail for it to start.  At work we currently use a combination of fully-baked AMIs and Elastic Beanstalk for our earlier work.  We've made enough configuration mistakes in the past that I've grown to prefer the rigidity of fully-baked AMIs.

Joe

You received this message because you are subscribed to a topic in the Google Groups "Terraform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/terraform-tool/ij9rEBqByX0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEuGOZxwtafYA4Hss2rUYxN93GYuqhrvoAs%2BaVOO3raL%3D5XTTQ%40mail.gmail.com.

[Dave Cunningham]

Why don't you do that last "icing on the cake" configuration in userdata / startup-script, so your image is more versatile.  As long as you have no external dependencies in your startup script this should present any more risk.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAOqnDhso6pYQ506yiCPHBb3rCK0388e%3D3%2BzdpmqnLP-S840q%3Dg%40mail.gmail.com.

[Joe Barnes]

Hey now that's a cool idea I've not thought of! I'll give it a shot. Thanks!

Joe

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTgExdZaDPjRyx_CMZYXV62z0VdVkgXv6cv3vR%3DdbqZDHw%40mail.gmail.com.

[Joe Barnes]

So I've tried out the approach that Max originally proposed, and it doesn't seem to work.  That is, I made a template_file with a local_exec that invokes packer.  Unfortunately, it appears that terraform reads the file before the local_exec is run, which seems reasonable considering all other resources are designed such that provisioners are run after the resource is created.

Has anyone gotten something like this to work?  I looked at the Make file idea earlier in this thread, but I'm not particularly familiar with Make coming from the JVM world.  I'm really interested in running packer within terraform because terraform produces several inputs (the VPC, subnet, and security group) for running packer, and of course packer produces an AMI for running the instances.

For anyone interested, click here for the project.

Joe

[Max Englander]

Hey Joe, I have a trick to suggest. Just so we both have the same point of reference, I am looking at this line in your project.

For your "packer" template_file, you can make the "filename" key "/dev/null" (assuming you're running this from some *nix variant). Have "bake.sh" write to "ami.tpl" just as you are doing, but inside of your Terraform config, read this in with "${file(...)}". This should ensure that "ami.tpl" won't be evaluated until "bake.sh" is run. Note that you will have to make your "packer" template_file an explicit dependency in your "aws_launch_configuration".

There are almost certainly some weird pitfalls to this approach, but it's been a little while since I messed around with this kind of thing so I unfortunately can't recall what they are.

I have an example of how to do this here, in case it's helpful.

Good luck!

[Joe Barnes]

Thanks a million, Max!  I was able to get my project rolling.  I tried to closely mimic your work, so I have two template providers.  It might be overkill, so I plan to revisit later to see if it can be simplified.

Joe

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to a topic in the Google Groups "Terraform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/terraform-tool/ij9rEBqByX0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/83b2e5b6-8984-46b6-a4d6-92d24ea9bf22%40googlegroups.com.

[Joe Barnes]

A quick update on this approach.  With the latest Terraform (0.6.3) and Packer (0.8.6), you may get one of those friendly bug admissions from Terraform when building the infrastructure the first time from scratch: 

* aws_launch_configuration.lift_as_conf: diffs didn't match during apply. This is a bug with Terraform and should be reported.

If you hard-code the AMI into the text file, comment-out the packer provisioner, re-run, it works.  You can then restore the provisioner and it continues to work every time.  It's just that initial creation that fails.  

When I get some time, I'm going to build it from scratch a second time to confirm this is not a one-time issue and report it.

Thanks,

Joe

To unsubscribe from this group and all its topics, send an email to terraform-tool+unsubscribe@googlegroups.com.