Generating Terraform Config

[Louis Garman]

Could people share their experiences of generating Terraform configuration? i.e. rather than writing their terraform configuration by hand, have it generated via a script. I see @eropple created a tool to help with this: Terraframe [1], a ruby DSL (I read he's had difficulties getting this off the ground [2], too). I'd be interested to hear from those that have tried or are following such an approach today, whether with Terraframe, or anything else, and what if any problems they ran into.

I'm an experienced Terraform user. I spent 6 months maintaining a complex AWS infrastructure, writing dozens of modules, refactoring them mercilessly, and maintaining a YAML file alongside it, running rake tasks to read it and inject variables into terraform via the command line. We did all this to manage multiple environments with varying parameters, and to manage the local state files for each (we stored remote state in Atlas). We had a Hashicorp guy come in, and he advised against this approach, and I didn't disagree with him but couldn't see an alternative. I kind of wish I gave the approach I'm asking about today a whirl (didn't cross my mind, perhaps I forgot Terraform supports JSON as well as HCL).

But why not? Something like the following sounds eminently sensible:

Scripts/DSL -> Terraform JSON -> Infrastructure

The scripts are responsible for gathering all the 'data' (all those resources that aren't actually infrastructure, as per #4169), and all the work that necessitates the use of a fully expressive language. That can be control structures (loops, conditional statements), string manipulation functions, traversing data structures, lookups of external data, etc,. The generated Terraform JSON would forgo all of those features, leaving what would instead be a pure specification of your infrastructure. It would be massively redundant, and a wonderful bore to read, being nothing but a catalog of infrastructure components referencing one another. Variables and modules could possible be done away with too.

This isn't very different to how many people build CloudFormation templates. They use one of the myriad DSL's to generate the JSON. They do this because they have to; CloudFormation doesn't have counts or much in the way of string functions like Terraform does. But even if it did, I'm pretty sure those DSL's would continue to thrive, for all the reasons given above.

[1] https://github.com/eropple/terraframe

[2] https://news.ycombinator.com/item?id=10795749

[Ameir Abdeldayem]

I've had some wild adventures wrangling HCL configs, and went the JSON route.  To facilitate this, I wrote https://github.com/ameir/terraform-php .  I haven't prepped it for public use (no documentation, and more examples could be added), but it's fairly straightforward.  It basically creates a hash, and serializes it to JSON.  I know PHP isn't sexy, but it gets the job done.

There are "helpers" that talk to the AWS API, which are useful for listing available AZs, subnets, etc.  There are also "macros" that generate most of a resource for you (e.g. IAM roles and policies).

There is a simple example of what you can do with it at https://github.com/ameir/terraform-php/blob/develop/tests/test.php .

If you have any issues or questions, feel free to create a Github issue.  I'll be extending it a bit more, but it's been doing well so far with creating fairly complex infrastructures on AWS.

Note that although I've only added macros/helpers for AWS so far, basic usage works for any Terraform resources.

Thanks,

Ameir

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/aec901f8-5e7e-40d6-bcb8-7aa878e8f5ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Cunningham]

Jsonnet was designed to be such a DSL and has been used for exactly what you described as well as a number of other use cases, including Packer, Cloud Formation, Kubernetes, Avro, and so on.  I did a write up of a pretty complex case study generating Terraform and Packer here. 

Jsonnet is hermetic and fully reproducible (as every config language should be) so by design it cannot talk to other servers to get more config data during config elaboration.  You do that before invoking Jsonnet, and pass that data in using parameters.  This guarantees being able to replay a previous version of the config and also gives you a way tow write an audit trail.  You have to write a script (or a Makefile) anyway to run Jsonnet then run Terraform, so you can put that stuff (e.g. fetching the latest github revision or whatever it is) in the wrapper script.

The experience working on that fractal demo lead me to work on a new tool called "micromanage" which is a commandline tool that wraps Terraform and Packer and exposes a new JSON spec (that has modified portions of Packer and Terraform JSON embedded inside it, so it's not really wrapped, just augmented) and uses Jsonnet to abstract that new JSON.  The purpose of this is to be service-oriented rather than infrastructure oriented.  It allows defining microservices that each have infrastructure, images, application config, etc. as well as nesting other microservices.  These can be imported from libraries, extended / modified / specialized for different environments, etc. using Jsonnet as the language.  The end game is to be able to construct and manage web apps using existing high-level components, e.g. building a youtube-like service from a standard application server, a standard user account storage service (that handles hashing, tokens, password reset emails, etc), a standard video transcription pipeline (e.g. built on cassandra + rabbit mq + a pool of processing machines), each of which can have various database services underneath them, expose REST interfaces, and have integrating monitoring, logging and alerting.  All of that high level stuff is done using Jsonnet templates, which compile down to essentially just Terraform and Packer config files (that embed bash fragments).  Currently this only supports GCE and I'm part way through adding AWS support.  It's all very experimental but I can show you if you're interested, because when you make that switch to make things service oriented instead of infrastructure oriented, it's a game changer for config abstraction.

On Wed, Jan 20, 2016 at 6:58 AM, Louis Garman <louis...@gmail.com> wrote:

--

[Louis Garman]

Thanks for sharing your projects Ameir and Dave. You've confirmed this is a valid approach and I look forward to adopting it on my next project.