Adding Conditionals

[Danny Cosson]

It seems like adding conditionals would be huge in making variables & modules more flexible. Both being able to do an "if" statement within a resource definition, and also conditionally declare a resource.  Wondering if others have thought about this and/or if it's already in the works.

As an example situation I just ran into, I abstracted "appserver" into a module in my project that creates and provisions an ec2 instance and creates a corresponding security group for it.  I wanted to be able to optionally launch the ec2 instance with an iam_instance_profile, but I don't think I can do this in the same module (unless I'm missing something). A null value for an optional resource that is the same as not setting it would work, but conditionals would work well too and would probably also be useful in a lot of cases where a null value wouldn't work.

[Armon Dadgar]

I do think that conditionals would go a long way in increasing the flexibility of Terraform.

I’m curious what kinds of situations you’d want to be able to use conditionals, and what

kinds of operators / comparisons would be useful.

I can at least see enabling / disabling resources based on environment variables as

being a huge step in the right direction, but curious to learn about other use cases users

may have.

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To post to this group, send email to terrafo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/7b552926-cc98-4c93-8e1e-c5f5f8c32de6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Cunningham]

<rant>

If you add language features in an ad-hoc way every time users want some new behavior, you end up with a bad language that is hard to maintain, hard to use, and full of gotchas / bad feature interactions.  I'd like to think I'm an expert on programming languages: I have a decade of experience with interpreters, compilers, I have published to top PL conferences and my PhD was in PL theory.  I've seen this sad story unfold so many times.  Especially with configuration languages that are typically designed by application domain experts as opposed to programming language experts.  But even actual programming languages aren't immune, e.g. PHP / Javascript.

First people want variables, then they want conditionals, then loops, then they want functions, then the ability to expand templates, etc.  Just look at nginx, puppet / salt / ansible.  Look at makefiles.  Look at bash.  Each of these things started off with a limited scope then evolved into a weird Turing-complete language that is hard to learn, has poor tool support, and is generally unpleasant.  The only reason they stop adding features is that the complexity budget runs out.  And then some poor bastard has to debug the garbage collector! (Yes, even bash has a garbage collector of sorts).

The trap is that (at least at first) these languages are trivial to implement.  It seems like you can learn on the job, devote a bit of time to it and move on.  Compared to other things I've worked on: compilers for GPUs, languages for massively parallel supercomputers, advanced static type systems / program analyses, optimizations...  All of that stuff is orders of magnitude harder than configuration languages.  With those things you fail *fast* if you try to be ad-hoc.  However, people think that because configuration languages are much simpler to *implement*, that means that you don't need a programming languages expert to design one.  Wrong!  You just fail more slowly!

Good configuration language design is critical for usability.  The fundamental balance of expressiveness (see Felleisen's paper) and the interaction of different language features is still of paramount importance, even in configuration languages!  You might think configurations are small, so programming language issues are not important.  This is true at first, but it quickly becomes very untrue.  I can't divulge how many lines of code Google has written in configuration languages, but I will say it is enough to need regression testing and automatic refactoring tools, just for the configurations!  That means the hard-learnt lessons about programming in the large (modularity, encapsuation, etc) play a role.  Also of extreme importance for usability is consistency with other languages / paradigms, which necessitates having a very thorough understanding of the semantics of those languages to make sure you don't accidentally deviate from them (it'll be too late to fix it later).  And then there's documentation: for any serious language, it is necessary to write a proper specification so that you don't end up with several implementations that are slightly different, and of course you need thousands of unit tests to catch bugs in corner cases.  Of course all of this assumes that the language is successful, but who doesn't plan for success?  Doing all this properly is a ton of work.

So I would be extremely wary of adding more language features to Terraform.  Keep it as a deployment tool.  Don't make another ad-hoc configuration language.  Deployment tools are already hard enough!  It is trivial for users to put something like Jsonnet in front of Terraform to get a more powerful configuration language, and you should encourage them to do that.  Some people want a tool like Jsonnet (which is based on the aforementioned Google configuration language), but some people want to write Python, Ruby, Javascript, Lua, Jinja-d YAML, whatever!  The OP's concerns are answered by any of these options.  Let people make their own decisions about how to generate / abstract their configurations.  Let them use the same languages to generate configurations for other tools too.  Every person has different preferences and every company's culture is different.  It makes no sense to prescribe a method when you can be agnostic instead.  Especially if it saves you a ton of work :)

Keeping these concerns separate will make the world a better place.

</rant>

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/etPan.54762b58.19e21bb2.2c4f%40Armons-MacBook-Air.local.

[Diego Zamboni]

I absolutely agree with David’s rant. I’d been meaning to write about this, but David said it better than I could have :) From my point of view, the Terraform configuration language can be simple and fairly static, because I will mostly be generating TF config files automatically using a configuration management tool (my poison of choice is CFEngine, which also suffers from its share of the problems David mentioned). It’s in that layer where the complexity and dynamic features will be.

—Diego

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTjOpegn2WiMMRv5b8xbY5-0w_CKVY6O%3Dg%3DTN8H%2B%2Bh7u%2BA%40mail.gmail.com.

[Mitchell Hashimoto]

I'll step in here.

I don't want anyone to construe Armon's comments to mean we're adding
conditionals or "language features" to Terraform's language. Armon was
just trying to understand use cases better.

We're heavily in the camp at HashiCorp that we don't want to build a
programming language into Terraform. If we did, we would've just
embedded a complete language (such as Lua). Instead, we chose to use a
configuration format (libucl), which we further simplified (removed
features) into our own format (HCL). We went one step further and
supported JSON as a complete and first class citizen into the
configuration.

This was a very calculated move.

HCL is our human-friendly configuration piece. JSON is the
machine-generatable configuration piece. Both are completely first
class, both are documented, and both are here to stay.

However, there are features in the configuration language which do
make sense as first class features. One of these is variables. Another
is pulling in default values from environment variables (although this
isn't done yet). Perhaps there is a case to be made for some form of
conditionals, although I'm not yet convinced.

Another example of our hesitation towards complication in the language
is the module system. We wrestled deeply on our side of whether to
introduce some sort of multi-cloud support into modules: having one
module be able to work on AWS, OpenStack, etc. In the end, we decided
for separate modules for each cloud but using directories. For
example:

terraform apply github.com/hashicorp/consul/terraform/aws

or

terraform apply github.com/hashicorp/consul/terraform/openstack

That being said, we're not so stubborn or stuck in our ways that we're
closed to new ideas. Armon and I very much welcome these ideas, even
if we have a feeling it isn't something we want. Therefore, I welcome
use cases and examples of where something might be useful.

In the end, we'll come up with some solution to the problem. And maybe
that solution is "use a real PL to generate this configuration in
JSON, because you can" but maybe the use case represents an extremely
common pattern that needs to be solved in Terraform itself. We can't
know unless we listen to our users, and that is what we do here.

So, thank you for all your feedback. We're certainly not making a
decision yet but I hope this shows you where we stand in our thinking
and our process.

Do not fear, HCL will always be what we feel is the minimum required
to hit the real world use cases.

Best,
Mitchell

> https://groups.google.com/d/msgid/terraform-tool/F6E2AA7F-B84F-451F-ADC4-1F31D0384CA0%40swisscom-labs.com.

[Kief Morris]

On Wed, Nov 26, 2014 at 8:33 PM, Armon Dadgar <armon....@gmail.com> wrote:

I can at least see enabling / disabling resources based on environment variables as

being a huge step in the right direction, but curious to learn about other use cases users

may have.

You can do this with the "count" attribute [1] on some resources, setting it to a variable that may be either 0 or 1.

But I agree with what others have said. Terraform's power comes from being focused on declaring your infrastructure. It's a different mental model from programming, and can be quite powerful. If you find that you're struggling to achieve something without programming constructs, you may need to step back and think about how to approach the problem within a declarative mindset. Or you may need to consider a different tool, either in addition to or instead of terraform.

I'm working with a team of developers and sysadmins using terraform, and it's a change in mindset for everyone, even people who are comfortable with automation tools like chef and puppet.

[1] https://www.terraform.io/docs/configuration/resources.html

 

[Alexander Piavlo]

I think I have a usecase, explained here https://github.com/hashicorp/terraform/issues/581 (if you think it's valid usecase?), where a "kind of" declarative conditional logic is needed to be handled by terrafrom and not 3rd party tool since it directly maps to terrafrom statefiles which terrafrom itself needs to manage.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CACOXMX6150dRQyd1VLRxnNQCbwYLmeQVMBKNx2zS8YtktvWviA%40mail.gmail.com.

[Kief Morris]

Let me see if I understand the use case:

We're using a terraform configuration which may be used in two different situations: 

1) Subnet_b exists and is not managed by terraform.

2) Subnet_b does not exist, so we want terraform to create and manage it.

What is the intent behind this? The wording of the issue suggests to me that this is a migration strategy: once we apply this configuration in situation 1, subnet_b becomes managed by the terraform configuration, and is now situation 2. Is this right?

If so, then this doesn't seem like the kind of thing we'd want to have in our terraform configuration, instead we'd want a one-off mechanism to import existing resources into terraform state.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CALhradntdnpAGwi_EBQ2iOpSNKTGsvkg0%2Bf_%3DhGD2YN-TpnmSg%40mail.gmail.com.

[David Cunningham]

I confused things by replying directly to the issue :)  Here is what I wrote:

> I think this doesn't need language support, instead you just need the ability to import the existing subnet into the tfstate file (and give it the name "b" in the process). Then Terraform would reuse the existing one instead of creating a new one to fill the "b" slot. I.e. that conditional if defined(...) essentially already exists in the core terraform planning logic to check if a resource already exists in the tfstate.

(I think I agree with Kief)

[Alexander Piavlo]

Yes but who'd know best if not terrafrom how to import provder resources into it's statefiles?

As well the import needs to be tightly coupled with the terraform config, since resource need to be imported and referenced in config by other resources.

So even if there is a separate import mechanism, it requires an enhanced terrafrom config support with ability to say try to import resource A based on the resource proivder api id, and then the import would generate a normal terrafrom config.

It can be a two pass terrafrom plan, first import existing resources and then do the usual plan.

The import is not one time operation, i'd want it to happen each time terrafrom plan/apply run , since we need to add/import new resources between runs.

I don't want to implement the import mechanism  in 3rd party tool as this this requires deep knowledge of terrafrom internals and would require heavy terrafrom code base reuse any way. Then terrafrom executes it's plan it already knows how to query the relevant provider api's of the resource it manages, it's seems logical be be able to plugin in to the process and say if resource is not managed yet (not exists in statefile) then here is an resource id (passed through tf variable) pls query it and work from the current state that you just got from API instead of a null state.

[Alexander Piavlo]

Yes i agree there is no need for native language support, it's more of like a macro support for terrarorm config, and import tool (like a c pre-processor) would convert the macro based config into native config (the general conditionals can be implemented as macros too and handled by the import tool), the main point is that the import tool needs to be part of terrafrom and not some 3rd party tool since it's not just a dump pre-processor as it need to understand terraform internals and statefiles and manipulate them and be able to query terrafrom proivders, and all this is something native terrafrom already knows how to do.

[Danny Cosson]

I definitely see the reasoning for limiting the complexity of terraform. I also don't know much about jsonnet but it sounds pretty powerful, I'll look into it.

Thinking about it a little more, existing features like count variables and map values do cover a lot of cases you could also use conditionals for. E.g. A variable that can be set to 0 gives the option of not creating a resource, you can use maps to choose ami based on region, etc.

With the example I just ran into I think it could be solved by adding a null value (a value equivalent to the parameter not being set). For instance if I want an optional parameter of a resource (e.g. iam_instance_profile) to still be optional but allow setting it via a variable I don't think there's currently a way to do it.

Hope that makes sense. Also, happy thanksgiving everyone! Thanks for putting out such a useful tool