compound variables - why is terraform so broken? Any workarounds?

[s...@stem.is]

The language ostensibly supports lists of maps and maps of lists, except that there is no way to extract values FROM such a structure once it is created.  I'm sure I'm not the first person to be sick and tired of having error-prone parallel lists that MUST have correlated order between them in order to ensure that I can extract values from both lists in parallel.  Is there some workaround that I'm not seeing?  I've got some resources about which I have to store information.  The only logical method for doing that is to store a map for each resource - but some of the keys in the map really need to be lists.  So how do people deal with this kind of problem?

resources = [

  {

    "name" = "resource1"

    "property1" = "property1 value"

    "property2" = ["val1, "val2", "val3"]

  },

  {

    "name" = "resource2"

    "property1" = "property1 value2"

    "property2" = ["val1, "val2", "val3"]

  }

]

I can access the map via resources[count.index], which is the one function/operator that seems to work correctly on compound structures.  However, the obvious syntax of resources[count.idex]["property2"] throws an error 'expected ")" but found "["'

There's the jsonencode() function, which will apparently correctly render a compound structure (not that I've tested it), but there's no tool for manipulating the json and getting a value back out of it, so that's useless.  

Is there some workaround for this that I'm not seeing?

And, really, why did hashicorp feel the need to invent their own language and parser for doing this kind of thing when json, yaml, python, ruby, even bash all have the ability to do variable replacement and ALL have more sophisticated tools for working with data structures?  If you are going to do it yourself, shouldn't your version be somehow superior to at least one thing that is already out there?  What's the advantage of this ridiculously broken language?  I lose DAYS to dealing with issues like these, not to mention the absolutely TERRIBLE error reporting that terraform tends to generate, often hiding the true error behind error messages that communicate absolutely nothing useful. 

It's not like I can have a list of lists, either, so I can't pull "property2" out into a separate structure.  I have to actually invert the entire mapping of name to every possible value of property2

property2_val1 = [name1, name2, name5]

property2_val2 = [name3, name4]

property2_val3 = [name1, name4, name5],

etc

And that only works if I have a known set of values in property2, since there is no way to look up the correct list by name in a map of lists.  This is beyond dumb. It is guaranteed to lead to errors, eventually, when someone fails to set up a new resource correctly.  It's the polar opposite of what a tool like terraform is supposed to be doing - making it easier and LESS error prone to manage infrastructure.

From what I can tell, the whole reason for not supporting compound structures seems to be because the code doesn't ever pass maps or lists through to a new scope when you call a function.  Instead, it renders a lists as a string with a known separator and then parses it back to a list the next time it is referred to as a list (which makes me wonder how it would even work if the values in a list contain the delimiter character).  But if you are going to do that, why not render a string that actually retains information about internal structures?  If a list is rendered to a string as "val1, val2, val3" and a map is rendered as "key1, val2, key2, val2, key3, val3" . then there is, indeed, no way to parse a map of lists or a list of maps, since there's no way to tell where one begins and the other ends.  But would it really be so difficult to render a string form as "[val1, val2, val3]" and a map as "{key1: value1, key2, val2, key3: val3}" so that a parser could differentiate between them?  Isn't this a computer languages 101 concept?  Never mind the whole 'why didn't you use json or yaml for this in the first place?" question.

[Dave Cunningham]

You can just pass JSON to Terraform and generate that JSON using jsonnet.org (or python, ruby, etc., but Jsonnet is designed for this).

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/253b9c91-5469-4a78-990e-53a0790091dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Adams]

Nothing you describe here has anything to do with HCL. You can use JSON instead.

But to the rest of your complaints, I think you're underestimating the complexity involved. It's not just about parsing the data structures. TF has made big strides in this area over the past two and a half years, and things are getting a lot better. BUT part of using TF effectively is unlearning some old habits and finding new ways to approach the problem. TF isn't a programming language, so it's a bad idea to talk about it as if it were.

--

[Samuel Gendler]

Huh?  How is that an answer to any of my problems described in the email?  I don't want to pass json to terraform. I want to have variables of compound type and still be able to get individual values back out.  Passing JSON in gets me mowhere, since I have no tools for manipulating json within terraform.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTgRdsGdmTDX0GiH_Q55HWiSJX0E7L%3DjDcAs5VaYqoqL9w%40mail.gmail.com.

[Samuel Gendler]

OK, so feel free to explain to me what 'old habits' I need to abandon to make this work correctly.

If I want to describe an object, I need to do it in ONE place, not in 5 places - because that's how you prevent errors.  If I have to maintain 5 separate lists rather than a list of maps with 5 keys, that is VERY MUCH a step BACKWARDS from a usability and error prone-ness concern.  At a minimum, I'm at risk of putting elements in one of the lists in the wrong order.  But as soon as you have a compound structure more complicated than that, it becomes effectively impossible to render the compound object as a series of flat structures, even if you aren't concerned about the sheer number of lists and maps you'd likely be required to create, because you would have no way to dynamically look them up by name, since you can't have a map of maps or a map of keys.  It's absurd to tell me that I just need to learn a new way to do things.  Either terraform provides a mechanism for me to describe all of the variables required to define a resource or it does not.  And clearly, HCL really does NOT do that.  What am I supposed to do, write an entire plugin just to get a datasource that can parse json and extract values one at a time?  That's an absurd workaround.  If I have to do that, why bother with terraform at all?!

And if I can 'use JSON instead' what does that mean exactly?  How do I use json?  I can put json in variables somehow?  If so, how do I then manipulate that JSON in order to extract anything out of it?  Is there documentation that mentions json anywhere, because I sure don't see it.  What I see is a language that is, at best, half-implemented.  Why support compound structures if none of the functions for manipulating structures can accept them?  I don't care about writing my templates in JSON.  I care about being able to use compound structures in variables.  Writing my template in JSON doesn't get me that, unless I can also use javascript as the interpreter when accessing variables.  So far as I can tell, JSON in templates is simply converted into HCL and is then just as broken as HCL itself.  Am I missing something here?  The only reference to JSON in the docs is a completely trivial example demonstrating how a template can be written in JSON instead of HCL.  No mention is made of a js interpreter or accessing js functions when doing variable replacement.  In fact the simple example provided in the doc is clearly still using HCL for variable replacement. If I can use JS to lookup values in json variables, there should be some documentation somewhere describing how to do it, no?

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAN3s8zb%3DogoFoR8RJ_iO6N%2B9vUzGfQmcjPBUbB-%2BOM2SHq7khw%40mail.gmail.com.

[Dave Cunningham]

You pass fully-pre-processed resources to Terraform in JSON (doesn't have to be JSON but that's the easiest thing to output from another tool) rather than doing that processing within Terraform itself.  That way, you leverage Terraform just for wrapping cloud APIs, and building plans for implementing changes.  You can use a programming language of your choice for processing (config generation).

https://github.com/google/jsonnet/tree/master/examples/terraform

On Fri, Jan 12, 2018 at 2:31 PM, Samuel Gendler <s...@stem.is> wrote:

Huh?  How is that an answer to any of my problems described in the email?  I don't want to pass json to terraform. I want to have variables of compound type and still be able to get individual values back out.  Passing JSON in gets me mowhere, since I have no tools for manipulating json within terraform.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-h7PtybrXn1HVti-aRC2CYHXsoq4K012FhPq9FXfUaYwQ%40mail.gmail.com.

[Samuel Gendler]

So workaround the glaring holes in terraform by actually doing all the work in an entirely different tool and then rendering the terraform config from there?  That's just a great solution. Isn't the whole point of terraform supposed to be to allow declarative infrastructure and state management?  Now I have to write imperative code to render the declarative template so that the tool can work?  At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?  If a tool is so lacking in features that you have to wrap it in yet another tool to make it even marginally useful, what utility does the tool actually offer?  The whole point is to keep state about resources in variables and to have STATIC templates for defining those resources.  If I have to have dynamically generated templates, that breaks everything else within my infrastructure, which was all implemented according to 'best practices' suggested by the terraform community, splitting my variables from my templates and using terragrunt to combine them at execution time.  But I can't do that if I have to render my templates dynamically.

But more to the point, json doesn't actually help with this problem, whether I pre-render it prior to feeding it to terraform or not.  My whole problem stems from wishing to create a list of similar resources with a single resource declaration and a list to iterate over in order to populate the resource with actual values.  The whole point is to declare a resource once, with a count, and then allow terraform to extract values from variables dynamically.  But I cannot do that, with jsonnet or terraform templates, because I have no way to do the dynamic lookups when trying to populate the variables at the point where terraform turns a resource with count > 1 into multiple resources.  The json solution offers absolutely NOTHING to assist with that - other than, I suppose, to render the entire list of nearly identical resources as a giant long list of individual resources in the template - which then means I won't be able to use terraform's splat syntax for referring to them en-masse or otherwise extracting values from one resource and using it in another, since my resources would be dynamically generated, so how to refer to them?  I suppose I have to manually iterate over the resources in jsonnet and construct output variables dynamically, too? JSON is barely a kludge for this problem, let alone a real solution.

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTjeLU5orQgOX5bdYB5w%2Bf_tG_SbBxs1ysQ-gLWpOXZVhg%40mail.gmail.com.

[Philip Nelson]

Not sure what about your excellent way of engaging volunteers to help you made me respond, but there is one other way to process more complex documents like you want. Put the full configuration you want into a json file, then write a shell script that takes an input parameter and outputs a more flattened json document with only what you need back. It can usually be wrangled to work pretty simply, not a ton of code, tools most of us use pretty regularly anyway. I used it once to pull values out of ansible vault for example. 

https://www.terraform.io/docs/providers/external/data_source.html

On Fri, Jan 12, 2018 at 2:20 PM, Samuel Gendler <s...@stem.is> wrote:

So workaround the glaring holes in terraform by actually doing all the work in an entirely different tool and then rendering the terraform config from there?  That's just a great solution. Isn't the whole point of terraform supposed to be to allow declarative infrastructure and state management?  Now I have to write imperative code to render the declarative template so that the tool can work?  At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?  If a tool is so lacking in features that you have to wrap it in yet another tool to make it even marginally useful, what utility does the tool actually offer?  The whole point is to keep state about resources in variables and to have STATIC templates for defining those resources.  If I have to have dynamically generated templates, that breaks everything else within my infrastructure, which was all implemented according to 'best practices' suggested by the terraform community, splitting my variables from my templates and using terragrunt to combine them at execution time.  But I can't do that if I have to render my templates dynamically.

But more to the point, json doesn't actually help with this problem, whether I pre-render it prior to feeding it to terraform or not.  My whole problem stems from wishing to create a list of similar resources with a single resource declaration and a list to iterate over in order to populate the resource with actual values.  The whole point is to declare a resource once, with a count, and then allow terraform to extract values from variables dynamically.  But I cannot do that, with jsonnet or terraform templates, because I have no way to do the dynamic lookups when trying to populate the variables at the point where terraform turns a resource with count > 1 into multiple resources.  The json solution offers absolutely NOTHING to assist with that - other than, I suppose, to render the entire list of nearly identical resources as a giant long list of individual resources in the template - which then means I won't be able to use terraform's splat syntax for referring to them en-masse or otherwise extracting values from one resource and using it in another, since my resources would be dynamically generated, so how to refer to them?  I suppose I have to manually iterate over the resources in jsonnet and construct output variables dynamically, too? JSON is barely a kludge for this problem, let alone a real solution.

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hRedkc3uThrDQfQtDGAycVFeGOVjQOxZ%3DYDdDgT45SAw%40mail.gmail.com.

[David Adams]

Samuel, do you have a specific question we can help you with? Lots of people on this list who use Terraform to manage lots of infrastructure successfully. But so far it's hard to know exactly what you're asking for.

In regards to what I think you are saying, about wanting to keep the data all together in on "object". Yeah, I'd also like this to be improved, but in the meantime, it can be worked around. If you don't want to use software that requires workarounds, that's cool. If you know of what that software is, let us know.

On Fri, Jan 12, 2018 at 2:20 PM, Samuel Gendler <s...@stem.is> wrote:

So workaround the glaring holes in terraform by actually doing all the work in an entirely different tool and then rendering the terraform config from there?  That's just a great solution. Isn't the whole point of terraform supposed to be to allow declarative infrastructure and state management?  Now I have to write imperative code to render the declarative template so that the tool can work?  At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?  If a tool is so lacking in features that you have to wrap it in yet another tool to make it even marginally useful, what utility does the tool actually offer?  The whole point is to keep state about resources in variables and to have STATIC templates for defining those resources.  If I have to have dynamically generated templates, that breaks everything else within my infrastructure, which was all implemented according to 'best practices' suggested by the terraform community, splitting my variables from my templates and using terragrunt to combine them at execution time.  But I can't do that if I have to render my templates dynamically.

But more to the point, json doesn't actually help with this problem, whether I pre-render it prior to feeding it to terraform or not.  My whole problem stems from wishing to create a list of similar resources with a single resource declaration and a list to iterate over in order to populate the resource with actual values.  The whole point is to declare a resource once, with a count, and then allow terraform to extract values from variables dynamically.  But I cannot do that, with jsonnet or terraform templates, because I have no way to do the dynamic lookups when trying to populate the variables at the point where terraform turns a resource with count > 1 into multiple resources.  The json solution offers absolutely NOTHING to assist with that - other than, I suppose, to render the entire list of nearly identical resources as a giant long list of individual resources in the template - which then means I won't be able to use terraform's splat syntax for referring to them en-masse or otherwise extracting values from one resource and using it in another, since my resources would be dynamically generated, so how to refer to them?  I suppose I have to manually iterate over the resources in jsonnet and construct output variables dynamically, too? JSON is barely a kludge for this problem, let alone a real solution.

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hRedkc3uThrDQfQtDGAycVFeGOVjQOxZ%3DYDdDgT45SAw%40mail.gmail.com.

[Dave Cunningham]

On Fri, Jan 12, 2018 at 3:20 PM, Samuel Gendler <s...@stem.is> wrote:

At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?

Terraform actually does a lot more than just calling the API:

- computing diffs

- figuring out what calls to make, e.g. whether the resource needs to be recreated to make the change

- doing changes in parallel

- figuring out the order required if they can't be done in parallel

- figuring out that if an object has to be recreated, sometimes other objects have to be recreated too

- correcting annoying errors in the APIs

- remembering what resources it's managing so it can delete them if you remove them from your config

All of this is "hard" and it'd be perfectly reasonable for a tool to do just that, without the language stuff.  The language stuff I'm referring to is not HCL but Terraform's variables, modules, count attribute, string interpolation syntax, etc.

 

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

For converting HCL to JSON i believe there are tools like this one https://github.com/kvz/json2hcl

But also as you have realized, if you go down this rabbit hole you have to go all the way - you would want to be entirely on one side or the other, because the two language worlds are not going to interoperate. So yes, it would take time to rewrite them.

[Josh Smift]

We've also been somewhat frustrated by HCL and its limitations. Our current intention is to check out terrascript (https://pypi.python.org/pypi/terrascript/) as a way to get a more full-featured configuration language that still lets us leverage the many other advantages of Terraform over other provisioning tools, like the AWS CLI (no logic at all around dependencies, no way to compare current and desired state, no way to make a plan and apply it as separate operations, etc), or Ansible (which we use and like a lot, but generally tends to lack the "compare current and desired state" part in particular).

[Samuel Gendler]

Ok, so I’ll write yet another tool to cause terraform to workaround large deficiencies in its own capabilities. Why does terraform have its own configuration language at all if you have to go to an external language to do even relatively simple tasks? It’s not as though questions about working around this limitation in terraform don’t go back years wherever the tool is discussed. 

Isn’t the whole point of terraform to allow me to put external state about resources into variables, and then allow terraform to render templates combined with variables in order to manage infrastructure in a declarative manner? Isn’t its declarative nature supposed to be one of terraform’s selling points? How does rendering templates dynamically via another language further terraform’s declarative nature?

If I have to write another external system that terraform can talk to whenever life gets complex, doesn’t that defeat the whole purpose of managing complexity WITHIN terraform? Now I no longer have the external state of my systems all contained in simple (and human-readable) variable declarations in my tfvars files. Now, some fraction of that external state is only available from the tool that I have to write for terraform to interact with. But if I wanted to write a bunch of code for managing state, why wouldnt I use terraform, which is supposedly a tool for doing exactly that? Instead of having all the state in one set of tfvars files, I’ll have it split between terraform and some external system using a completely different syntax. I’ll have to ensure that system is tested. There are revision control considerations. Access control considerations. Plus, it just breaks the consistency and declarative nature of everything that terraform is supposed to be. Now I need a meta-terraform for rendering templates that terraform can actually interact with? 

This solution GREATLY adds to the complexity of the problem and, frankly, I don’t even see how it really helps, since I’m trying to make things function via a single resource declaration with a count, meaning I need to be able to pass some kind of unique value for each instance, which gets me right back to maintaining multiple data structures that have to be maintained, only now they aren’t even in the same file or managed by the same tool. At the very least, this likely means that I’ve got some complex Json structure in a file in my template repository, along with a list of keys in a tfvars file in my environments repo. That’s arguably worse than multiple parallel structures all contained within my tfvars files.

And if you object to how I communicate about this issue, it’s only because I’m fairly upset that I’ve been using terraform for 2 months now, and it consistently takes longer and is more difficult to do pretty much everything I ever try compared to just implementing it myself. I’ve spent 8 weeks automating an infrastructure that I could have automated manually with some scripts in the language of my choice in 2 weeks or less. I’ve been working under the assumption that I was climbing a learning curve, but the reality is that learning curve isn’t the problem. The problem is that terraform simply doesn’t work very well. In those 8 weeks, I’ve seen a software release that was not even remotely backward compatible with previous versions, without deprecation of the old syntax or even a warning prior to release. And I’ve run into intractable problems with functionality like this one on a weekly, if not daily, basis. The whole ‘automate our automation tool with an external language’ response is such a cop out to providing a functional tool. As was hashicorp’s response to the whole ‘we broke all your templates with 0.11’ issue. Backward compatibility? What’s that?

If I could do it over again, I would never have wasted all of the time I put into terraform in the first place. I’d be far ahead of where I am AND my infrastructure automation would actually function in a consistent manner. The idea that I should have to overcome deficiencies in HCL by generating ‘expanded’ HCL (which is really just HCL but less human-readable) in some meta-tool just blows my mind. It would be one thing if that were a temporary workaround  while waiting for an imminent fix, but as someone pointed out in this thread, it’s been 2.5 years now. In 2.5 years, I can implement all of the infrastructure automation I need with direct access to the AWS-SDK in the language of my choice. 

On Fri, Jan 12, 2018 at 12:47 Philip Nelson <panman...@gmail.com> wrote:

Not sure what about your excellent way of engaging volunteers to help you made me respond, but there is one other way to process more complex documents like you want. Put the full configuration you want into a json file, then write a shell script that takes an input parameter and outputs a more flattened json document with only what you need back. It can usually be wrangled to work pretty simply, not a ton of code, tools most of us use pretty regularly anyway. I used it once to pull values out of ansible vault for example. 

https://www.terraform.io/docs/providers/external/data_source.html

On Fri, Jan 12, 2018 at 2:20 PM, Samuel Gendler <s...@stem.is> wrote:

So workaround the glaring holes in terraform by actually doing all the work in an entirely different tool and then rendering the terraform config from there?  That's just a great solution. Isn't the whole point of terraform supposed to be to allow declarative infrastructure and state management?  Now I have to write imperative code to render the declarative template so that the tool can work?  At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?  If a tool is so lacking in features that you have to wrap it in yet another tool to make it even marginally useful, what utility does the tool actually offer?  The whole point is to keep state about resources in variables and to have STATIC templates for defining those resources.  If I have to have dynamically generated templates, that breaks everything else within my infrastructure, which was all implemented according to 'best practices' suggested by the terraform community, splitting my variables from my templates and using terragrunt to combine them at execution time.  But I can't do that if I have to render my templates dynamically.

But more to the point, json doesn't actually help with this problem, whether I pre-render it prior to feeding it to terraform or not.  My whole problem stems from wishing to create a list of similar resources with a single resource declaration and a list to iterate over in order to populate the resource with actual values.  The whole point is to declare a resource once, with a count, and then allow terraform to extract values from variables dynamically.  But I cannot do that, with jsonnet or terraform templates, because I have no way to do the dynamic lookups when trying to populate the variables at the point where terraform turns a resource with count > 1 into multiple resources.  The json solution offers absolutely NOTHING to assist with that - other than, I suppose, to render the entire list of nearly identical resources as a giant long list of individual resources in the template - which then means I won't be able to use terraform's splat syntax for referring to them en-masse or otherwise extracting values from one resource and using it in another, since my resources would be dynamically generated, so how to refer to them?  I suppose I have to manually iterate over the resources in jsonnet and construct output variables dynamically, too? JSON is barely a kludge for this problem, let alone a real solution.

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

https://github.com/google/jsonnet/tree/master/examples/terraform

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/253b9c91-5469-4a78-990e-53a0790091dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTgRdsGdmTDX0GiH_Q55HWiSJX0E7L%3DjDcAs5VaYqoqL9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-h7PtybrXn1HVti-aRC2CYHXsoq4K012FhPq9FXfUaYwQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTjeLU5orQgOX5bdYB5w%2Bf_tG_SbBxs1ysQ-gLWpOXZVhg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hRedkc3uThrDQfQtDGAycVFeGOVjQOxZ%3DYDdDgT45SAw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CACRhdyZ7tMQ4OSHwe8MWnUmQHnx_9ED4Ug8PcNLNAX0rQ5gNGA%40mail.gmail.com.

[David Adams]

Here's my advice. If you can reimplement your eight weeks of TF work in two weeks in a system you think is better, you should switch to that now and throw out the Terraform code you've written. Don't chase the sunk cost. If Terraform is not right for your project's needs, then it's not going to be in two weeks from now, either. You will save yourself a lot of grief by moving to your better, faster, more robust and elegant solution right now.

On Fri, Jan 12, 2018 at 3:28 PM, Samuel Gendler <s...@stem.is> wrote:

Ok, so I’ll write yet another tool to cause terraform to workaround large deficiencies in its own capabilities. Why does terraform have its own configuration language at all if you have to go to an external language to do even relatively simple tasks? It’s not as though questions about working around this limitation in terraform don’t go back years wherever the tool is discussed. 

Isn’t the whole point of terraform to allow me to put external state about resources into variables, and then allow terraform to render templates combined with variables in order to manage infrastructure in a declarative manner? Isn’t its declarative nature supposed to be one of terraform’s selling points? How does rendering templates dynamically via another language further terraform’s declarative nature?

If I have to write another external system that terraform can talk to whenever life gets complex, doesn’t that defeat the whole purpose of managing complexity WITHIN terraform? Now I no longer have the external state of my systems all contained in simple (and human-readable) variable declarations in my tfvars files. Now, some fraction of that external state is only available from the tool that I have to write for terraform to interact with. But if I wanted to write a bunch of code for managing state, why wouldnt I use terraform, which is supposedly a tool for doing exactly that? Instead of having all the state in one set of tfvars files, I’ll have it split between terraform and some external system using a completely different syntax. I’ll have to ensure that system is tested. There are revision control considerations. Access control considerations. Plus, it just breaks the consistency and declarative nature of everything that terraform is supposed to be. Now I need a meta-terraform for rendering templates that terraform can actually interact with? 

This solution GREATLY adds to the complexity of the problem and, frankly, I don’t even see how it really helps, since I’m trying to make things function via a single resource declaration with a count, meaning I need to be able to pass some kind of unique value for each instance, which gets me right back to maintaining multiple data structures that have to be maintained, only now they aren’t even in the same file or managed by the same tool. At the very least, this likely means that I’ve got some complex Json structure in a file in my template repository, along with a list of keys in a tfvars file in my environments repo. That’s arguably worse than multiple parallel structures all contained within my tfvars files.

And if you object to how I communicate about this issue, it’s only because I’m fairly upset that I’ve been using terraform for 2 months now, and it consistently takes longer and is more difficult to do pretty much everything I ever try compared to just implementing it myself. I’ve spent 8 weeks automating an infrastructure that I could have automated manually with some scripts in the language of my choice in 2 weeks or less. I’ve been working under the assumption that I was climbing a learning curve, but the reality is that learning curve isn’t the problem. The problem is that terraform simply doesn’t work very well. In those 8 weeks, I’ve seen a software release that was not even remotely backward compatible with previous versions, without deprecation of the old syntax or even a warning prior to release. And I’ve run into intractable problems with functionality like this one on a weekly, if not daily, basis. The whole ‘automate our automation tool with an external language’ response is such a cop out to providing a functional tool. As was hashicorp’s response to the whole ‘we broke all your templates with 0.11’ issue. Backward compatibility? What’s that?

If I could do it over again, I would never have wasted all of the time I put into terraform in the first place. I’d be far ahead of where I am AND my infrastructure automation would actually function in a consistent manner. The idea that I should have to overcome deficiencies in HCL by generating ‘expanded’ HCL (which is really just HCL but less human-readable) in some meta-tool just blows my mind. It would be one thing if that were a temporary workaround  while waiting for an imminent fix, but as someone pointed out in this thread, it’s been 2.5 years now. In 2.5 years, I can implement all of the infrastructure automation I need with direct access to the AWS-SDK in the language of my choice. 

On Fri, Jan 12, 2018 at 12:47 Philip Nelson <panman...@gmail.com> wrote:

Not sure what about your excellent way of engaging volunteers to help you made me respond, but there is one other way to process more complex documents like you want. Put the full configuration you want into a json file, then write a shell script that takes an input parameter and outputs a more flattened json document with only what you need back. It can usually be wrangled to work pretty simply, not a ton of code, tools most of us use pretty regularly anyway. I used it once to pull values out of ansible vault for example. 

https://www.terraform.io/docs/providers/external/data_source.html

On Fri, Jan 12, 2018 at 2:20 PM, Samuel Gendler <s...@stem.is> wrote:

So workaround the glaring holes in terraform by actually doing all the work in an entirely different tool and then rendering the terraform config from there?  That's just a great solution. Isn't the whole point of terraform supposed to be to allow declarative infrastructure and state management?  Now I have to write imperative code to render the declarative template so that the tool can work?  At that point, why wouldn't I just write imperative code which calls the AWS-SDK directly - a method which is both simpler and FAR faster to implement than what you've suggested?  If a tool is so lacking in features that you have to wrap it in yet another tool to make it even marginally useful, what utility does the tool actually offer?  The whole point is to keep state about resources in variables and to have STATIC templates for defining those resources.  If I have to have dynamically generated templates, that breaks everything else within my infrastructure, which was all implemented according to 'best practices' suggested by the terraform community, splitting my variables from my templates and using terragrunt to combine them at execution time.  But I can't do that if I have to render my templates dynamically.

But more to the point, json doesn't actually help with this problem, whether I pre-render it prior to feeding it to terraform or not.  My whole problem stems from wishing to create a list of similar resources with a single resource declaration and a list to iterate over in order to populate the resource with actual values.  The whole point is to declare a resource once, with a count, and then allow terraform to extract values from variables dynamically.  But I cannot do that, with jsonnet or terraform templates, because I have no way to do the dynamic lookups when trying to populate the variables at the point where terraform turns a resource with count > 1 into multiple resources.  The json solution offers absolutely NOTHING to assist with that - other than, I suppose, to render the entire list of nearly identical resources as a giant long list of individual resources in the template - which then means I won't be able to use terraform's splat syntax for referring to them en-masse or otherwise extracting values from one resource and using it in another, since my resources would be dynamically generated, so how to refer to them?  I suppose I have to manually iterate over the resources in jsonnet and construct output variables dynamically, too? JSON is barely a kludge for this problem, let alone a real solution.

And looking at the jsonnet examples, I can only imagine how much coding by trial-and-error I'll have to do to even figure out what the JSON syntax is supposed to look like, since it looks pretty much nothing like the HCL versions.  Every single template is going to require hours or days of trial and error to figure out how to structure things so that it actually works.

https://github.com/google/jsonnet/tree/master/examples/terraform

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/253b9c91-5469-4a78-990e-53a0790091dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTgRdsGdmTDX0GiH_Q55HWiSJX0E7L%3DjDcAs5VaYqoqL9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-h7PtybrXn1HVti-aRC2CYHXsoq4K012FhPq9FXfUaYwQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CANarjTjeLU5orQgOX5bdYB5w%2Bf_tG_SbBxs1ysQ-gLWpOXZVhg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hRedkc3uThrDQfQtDGAycVFeGOVjQOxZ%3DYDdDgT45SAw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CACRhdyZ7tMQ4OSHwe8MWnUmQHnx_9ED4Ug8PcNLNAX0rQ5gNGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hpaX1iTOx3BecxAEoN6%2BU0PhfEJsL5FNtrFLcb3PO71A%40mail.gmail.com.

[Josh Smift]

It sounds like you're pretty upset about having chosen Terraform as a tool. If you don't think it's a good fit for what you're trying to do, you probably shouldn't use it.

Terraform solves a *ton* of problems we have, *despite* the shortcomings of HCL, so we're still using it.

If that's not true for you, maybe you should stop.

Are you hoping to get something out of this conversation other than a place to express your frustration? Are you looking for advice, or alternatives, or validation, or ... ?

I definitely agree that Terraform isn't the tool for everyone. If you feel like it isn't the tool for you, I'm sure your feelings are valid.

[Samuel Gendler]

Well, I was looking for a possible solution when I started the conversation, but when the answer to that question was ‘write a meta-tool to generate a less human-manageable form of terraform’s inputs so you don’t have to worry about the shortcomings’ it mostly became an opportunity to vent. 

I’m already using one tool to manage terraform’s shortcomings - terragrunt. I would have to abandon that to do what you suggest, thereby losing the functionality that it brings to the table for managing larger projects with larger teams of developers. Or else I would have to hack into terragrunt a feature that would render a template before running terraform. If a tool requires layer after layer of meta programming to be useful, it ceases to be useful. I’m just trading one form of complexity for another.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/5f71cfb1-c898-48c0-a730-9c347ecdb4af%40googlegroups.com.

[Lowe Schmidt]

You're venting at people that volunteer their time, and they are actively trying to help you solve the problem you claim to have. Except, you don't seem to want to solve the problem, you seem to get the validation that TF is "crap".

Ditch TF and write it yourself.

--

Lowe Schmidt | +46 723 867 157

On 13 January 2018 at 00:14, Samuel Gendler <s...@stem.is> wrote:

Well, I was looking for a possible solution when I started the conversation, but when the answer to that question was ‘write a meta-tool to generate a less human-manageable form of terraform’s inputs so you don’t have to worry about the shortcomings’ it mostly became an opportunity to vent. 

I’m already using one tool to manage terraform’s shortcomings - terragrunt. I would have to abandon that to do what you suggest, thereby losing the functionality that it brings to the table for managing larger projects with larger teams of developers. Or else I would have to hack into terragrunt a feature that would render a template before running terraform. If a tool requires layer after layer of meta programming to be useful, it ceases to be useful. I’m just trading one form of complexity for another.

On Fri, Jan 12, 2018 at 13:44 Josh Smift <jbs.care...@gmail.com> wrote:

It sounds like you're pretty upset about having chosen Terraform as a tool. If you don't think it's a good fit for what you're trying to do, you probably shouldn't use it.

Terraform solves a *ton* of problems we have, *despite* the shortcomings of HCL, so we're still using it.

If that's not true for you, maybe you should stop.

Are you hoping to get something out of this conversation other than a place to express your frustration? Are you looking for advice, or alternatives, or validation, or ... ?

I definitely agree that Terraform isn't the tool for everyone. If you feel like it isn't the tool for you, I'm sure your feelings are valid.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/5f71cfb1-c898-48c0-a730-9c347ecdb4af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAGcMw-hgxF9wdz3j6cKkvAfouqk%2Bfc6Jt_gZgdpj%2BfGgi7u2vA%40mail.gmail.com.

[Tim Sattarov]

I think I might have a solution for that:

I use nested structures and access them through map and list functions and their compositions
something like this:

resource "aws_subnet" "cache" {
  count             = "${length(var.subnet_cidr["cache"])}"
  vpc_id            = "${aws_vpc.vpc.id}"
  cidr_block        = "${element(var.subnet_cidr["cache"],count.index)}"
  availability_zone = "${var.az[count.index]}"
}

Where subnet_cidr looks like:

variable "subnet_cidr" {
  type = "map"

  default = {
    public = [
      "10.100.0.0/24",
      "10.100.1.0/24",
      "10.100.2.0/24",
      "10.100.3.0/24",
      "10.100.4.0/24",
    ]

    cache = [
      "10.100.150.0/24",
      "10.100.151.0/24",
      "10.100.152.0/24",
      "10.100.153.0/24",
      "10.100.154.0/24",
    ]
}

​

Is it what you are looking for?

-- 
Tim, the beardless keeper of keys and grounds @ AgileBits Inc.
~~~~~~~~~~~~~~~~~~~~~~~
1Password remembers all your passwords for you. It keeps your digital life secure and always available, safe behind the one password that only you know.
Create your 1Password account today at https://1password.com

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/253b9c91-5469-4a78-990e-53a0790091dd%40googlegroups.com.