Use of standard file formats/syntax?

[Robert Parrott]

Hi All,

First off I'm pretty pumped about the terraform project and its potential. However I'd had a steep learning curve, and in reflecting on that, much of it has to do with the non-standard syntax used across the board, and the lack of feedback in the tool about the syntax. As such, it seems like a question that could be asked is: why not use more standard syntax?

In particular, the TFVARS file format could very easily be based on INI syntax, and use the "gcfg" library. In addition, I wonder why YAML was not considered as the format for the terraform definition files. A distinct advantage of YAML is that it is straightforward to convert JSON to YAML, so you get both human readable and machine readable formats.

One real advantage of using standard formats is that you can leverage pre-canned libraries, and the error checking and reporting they provide out of the box.

At this point in the development, it appears that the TFVARS file is INI compatible, as long as you allow for a missing "[DEFAULT]" section header, and YAML support should be very easy to add alongside your current syntax and JSON, while maintaining the interpolation syntax.

Thoughts?

[Mitchell Hashimoto]

Robert,

We support JSON for all configuration formats in Terraform, so you can
use that. This isn't a second class citizen, it is fully supported,
has many unit tests, and is purposely built-in to support machine
interaction with Terraform (which we're already building upon). I
suppose I should add this to a FAQ or something, but let me go through
why we chose to create our own format versus using another.

First, a sentence or two on a brief history of my interaction with
configuration formats. I first started Vagrant, which uses a Ruby DSL
for configuration. Throughout the years, this has worked, but a good
number of people have always asked for a standard data structure
format (JSON, YAML, etc.). Despite the fact that Ruby can load any of
these formats, people wanted an officially supported one. Next, I
started Packer, which used JSON. A good number of people have now
asked for something with logic, comments, better human-readability.
The thing is: you can't really win, and its not a bad thing, it just
must be accepted that you can't please everyone here.

So in Terraform, we set out to bring the best of two paradigms: human
friendly and machine interoperable. I firmly believe this format
cannot be a single format, so we supported both our custom HCL and
JSON.

INI is a great format. We use it for a lot of smaller projects (prior
to HCL, now we just use that everywhere). My main disagreement with
INI is that nested structures is quite ugly, and anything other than
string to string mappings is custom and is also ugly. I'd have to
write a parser anyways, basically. The example you even used "gcfg" is
based off of gitconfig, which is a strictly string-to-string small
hierarchy config. They use periods for hierarchy "core.foo". There is
a lot of hierarchy in Terraform files, so it'd be tricky. TFVars is
easier, but HCL and gcfg are identical in that case (except you need
quotes)...

YAML is just a horrid format (obviously my opinion). I've never in my
life used a format and been so sad. This strong opinion is why you'll
never see any HashiCorp support YAML in any official way except
perhaps as interoperability with another project/tool/system. I used
Salt for a good long while, Rails before that, Puppet has some YAML,
etc. What I'm trying to say is that I've given YAML a good fair
chance. There are a few problems with YAML I just despise:

1.) It is very easy to not know what data structure something is
making. I found myself constantly asking "is this a map? is this a
list of maps? is this a map with one list?" I consider myself a
not-stupid individual but getting trapped into these confusions
routinely is a bad sign.

2.) Simple primitive-to-primitive YAML is simple, but then you get
to references (&foo) and inheritence and all this madness using a
syntax that isn't very straightforward. Did that "*foo" just save that
and then not use it? Does "&foo" inherit multiple times? The
complexity is very much real: the specification for YAML is 86 pages
(printed). That is a bit shocking for a data structure language.

I really love JSON though. JSON is simple, the specification is
simple, the types are simple. The downside of JSON is a lack of
comments, and a noisy syntax: you get a lot of quotes, brackets,
commas. The noisiness is very much like Lisp and parentheses (have you
ever counted how many brackets and commas are in a basic JSON
structure?). The result is something where a good % of the characters
in a JSON file aren't human-friendly and the lack of comments just
adds to the noise. Therefore, I see JSON as a fantastic machine
interoperation language, but a sub-par (but not terrible relative to
others) human language.

Perhaps a solution is to extend the syntax document even further.
We've done our best to be very explicit about syntax:
http://www.terraform.io/docs/configuration/resources.html (see
"Syntax")

I hope this helps.

Best,
Mitchell

> --
> You received this message because you are subscribed to the Google Groups
> "Terraform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to terraform-too...@googlegroups.com.
> To post to this group, send email to terrafo...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/terraform-tool/36cdb574-2749-4242-8ee3-f30295365c69%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Cunningham]

Had to respond to this thread with a few of my own cents :)

The fact that Terraform accepts JSON makes it infinitely more attractive for me.  In my opinion the configuration language really has nothing to do with Terraform's core objectives.  Terraform provides useful functionality and there are an unbounded number of ways that this functionality can be described.  Prescribing one would be suicide, since there is no sweetest sweet spot in this vast design space and everyone has a different opinion.  It's easier to move mountains than change peoples' opinions about configuration languages.  At least mountains are made of physical substances that behave according to consistent well-understood sets of rules.

I see Terraform as something like LLVM, that provides a neatly modular and squared out set of low-level / straight-forward behaviors.  A solid foundation to build on.  By providing only that, these sorts of language debates don't even happen.

The caveat of course is that no-one writes directly to LLVM so there needs to be some example front end to demonstrate the layer underneath.  That means there has to exist at least one high level language that's easy to understand (without having to learn it).  Mitchell has written HCL to be that language, but you don't have to use it.

So, basically:

Supporting JSON:  Excellent, this avoids the issue entirely.  Use Jsonnet / Nix / Coil / Pystachio or any number of configuration languages / libraries to generate the JSON, according to your opinions, company culture, etc.  You can even roll your own, as people have done for Cloud Formation.  You can even use YAML, just do this and then run Terraform on the result.  Personally I recommend Jsonnet, as I wrote it (with some help from my colleagues), but if you want to use YAML then it's not that hard to do so and good luck with making sure all the whitespace lines up and using - in exactly the right places :)

HCL: Good for writing examples on the website so that the JSON verbosity doesn't put people off.

FWIW: Here is the Terraform "basic two tier AWS" example in YAML.  You can use an online converter to see the equivalent JSON that can be fed directly to Terraform.

---

  variable: 

    - aws_access_key: {}

    - aws_secret_key: {}

    - key_path: {}

    - key_name: {}

    - aws_region: 

        default: "us-west-2"

    - aws_amis: 

        default: 

          eu-west-1: "ami-b1cf19c6"

          us-east-1: "ami-de7ab6b6"

          us-west-1: "ami-3f75767a"

          us-west-2: "ami-21f78e11"

  provider: 

    aws: 

      access_key: "${var.aws_access_key}"

      secret_key: "${var.aws_secret_key}"

      region: "${var.aws_region}"

  resource: 

    - aws_security_group: 

        default: 

          name: "terraform_example"

          description: "Used in the terraform"

          ingress: 

            - from_port: 22

              to_port: 22

              protocol: "tcp"

              cidr_blocks: 

                - "0.0.0.0/0"

            - from_port: 80

              to_port: 80

              protocol: "tcp"

              cidr_blocks: 

                - "0.0.0.0/0"

    - aws_elb: 

        web: 

          name: "terraform-example-elb"

          availability_zones: 

            - "${aws_instance.web.availability_zone}"

          listener: 

            instance_port: 80

            instance_protocol: "http"

            lb_port: 80

            lb_protocol: "http"

          instances: 

            - "${aws_instance.web.id}"

    - aws_instance: 

        web: 

          connection: 

            user: "ubuntu"

            key_file: "${var.key_path}"

          instance_type: "m1.small"

          ami: "${lookup(var.aws_amis, var.aws_region)}"

          key_name: "${var.key_name}"

          security_groups: 

            - "${aws_security_group.default.name}"

          provisioner: 

            remote-exec: 

              inline: 

                - "sudo apt-get -y update"

                - "sudo apt-get -y install nginx"

                - "sudo service nginx start"

  output: 

    address: 

      value: "${aws_elb.web.dns_name}"

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CACOXMX441aPHVO1GT5i9m9WehcHtMYeqx46GsEJHGTD2HyoMqA%40mail.gmail.com.

[Rob Parrott]

Hi Mitchell,

Good stuff. I appreciate the thoughtful response. I don’t seem to get the shivers with YAML that you do, but then again I’ve done a good bit of python programming and don’t have as strong an opinion about whitespace. I’ve also not tried to push YAML to it’s limits.

My experience has been trying to tame complex JSON CloudFormation templates, and make them easier to work with for users. We built a tool called nepho (https://github.com/huit/nepho) which I see terraform as supplanting at some point. The intent of this tool was to make JSON and other machine syntaxes easier to work with, and introduced tempting (Jinja2) to allow for more compact JSON expressions, comments, includes and a standard library. The results were mixed, and users seemed more willing to bear the 20 page JSON code instead of learn a more compact but complex syntax. YAML seems like a sane alternative, with much better support for block literals (there’s nothing worse than trying to encode an embedded USERDATA script as JSON, comments, more liberal quoting and whitespace rules, and is both machine & human readable.  

It seems that most of the concerns that you’ve expressed about YAML would be handled by a sane YAML library; I assume the end result of interpreting the configuration, whether HCL, JSON or YAML, is the object hierarchy with a number of interpolation references. As such, providing YAML support — even as a preprocessing step YAML->JSON —  might be a good way for those who don’t mind it to get started with the tool.

As for INI format, I was thinking of it as a way of specifying and scoping simpler variables as TFVARS does now, with a familiar human-readable format.  If you’re looking for the ability to store nested objects, then JSON seems to make more sense then either the current format or INI. 

Totally on the same page about JSON. 

Given your commitment to the current formats, I’d really encourage your team to focus on more user-friendly interactions around the HCL & TFVARS parsing. This should include where possible filenames, line numbers, and clear reasons for failures. So far in my time working with terraform, I’ve only seen the tool hang, a generic usage message, or the tool succeed. The only difference between those invocations were syntax issues. As a result I spent a full day debugging an issue with a published example, instead of using the tool for what I was really trying to do with it. Right now I’m hesitant to do more with it, because I’m concerned the current behavior would turn others off to it prematurely. 

Anyway, I love your passion and am enthusiastic about the direction terraform is going. 

Thanks again!

Rob

[Mitchell Hashimoto]

Rob,

Inline:

On Mon, Sep 8, 2014 at 4:07 AM, Rob Parrott <robpa...@gmail.com> wrote:
> Hi Mitchell,
>
> Good stuff. I appreciate the thoughtful response. I don’t seem to get the shivers with YAML that you do, but then again I’ve done a good bit of python programming and don’t have as strong an opinion about whitespace. I’ve also not tried to push YAML to it’s limits.
>
> My experience has been trying to tame complex JSON CloudFormation templates, and make them easier to work with for users. We built a tool called nepho (https://github.com/huit/nepho) which I see terraform as supplanting at some point. The intent of this tool was to make JSON and other machine syntaxes easier to work with, and introduced tempting (Jinja2) to allow for more compact JSON expressions, comments, includes and a standard library. The results were mixed, and users seemed more willing to bear the 20 page JSON code instead of learn a more compact but complex syntax. YAML seems like a sane alternative, with much better support for block literals (there’s nothing worse than trying to encode an embedded USERDATA script as JSON, comments, more liberal quoting and whitespace rules, and is both machine & human readable.
>
> It seems that most of the concerns that you’ve expressed about YAML would be handled by a sane YAML library; I assume the end result of interpreting the configuration, whether HCL, JSON or YAML, is the object hierarchy with a number of interpolation references. As such, providing YAML support — even as a preprocessing step YAML->JSON — might be a good way for those who don’t mind it to get started with the tool.
>
> As for INI format, I was thinking of it as a way of specifying and scoping simpler variables as TFVARS does now, with a familiar human-readable format. If you’re looking for the ability to store nested objects, then JSON seems to make more sense then either the current format or INI.
>
> Totally on the same page about JSON.
>
> Given your commitment to the current formats, I’d really encourage your team to focus on more user-friendly interactions around the HCL & TFVARS parsing. This should include where possible filenames, line numbers, and clear reasons for failures. So far in my time working with terraform, I’ve only seen the tool hang, a generic usage message, or the tool succeed. The only difference between those invocations were syntax issues. As a result I spent a full day debugging an issue with a published example, instead of using the tool for what I was really trying to do with it. Right now I’m hesitant to do more with it, because I’m concerned the current behavior would turn others off to it prematurely.

We're working on improving this. The hang was just fixed yesterday. I
think the reasons for this are unfortunately due to us switching to a
new pure-Go HCL library in 0.2, so we're working out various kinks in
the lexer now. We're focused on providing good error messages and
improving them as time goes on.

Best,
Mitchell

> To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/etPan.540d8e0f.74b0dc51.145%40zephyr.local.