Using AWS elb.dns_name within instance config

[ma...@cockroachlabs.com]

Hello,

I have a slightly peculiar setup where my AWS instances need the load balancer's address as an argument on the command line.

The reason for this is to allow discovery of existing nodes. we don't need an exhaustive list, just a random one, with retries if the target is not good.

There are other ways of doing this (actual registry, round-robin DNS, etc...) but using a load balancer is fairly simple and reliable, and avoids extra infrastructure.

The problem I run into is a dependency cycle between aws_elb and aws_instance due to the aws_elb.instances field.

I currently get around it by starting the load balancer with an instance count of 0, outputting its dns_name, and using it as a variable on a second run with an instance count > 0.

While this works, it's a bit tedious from a user's perspective. Ideally, I would like to break the dependency cycle so that I could just use aws_elb.dns_name in the aws_instance config.

One way of doing this would be to reverse the instance <-> elb relationship by having a new "aws_instance.elb" field which would denote a new backend on a load balancer.

In our current deployment tool (also written in go), this is pretty much what I do: create the load balancer first then have instance creation trigger elb updates.

aws_elb.availability_zones can also refer to instances. I get around this by having a list of zones in a given region passed to the ELB.

I've only briefly examined the code, so I do not know how complex this would be, or whether it's even desirable. Hence the email rather than an issue.

I'd also be curious to see if anyone has a similar situation.

For reference, my current config (working but tedious due to the multiple steps) can be found at: https://github.com/cockroachdb/cockroach-prod/tree/marc/terraform_aws/terraform

Our custom deployment tool (which I'm trying to replace with terraform) is the rest of the repository.

Thank you,

  - Marc

[Paul Hinze]

Hi Marc,

Thanks for the clear description of your use case.

One suggestion that immediately comes to mind is to push the DNS name lookup down to a runtime lookup on your instances. You could add a cloud-init boot script to have your instances query AWS APIs to discover any load balancer(s) they're attached to and snag the DNS name's that way.

You can also work around the cycle by separating your provisioner out into a third external `null_resource` that references both the instance and the ELB:

variable "count" { default = 3 }

resource "aws_elb" "app" {

  instances = ["${aws_instance.app.*.id}]

}

resource "aws_instance" "app" {

  count = "${var.count}"

  # ...

}

resource "null_resource" "app-provisioner" {

  count = "${var.count}

  provisioner "remote-exec" {

    connection {

      host = "${element(aws_instance.app.*.public_ip, count.index)}"

      # ... user, etc

    }

    inline = [ "/usr/local/bin/bootstrap-with-elb.sh ${aws_elb.app.public_dns}" ]

  }

}

For a more direct solution, though, in other similarly shaped scenarios we have addressed legitimate dependency cycle issues like this by breaking out an additional top level resource to model the attachment in a declarative way.

Something like this:

# ==> Proposed config, does not work today <==

variable "count" { default = 3 }

resource "aws_elb" "app" {

  # no instances field specified

}

resource "aws_instance" "app" {

  count = "${var.count}"

  # ... provisioners here, reference ELB normally

}

# New proposed resource:

resource "aws_elb_attachment" "app" {

  count    = "${var.count}"

  elb      = "${aws_elb.app.id}"

  instance = "${element(aws_instance.app.*.id, count.index)}"
}

By separating out the attachment as a resource, the aws_instance is free to reference the aws_elb without a cycle.

I filed this last idea as a GitHub issue that you can track: https://github.com/hashicorp/terraform/issues/3999

Hope this helps!

Paul

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/e9ef8081-db96-4329-9b62-adf8c0b40c81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Berhault]

Thanks Paul, I'll give the null_resource a try. I've used those before but had not thought of using them for this.

Your proposal also sounds good and would definitely solve my particular scenario.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAJpeNAO%2BPknPETzZC5F%2BFDn_uRi7c-8PvWg%2BVTLZMaSGjXwEbA%40mail.gmail.com.

[Marc Berhault]

Just tried the null_resource to break the dependency cycle, it worked like a charm. Thank you Paul.