Using Terraform in Existing VPC

[Paul Fordahl]

So I have a VPC already created and I want terraform to manage it.

I have edited the state files with the values of the VPC that was created by hand, but still I get errors when try to apply.

Does anyone have tips on doing this, maybe I am missing something...

Thanks,

Paul

[Ryan Bowlby]

What I do is break each environment into a separate directory and tf file. I have a global variables.tf that is shared across all the environment dirs using a symlink. In the variables file I keep all the resource IDs for those low-level network pieces that came into existence ahead of my use of terraform.

terraform structure:

~/repos/terraform (master●)$ tree

.

├── base

│   ├── main.tf

│   ├── provider.tf

│   ├── terraform.tfstate

│   ├── terraform.tfstate.backup

│   ├── terraform.tfvars -> ../terraform.tfvars

│   └── variables.tf -> ../variables.tf

├── environments

│   ├── int

│   │   ├── main.tf

│   │   ├── provider.tf

│   │   ├── terraform.tfstate

│   │   ├── terraform.tfstate.backup

│   │   ├── terraform.tfvars -> ../../terraform.tfvars

│   │   └── variables.tf -> ../../variables.tf

│   ├── prd

│   │   ├── main.tf

│   │   ├── provider.tf

│   │   ├── terraform.tfstate

│   │   ├── terraform.tfstate.backup

│   │   ├── terraform.tfvars -> ../../terraform.tfvars

│   │   └── variables.tf -> ../../variables.tf

│   ├── stg

│   │   ├── main.tf

│   │   ├── provider.tf

│   │   ├── terraform.tfstate

│   │   ├── terraform.tfstate.backup

│   │   ├── terraform.tfvars -> ../../terraform.tfvars

│   │   └── variables.tf -> ../../variables.tf

│   └── tst

│       ├── main.tf

│       ├── provider.tf

│       ├── terraform.tfstate

│       ├── terraform.tfstate.backup

│       ├── terraform.tfvars -> ../../terraform.tfvars

│       └── variables.tf -> ../../variables.tf

├── modules

│   ├── admin

│   │   ├── main.tf

│   │   └── variables.tf

│   ├── batch

│   │   ├── main.tf

│   │   └── variables.tf

│   ├── image

│   │   ├── main.tf

│   │   └── variables.tf

│   ├── poolmonitor

│   │   ├── main.tf

│   │   └── variables.tf

│   ├── upload

│   │   ├── main.tf

│   │   └── variables.tf

│   └── web

│       ├── main.tf

│       └── variables.tf

├── secrets

│   ├── domain-cert.pem

│   ├── domain-chain.pem

│   ├── domain-key.pem

├── terraform.tfvars

├── user_data.txt

└── variables.tf

* sercrets and terraform.tfvars in .gitignore

An example of the kind of statically defined resources contained in the variables.tf:

variable "nonprd_vpcs" {

  default = {

    us-east-1 = "vpc-77777777"

    eu-west-1 = "vpc-88888888"

    us-west-2 = "vpc-22222222"

    us-west-1 = "vpc-44444444"

    ap-southeast-2 = "vpc-1233333"

    ap-northeast-1 = "vpc-9999999"

  }

}

variable "nonprd_public_subnets" {

  default = {

    us-east-1 = "subnet-000000,subnet-12333333"

    eu-west-1 = ""

    us-west-2 = "subnet-12333331,subnet-7777777"

    us-west-1 = ""

    ap-southeast-2 = ""

    ap-northeast-1 = ""

  }

}

Then a given environments main.tf that strictly makes use of modules for services may begin like so:

// env specific settings

variable "environment" { default = "int" }

variable "region" { default = "us-west-1" }

variable "instance_type" { default = "m3.large" }

module "upload" {

  source = "../../modules/upload"

  region = "${var.region}"

  vpc_id = "${lookup(var.nonprd_vpcs, var.region)}"

  subnets = "${lookup(var.nonprd_public_subnets, var.region)}"

....

}

In the end the VPC ids are only ever defined in one place. I use this pattern for VPCs, ec2 keys, subnets, and a few other low-level aws resources that preexisted terraform's use in our production environment.

Hope that helps. I hope to potentially get away from state-files-per-env. I think there's a better pattern. I can't leave this one till I can easily target a taint or destroy to a specific environment. Since "environment" is an abstract construct in Terraform I would have to represent an environment as a module or make use of features that don't exist (resource tagging/metaparams) to target/filter resources in a specific environment to take action upon.

-Ryan

[Paul Fordahl]

Thank you very much for sharing that, that is a great layout and I am always looking for how other people more experienced then myself lay it out! :)

But the one thing is that in a VPC that was created with out terraform how can you change the TF state files so that terraform can then spin up resources in there?

I know some people said create a VPC with Terraform and change the values of that TF state file with the ones you did by hand, but I have done that and it fails.

Do I need to remove the tf state backup so it does not have anything to compair against so it won't try to set the new values?

I feel I am close just getting lost in the weeds on this one....

Thanks!

Paul

[Martin Atkins]

Paul,

Did you get a specific error message when you tried to refresh or plan after manually inserting the VPC into your state file?

If you're using remote state then there are some caveats to watch out for with regards to the state serial number, but if you're working locally then it should work fine in most cases. Any detail you can share from the error message will hopefully help figure out what edge-case you've hit that's making this not work for you. :)